[Bug] [Config] Dependency conflict with using seatunnel-config-shade

liunaijie commented 1 week ago

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

I am try to use seatunnel-config-shade library to generate config file out of seatunnel. But find some issue.

Sample Code

Config config = ConfigFactory.parseFile(new File(configFilePath));
 // resolve with global properties
return config.root().render(CONFIG_RENDER_OPTIONS);

Then I use this result submit to SeaTunnel via RestAPI.

error message

java.util.LinkedHashMap cannot be cast to java.util.ArrayList

The submitted config to SeaTunnel is:

{
    "sink" : {
        "Console" : {}
    },
    "source" : {
        "FakeSource" : {
            "schema" : {
                "fields" : {
                    "name" : "string",
                    "age" : "int"
                }
            },
            "row.num" : 16,
            "parallelism" : 2,
            "result_table_name" : "fake"
        }
    },
    "env" : {
        "job.mode" : "BATCH",
        "parallelism" : 2,
        "checkpoint.interval" : 10000
    }
}

Issue

The generated config format is not right, the right format should be:

"source" : [ {xxxxx} ]

But error format is :

"source" : {xxxxx}

And when parse config file, we has a type cast the source to List<Map<String, Object>> https://github.com/apache/seatunnel/blob/132278c06a6f2af12934b948f65f8baf4b69db79/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java#L142

But If using origin typesafe-config to parse Config File, It will read source as SimpleConfigObject. So when cast to List we will got exception.

What did I do

Delete seatunnel-config-base library, find it will get NoClassDefFoundError error
using arthas to check check the class, I find the overwrite class under this folder https://github.com/apache/seatunnel/tree/dev/seatunnel-config/seatunnel-config-shade/src/main/java/org/apache/seatunnel/shade/com/typesafe/config all will load from seatunnel-config-shade the not overwrite class will load from seatunnel-config-base

About Env

This also rediculous, In my local desktop, mac/windows it can work well. But In sometime on special env, it will got this issue. Now I am using Centos7, it will got issue.

SeaTunnel Version

2.3.8

SeaTunnel Config

demo batch config file

Running Command

out of seatunnel, using `seatunnel-config-shade` library to generate config

Error Exception

java.util.LinkedHashMap cannot be cast to java.util.ArrayList

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

liunaijie commented 1 week ago

Hisoka-X commented 1 week ago

cc @hailin0

liunaijie commented 1 week ago

This pr fix the issue https://github.com/apache/seatunnel/pull/7893

apache / seatunnel