search

wgzhao / Addax

Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration.
https://wgzhao.github.io/Addax/
Apache License 2.0
1.12k stars 289 forks source link

自行编译项目后,执行 addax.sh 时无法正常加载 shared和plugin/*/*/libs的依赖 #990

Closed lanyugod closed 6 months ago

lanyugod commented 6 months ago

我根据项目文档中介绍使用命令： mvn clean package mvn package assembly:single bash shrink_package.sh 三条命令打包成 addax-4.0.8.tar.gz 到自己指定目录后解压，文件目录下有：

drwxr-xr-x    6 lanyu  staff   192B Dec 18 18:00 bin
drwxr-xr-x    4 lanyu  staff   128B Dec 18 16:45 conf
drwxr-xr-x   30 lanyu  staff   960B Dec 18 16:45 job
drwxr-xr-x   54 lanyu  staff   1.7K Dec 18 16:45 lib
drwxr-xr-x    4 lanyu  staff   128B Dec 18 16:45 plugin
drwxr-xr-x  572 lanyu  staff    18K Dec 18 17:36 shared

其中 shared中有所需要的 jar包 : hadoop-apache-3.2.0-14.jar

└── shared
    ├── HikariCP-2.6.1.jar
    ├── HikariCP-java7-2.4.12.jar
    ├── SparseBitSet-1.2.jar
    ├── accessors-smart-1.2.jar
    ├── accessors-smart-2.4.7.jar
    ├── activation-1.1.jar
    ├── aircompressor-0.10.jar
    ├── aircompressor-0.21.jar
    ├── animal-sniffer-annotations-1.17.jar
    ├── annotations-13.0.jar
    ├── annotations-2.0.3.jar
    ├── ant-1.9.1.jar
    ├── ant-launcher-1.9.1.jar
    ├── antlr-runtime-3.5.2.jar
    ├── aopalliance-1.0.jar
    ├── aopalliance-repackaged-2.5.0-b32.jar
    ├── apache-jsp-9.3.20.v20170531.jar
    ├── apache-jstl-9.3.20.v20170531.jar
    ├── apacheds-i18n-2.0.0-M15.jar
    ├── apacheds-kerberos-codec-2.0.0-M15.jar
    ├── api-asn1-api-1.0.0-M20.jar
    ├── api-util-1.0.0-M20.jar
    ├── argparse4j-0.8.1.jar
    ├── arrow-format-0.8.0.jar
    ├── arrow-memory-0.8.0.jar
    ├── arrow-vector-0.8.0.jar
    ├── asm-3.1.jar
    ├── asm-5.0.3.jar
    ├── asm-5.0.4.jar
    ├── asm-9.1.jar
    ├── asm-all-5.0.2.jar
    ├── asm-analysis-5.0.3.jar
    ├── asm-commons-5.0.1.jar
    ├── asm-commons-5.0.3.jar
    ├── asm-tree-5.0.1.jar
    ├── asm-tree-5.0.3.jar
    ├── asm-util-5.0.3.jar
    ├── async-1.4.1.jar
    ├── audience-annotations-0.5.0.jar
    ├── avro-1.7.4.jar
    ├── avro-1.7.7.jar
    ├── bonecp-0.8.0.RELEASE.jar
    ├── cassandra-driver-core-3.11.0-shaded.jar
    ├── cassandra-driver-core-3.11.0.jar
    ├── checker-qual-2.11.1.jar
    ├── checker-qual-2.5.2.jar
    ├── checker-qual-3.5.0.jar
    ├── clickhouse-client-0.3.2.jar
    ├── clickhouse-grpc-client-0.3.2-netty.jar
    ├── clickhouse-http-client-0.3.2-shaded.jar
    ├── clickhouse-http-client-0.3.2.jar
    ├── clickhouse-jdbc-0.3.2.jar
    ├── commons-beanutils-1.7.0.jar
    ├── commons-beanutils-1.9.4.jar
    ├── commons-beanutils-core-1.8.0.jar
    ├── commons-cli-1.2.jar
    ├── commons-codec-1.11.jar
    ├── commons-codec-1.15.jar
    ├── commons-codec-1.7.jar
    ├── commons-codec-1.9.jar
    ├── commons-collections-3.2.1.jar
    ├── commons-collections-3.2.2.jar
    ├── commons-collections4-4.4.jar
    ├── commons-compress-1.19.jar
    ├── commons-compress-1.21.jar
    ├── commons-compress-1.4.1.jar
    ├── commons-configuration-1.6.jar
    ├── commons-configuration2-2.1.1.jar
    ├── commons-crypto-1.0.0.jar
    ├── commons-csv-1.0.jar
    ├── commons-csv-1.8.jar
    ├── commons-csv-1.9.0.jar
    ├── commons-daemon-1.0.13.jar
    ├── commons-dbcp-1.4.jar
    ├── commons-dbcp2-2.8.0.jar
    ├── commons-digester-1.8.jar
    ├── commons-el-1.0.jar
    ├── commons-httpclient-3.1.jar
    ├── commons-io-2.11.0.jar
    ├── commons-lang-2.6.jar
    ├── commons-lang3-3.12.0.jar
    ├── commons-logging-1.2.jar
    ├── commons-math-2.2.jar
    ├── commons-math3-3.1.1.jar
    ├── commons-math3-3.6.1.jar
    ├── commons-net-3.1.jar
    ├── commons-net-3.6.jar
    ├── commons-net-3.8.0.jar
    ├── commons-pool-1.5.4.jar
    ├── commons-pool2-2.10.0.jar
    ├── commons-pool2-2.8.1.jar
    ├── commons-text-1.4.jar
    ├── converter-gson-2.9.0.jar
    ├── converter-moshi-2.9.0.jar
    ├── converter-scalars-2.9.0.jar
    ├── curator-client-2.12.0.jar
    ├── curator-client-2.13.0.jar
    ├── curator-client-2.7.1.jar
    ├── curator-framework-2.12.0.jar
    ├── curator-framework-2.13.0.jar
    ├── curator-framework-2.7.1.jar
    ├── curator-recipes-2.12.0.jar
    ├── curator-recipes-2.13.0.jar
    ├── curator-recipes-2.7.1.jar
    ├── curvesapi-1.06.jar
    ├── datanucleus-api-jdo-4.2.4.jar
    ├── datanucleus-core-4.1.17.jar
    ├── datanucleus-rdbms-4.1.19.jar
    ├── db2jcc-db2jcc4.jar
    ├── derby-10.14.1.0.jar
    ├── disruptor-3.3.6.jar
    ├── dnsjava-2.1.7.jar
    ├── dom4j-1.6.1.jar
    ├── dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
    ├── druid-1.2.6.jar
    ├── ecj-4.4.2.jar
    ├── ehcache-3.3.1.jar
    ├── error_prone_annotations-2.2.0.jar
    ├── error_prone_annotations-2.3.4.jar
    ├── failureaccess-1.0.1.jar
    ├── failureaccess-1.0.jar
    ├── fastjson-1.2.76.jar
    ├── fastutil-6.5.6.jar
    ├── findbugs-annotations-1.3.9-1.jar
    ├── flatbuffers-1.2.0-3f79e055.jar
    ├── fluent-hc-4.5.13.jar
    ├── fst-2.50.jar
    ├── geronimo-jcache_1.0_spec-1.0-alpha-1.jar
    ├── gson-2.2.4.jar
    ├── gson-2.8.0.jar
    ├── gson-2.8.8.jar
    ├── gson-fire-1.8.4.jar
    ├── guava-12.0.1.jar
    ├── guava-13.0.1.jar
    ├── guava-19.0.jar
    ├── guava-21.0.jar
    ├── guava-27.0-jre.jar
    ├── guava-29.0-jre.jar
    ├── guice-3.0.jar
    ├── guice-assistedinject-3.0.jar
    ├── guice-servlet-3.0.jar
    ├── guice-servlet-4.0.jar
    ├── hadoop-annotations-2.6.0.jar
    ├── hadoop-annotations-2.7.4.jar
    ├── hadoop-annotations-2.7.5.jar
    ├── hadoop-annotations-3.2.2.jar
    ├── hadoop-apache-3.2.0-14.jar
    ├── hadoop-auth-2.7.1.jar
    ├── hadoop-auth-2.7.4.jar
    ├── hadoop-auth-3.2.2.jar
    ├── hadoop-client-2.7.1.jar
    ├── hadoop-client-2.7.4.jar
    ├── hadoop-common-2.6.0.jar
    ├── hadoop-common-2.7.4.jar
    ├── hadoop-common-2.7.5.jar
    ├── hadoop-common-3.2.2.jar
    ├── hadoop-distcp-2.7.1.jar
    ├── hadoop-hdfs-2.2.0.jar
    ├── hadoop-hdfs-2.7.4.jar
    ├── hadoop-mapreduce-client-app-2.7.1.jar
    ├── hadoop-mapreduce-client-app-2.7.4.jar
    ├── hadoop-mapreduce-client-common-2.7.1.jar
    ├── hadoop-mapreduce-client-common-2.7.4.jar
    ├── hadoop-mapreduce-client-core-2.7.1.jar
    ├── hadoop-mapreduce-client-core-2.7.4.jar
    ├── hadoop-mapreduce-client-core-2.7.5.jar
    ├── hadoop-mapreduce-client-jobclient-2.7.1.jar
    ├── hadoop-mapreduce-client-jobclient-2.7.4.jar
    ├── hadoop-mapreduce-client-shuffle-2.7.1.jar
    ├── hadoop-mapreduce-client-shuffle-2.7.4.jar
    ├── hadoop-yarn-api-2.7.4.jar
    ├── hadoop-yarn-api-3.1.0.jar
    ├── hadoop-yarn-client-2.7.1.jar
    ├── hadoop-yarn-client-2.7.4.jar
    ├── hadoop-yarn-common-2.7.4.jar
    ├── hadoop-yarn-common-2.7.5.jar
    ├── hadoop-yarn-common-3.1.0.jar
    ├── hadoop-yarn-registry-3.1.0.jar
    ├── hadoop-yarn-server-applicationhistoryservice-3.1.0.jar
    ├── hadoop-yarn-server-common-2.7.4.jar
    ├── hadoop-yarn-server-common-3.1.0.jar
    ├── hadoop-yarn-server-resourcemanager-3.1.0.jar
    ├── hadoop-yarn-server-web-proxy-3.1.0.jar
    ├── hamcrest-core-1.3.jar
    ├── hbase-annotations-1.4.13.jar
    ├── hbase-annotations-1.4.6.jar
    ├── hbase-client-1.4.13.jar
    ├── hbase-client-1.4.6.jar
    ├── hbase-client-2.0.0-alpha4.jar
    ├── hbase-client-2.2.5.jar
    ├── hbase-common-1.4.13.jar
    ├── hbase-common-1.4.6-tests.jar
    ├── hbase-common-1.4.6.jar
    ├── hbase-common-2.0.0-alpha4-tests.jar
    ├── hbase-common-2.0.0-alpha4.jar
    ├── hbase-common-2.2.5.jar
    ├── hbase-hadoop-compat-1.4.6.jar
    ├── hbase-hadoop-compat-2.0.0-alpha4.jar
    ├── hbase-hadoop-compat-2.2.5.jar
    ├── hbase-hadoop2-compat-1.4.6.jar
    ├── hbase-hadoop2-compat-2.0.0-alpha4.jar
    ├── hbase-hadoop2-compat-2.2.5.jar
    ├── hbase-http-2.0.0-alpha4.jar
    ├── hbase-mapreduce-2.0.0-alpha4.jar
    ├── hbase-metrics-1.4.6.jar
    ├── hbase-metrics-2.0.0-alpha4.jar
    ├── hbase-metrics-2.2.5.jar
    ├── hbase-metrics-api-1.4.6.jar
    ├── hbase-metrics-api-2.0.0-alpha4.jar
    ├── hbase-metrics-api-2.2.5.jar
    ├── hbase-prefix-tree-1.4.6.jar
    ├── hbase-prefix-tree-2.0.0-alpha4.jar
    ├── hbase-procedure-1.4.6.jar
    ├── hbase-procedure-2.0.0-alpha4.jar
    ├── hbase-protocol-1.4.13.jar
    ├── hbase-protocol-1.4.6.jar
    ├── hbase-protocol-2.0.0-alpha4.jar
    ├── hbase-protocol-2.2.5.jar
    ├── hbase-protocol-shaded-2.0.0-alpha4.jar
    ├── hbase-protocol-shaded-2.2.5.jar
    ├── hbase-replication-2.0.0-alpha4.jar
    ├── hbase-server-1.4.6.jar
    ├── hbase-server-2.0.0-alpha4.jar
    ├── hbase-shaded-gson-3.0.0.jar
    ├── hbase-shaded-miscellaneous-1.0.1.jar
    ├── hbase-shaded-miscellaneous-2.2.1.jar
    ├── hbase-shaded-netty-1.0.1.jar
    ├── hbase-shaded-netty-2.2.1.jar
    ├── hbase-shaded-protobuf-1.0.1.jar
    ├── hbase-shaded-protobuf-2.2.1.jar
    ├── hive-apache-3.1.2-16.jar
    ├── hive-classification-3.1.2.jar
    ├── hive-common-3.1.2.jar
    ├── hive-jdbc-3.1.2.jar
    ├── hive-llap-client-3.1.2.jar
    ├── hive-llap-common-3.1.2-tests.jar
    ├── hive-llap-common-3.1.2.jar
    ├── hive-llap-server-3.1.2.jar
    ├── hive-llap-tez-3.1.2.jar
    ├── hive-metastore-3.1.2.jar
    ├── hive-serde-3.1.2.jar
    ├── hive-service-3.1.2.jar
    ├── hive-service-rpc-3.1.2.jar
    ├── hive-shims-0.23-3.1.2.jar
    ├── hive-shims-3.1.2.jar
    ├── hive-shims-common-3.1.2.jar
    ├── hive-shims-scheduler-3.1.2.jar
    ├── hive-standalone-metastore-3.1.2.jar
    ├── hive-storage-api-2.7.0.jar
    ├── hive-upgrade-acid-3.1.2.jar
    ├── hk2-api-2.5.0-b32.jar
    ├── hk2-locator-2.5.0-b32.jar
    ├── hk2-utils-2.5.0-b32.jar
    ├── hppc-0.7.2.jar
    ├── htrace-core-3.0.4.jar
    ├── htrace-core-3.1.0-incubating.jar
    ├── htrace-core-3.2.0-incubating.jar
    ├── htrace-core4-4.1.0-incubating.jar
    ├── httpasyncclient-4.1.3.jar
    ├── httpclient-4.0.1.jar
    ├── httpclient-4.2.5.jar
    ├── httpclient-4.5.13.jar
    ├── httpclient-4.5.2.jar
    ├── httpclient-4.5.3.jar
    ├── httpclient-4.5.8.jar
    ├── httpcore-4.0.1.jar
    ├── httpcore-4.2.4.jar
    ├── httpcore-4.4.11.jar
    ├── httpcore-4.4.13.jar
    ├── httpcore-4.4.4.jar
    ├── httpcore-4.4.6.jar
    ├── httpcore-nio-4.4.6.jar
    ├── httpmime-4.5.13.jar
    ├── hutool-all-5.8.0.jar
    ├── i18n-util-1.0.4.jar
    ├── icu4j-60.2.jar
    ├── icu4j-charset-60.2.jar
    ├── icu4j-localespi-60.2.jar
    ├── influxdb-client-core-3.3.0.jar
    ├── influxdb-client-java-3.3.0.jar
    ├── influxdb-client-utils-3.3.0.jar
    ├── influxdb-java-2.22.jar
    ├── j2objc-annotations-1.1.jar
    ├── j2objc-annotations-1.3.jar
    ├── jackson-annotations-2.7.0.jar
    ├── jackson-annotations-2.9.0.jar
    ├── jackson-annotations-2.9.10.jar
    ├── jackson-core-2.7.9.jar
    ├── jackson-core-2.9.10.jar
    ├── jackson-core-2.9.5.jar
    ├── jackson-core-asl-1.9.13.jar
    ├── jackson-core-asl-1.9.2.jar
    ├── jackson-databind-2.7.9.3.jar
    ├── jackson-databind-2.9.10.4.jar
    ├── jackson-databind-2.9.5.jar
    ├── jackson-jaxrs-1.8.3.jar
    ├── jackson-jaxrs-1.9.13.jar
    ├── jackson-jaxrs-1.9.2.jar
    ├── jackson-jaxrs-base-2.7.8.jar
    ├── jackson-jaxrs-json-provider-2.7.8.jar
    ├── jackson-mapper-asl-1.9.13.jar
    ├── jackson-mapper-asl-1.9.2.jar
    ├── jackson-module-jaxb-annotations-2.7.8.jar
    ├── jackson-xc-1.8.3.jar
    ├── jackson-xc-1.9.2.jar
    ├── jamon-runtime-2.3.1.jar
    ├── jamon-runtime-2.4.1.jar
    ├── jasper-compiler-5.5.23.jar
    ├── jasper-runtime-5.5.23.jar
    ├── java-util-1.9.0.jar
    ├── java-xmlbuilder-0.4.jar
    ├── javacsv-2.0.jar
    ├── javadbf-1.13.2.jar
    ├── javassist-3.20.0-GA.jar
    ├── javax.activation-api-1.2.0.jar
    ├── javax.annotation-api-1.2.jar
    ├── javax.el-3.0.1-b12.jar
    ├── javax.inject-1.jar
    ├── javax.inject-2.5.0-b32.jar
    ├── javax.jdo-3.2.0-m3.jar
    ├── javax.servlet-api-3.1.0.jar
    ├── javax.servlet.jsp-2.3.2.jar
    ├── javax.servlet.jsp-api-2.3.1.jar
    ├── javax.ws.rs-api-2.0.1.jar
    ├── javolution-5.5.1.jar
    ├── jaxb-api-2.2.11.jar
    ├── jaxb-api-2.2.2.jar
    ├── jaxb-impl-2.2.3-1.jar
    ├── jcip-annotations-1.0-1.jar
    ├── jcodings-1.0.18.jar
    ├── jcodings-1.0.8.jar
    ├── jdo-api-3.0.1.jar
    ├── jedis-3.7.0.jar
    ├── jersey-client-1.19.jar
    ├── jersey-client-1.9.jar
    ├── jersey-client-2.25.1.jar
    ├── jersey-common-2.25.1.jar
    ├── jersey-container-servlet-core-2.25.1.jar
    ├── jersey-core-1.19.jar
    ├── jersey-core-1.9.jar
    ├── jersey-guava-2.25.1.jar
    ├── jersey-guice-1.19.jar
    ├── jersey-guice-1.9.jar
    ├── jersey-json-1.19.jar
    ├── jersey-json-1.9.jar
    ├── jersey-media-jaxb-2.25.1.jar
    ├── jersey-server-1.19.jar
    ├── jersey-server-1.9.jar
    ├── jersey-server-2.25.1.jar
    ├── jersey-servlet-1.19.jar
    ├── jest-6.3.1.jar
    ├── jest-common-6.3.1.jar
    ├── jets3t-0.9.0.jar
    ├── jettison-1.1.jar
    ├── jetty-6.1.26.jar
    ├── jetty-annotations-9.3.20.v20170531.jar
    ├── jetty-client-9.3.20.v20170531.jar
    ├── jetty-http-9.3.20.v20170531.jar
    ├── jetty-http-9.4.20.v20190813.jar
    ├── jetty-io-9.3.20.v20170531.jar
    ├── jetty-io-9.4.20.v20190813.jar
    ├── jetty-jaas-9.3.20.v20170531.jar
    ├── jetty-jndi-9.3.20.v20170531.jar
    ├── jetty-plus-9.3.20.v20170531.jar
    ├── jetty-rewrite-9.3.20.v20170531.jar
    ├── jetty-runner-9.3.20.v20170531.jar
    ├── jetty-schemas-3.1.jar
    ├── jetty-security-9.3.20.v20170531.jar
    ├── jetty-security-9.4.20.v20190813.jar
    ├── jetty-server-9.3.20.v20170531.jar
    ├── jetty-server-9.4.20.v20190813.jar
    ├── jetty-servlet-9.3.20.v20170531.jar
    ├── jetty-servlet-9.4.20.v20190813.jar
    ├── jetty-sslengine-6.1.26.jar
    ├── jetty-util-6.1.26.jar
    ├── jetty-util-9.3.20.v20170531.jar
    ├── jetty-util-9.4.20.v20190813.jar
    ├── jetty-util-ajax-9.3.19.v20170502.jar
    ├── jetty-webapp-9.3.20.v20170531.jar
    ├── jetty-webapp-9.4.20.v20190813.jar
    ├── jetty-xml-9.3.20.v20170531.jar
    ├── jetty-xml-9.4.20.v20190813.jar
    ├── jffi-1.2.16-native.jar
    ├── jffi-1.2.16.jar
    ├── jline-0.9.94.jar
    ├── jline-2.11.jar
    ├── jline-2.12.jar
    ├── jnr-constants-0.9.9.jar
    ├── jnr-ffi-2.1.7.jar
    ├── jnr-posix-3.0.44.jar
    ├── jnr-x86asm-1.0.2.jar
    ├── joda-time-1.6.jar
    ├── joda-time-2.10.13.jar
    ├── joda-time-2.9.9.jar
    ├── joni-2.1.11.jar
    ├── joni-2.1.2.jar
    ├── jpam-1.1.jar
    ├── jsch-0.1.42.jar
    ├── jsch-0.1.54.jar
    ├── jsch-0.1.55.jar
    ├── json-1.8.jar
    ├── json-io-2.5.1.jar
    ├── json-path-2.6.0.jar
    ├── json-smart-2.3.jar
    ├── json-smart-2.4.7.jar
    ├── jsp-2.1-6.1.14.jar
    ├── jsp-api-2.1-6.1.14.jar
    ├── jsp-api-2.1.jar
    ├── jsr305-2.0.1.jar
    ├── jsr305-3.0.0.jar
    ├── jsr305-3.0.2.jar
    ├── jsr311-api-1.1.1.jar
    ├── jta-1.1.jar
    ├── junit-4.12.jar
    ├── kerb-admin-1.0.1.jar
    ├── kerb-client-1.0.1.jar
    ├── kerb-common-1.0.1.jar
    ├── kerb-core-1.0.1.jar
    ├── kerb-crypto-1.0.1.jar
    ├── kerb-identity-1.0.1.jar
    ├── kerb-server-1.0.1.jar
    ├── kerb-simplekdc-1.0.1.jar
    ├── kerb-util-1.0.1.jar
    ├── kerby-asn1-1.0.1.jar
    ├── kerby-config-1.0.1.jar
    ├── kerby-pkix-1.0.1.jar
    ├── kerby-util-1.0.1.jar
    ├── kerby-xdr-1.0.1.jar
    ├── kotlin-stdlib-1.3.70.jar
    ├── kotlin-stdlib-1.4.10.jar
    ├── kotlin-stdlib-common-1.3.70.jar
    ├── kotlin-stdlib-common-1.4.0.jar
    ├── kotlin-stdlib-jdk7-1.4.10.jar
    ├── kotlin-stdlib-jdk8-1.4.10.jar
    ├── kudu-client-1.15.0.jar
    ├── leveldbjni-all-1.8.jar
    ├── libfb303-0.9.3.jar
    ├── libthrift-0.9.0.jar
    ├── libthrift-0.9.3.jar
    ├── listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
    ├── log4j-1.2-api-2.10.0.jar
    ├── log4j-1.2.16.jar
    ├── log4j-1.2.17.jar
    ├── log4j-api-2.10.0.jar
    ├── log4j-api-2.14.1.jar
    ├── log4j-core-2.10.0.jar
    ├── log4j-slf4j-impl-2.10.0.jar
    ├── log4j-web-2.10.0.jar
    ├── logback-classic-1.2.10.jar
    ├── logback-core-1.2.10.jar
    ├── logging-interceptor-4.7.2.jar
    ├── logging-interceptor-4.9.1.jar
    ├── lz4-java-1.8.0.jar
    ├── lzo-core-1.0.6.jar
    ├── metrics-core-2.2.0.jar
    ├── metrics-core-3.1.0.jar
    ├── metrics-core-3.2.2.jar
    ├── metrics-core-3.2.6.jar
    ├── metrics-json-3.1.0.jar
    ├── metrics-jvm-3.1.0.jar
    ├── mongo-java-driver-3.12.9.jar
    ├── moshi-1.8.0.jar
    ├── msgpack-core-0.9.0.jar
    ├── mssql-jdbc-6.2.1.jre7.jar
    ├── mssql-jdbc-9.2.1.jre8.jar
    ├── mysql-connector-java-8.0.16.jar
    ├── netty-3.10.6.Final.jar
    ├── netty-3.6.2.Final.jar
    ├── netty-3.7.0.Final.jar
    ├── netty-all-4.1.17.Final.jar
    ├── netty-all-4.1.8.Final.jar
    ├── netty-buffer-4.0.56.Final.jar
    ├── netty-buffer-4.1.17.Final.jar
    ├── netty-buffer-4.1.45.Final.jar
    ├── netty-codec-4.0.56.Final.jar
    ├── netty-codec-4.1.45.Final.jar
    ├── netty-common-4.0.56.Final.jar
    ├── netty-common-4.1.17.Final.jar
    ├── netty-common-4.1.45.Final.jar
    ├── netty-handler-4.0.56.Final.jar
    ├── netty-handler-4.1.45.Final.jar
    ├── netty-resolver-4.1.45.Final.jar
    ├── netty-transport-4.0.56.Final.jar
    ├── netty-transport-4.1.45.Final.jar
    ├── netty-transport-native-epoll-4.1.45.Final.jar
    ├── netty-transport-native-unix-common-4.1.45.Final.jar
    ├── nimbus-jose-jwt-7.9.jar
    ├── ojdbc8-19.7.0.0.jar
    ├── okhttp-4.7.2.jar
    ├── okhttp-4.9.1.jar
    ├── okio-2.6.0.jar
    ├── okio-2.8.0.jar
    ├── opencsv-2.3.jar
    ├── orc-core-1.5.6.jar
    ├── orc-shims-1.5.6.jar
    ├── osgi-resource-locator-1.0.1.jar
    ├── paranamer-2.3.jar
    ├── parquet-hadoop-bundle-1.10.0.jar
    ├── phoenix-client-hbase-2.1-5.1.2.jar
    ├── phoenix-core-4.14.3-HBase-1.4.jar
    ├── poi-5.1.0.jar
    ├── poi-ooxml-5.1.0.jar
    ├── poi-ooxml-lite-5.1.0.jar
    ├── postgresql-42.3.1.jar
    ├── protobuf-java-2.5.0.jar
    ├── protobuf-java-3.6.1.jar
    ├── re2j-1.1.jar
    ├── reactive-streams-1.0.3.jar
    ├── redis-replicator-3.5.5.jar
    ├── retrofit-2.9.0.jar
    ├── rxjava-2.2.19.jar
    ├── servlet-api-2.5-6.1.14.jar
    ├── servlet-api-2.5.jar
    ├── slf4j-api-1.7.32.jar
    ├── slf4j-log4j12-1.6.1.jar
    ├── slf4j-log4j12-1.7.10.jar
    ├── slf4j-log4j12-1.7.25.jar
    ├── snappy-0.3.jar
    ├── snappy-java-1.0.4.1.jar
    ├── snappy-java-1.0.5.jar
    ├── snappy-java-1.1.8.4.jar
    ├── sqlite-jdbc-3.36.0.3.jar
    ├── sqlline-1.2.0.jar
    ├── sqlline-1.3.0.jar
    ├── stax-api-1.0-2.jar
    ├── stax2-api-3.1.4.jar
    ├── stream-2.9.5.jar
    ├── swagger-annotations-1.6.1.jar
    ├── taglibs-standard-impl-1.2.5.jar
    ├── taglibs-standard-spec-1.2.5.jar
    ├── taos-jdbcdriver-2.0.35.jar
    ├── tephra-api-0.14.0-incubating.jar
    ├── tephra-api-0.6.0.jar
    ├── tephra-core-0.14.0-incubating.jar
    ├── tephra-core-0.6.0.jar
    ├── tephra-hbase-compat-1.0-0.6.0.jar
    ├── tephra-hbase-compat-1.4-0.14.0-incubating.jar
    ├── token-provider-1.0.1.jar
    ├── transaction-api-1.1.jar
    ├── twill-api-0.6.0-incubating.jar
    ├── twill-api-0.8.0.jar
    ├── twill-common-0.6.0-incubating.jar
    ├── twill-common-0.8.0.jar
    ├── twill-core-0.6.0-incubating.jar
    ├── twill-core-0.8.0.jar
    ├── twill-discovery-api-0.6.0-incubating.jar
    ├── twill-discovery-api-0.8.0.jar
    ├── twill-discovery-core-0.6.0-incubating.jar
    ├── twill-discovery-core-0.8.0.jar
    ├── twill-zookeeper-0.6.0-incubating.jar
    ├── twill-zookeeper-0.8.0.jar
    ├── validation-api-1.1.0.Final.jar
    ├── websocket-api-9.3.20.v20170531.jar
    ├── websocket-client-9.3.20.v20170531.jar
    ├── websocket-common-9.3.20.v20170531.jar
    ├── websocket-server-9.3.20.v20170531.jar
    ├── websocket-servlet-9.3.20.v20170531.jar
    ├── woodstox-core-5.0.3.jar
    ├── xml-apis-1.0.b2.jar
    ├── xml-apis-1.4.01.jar
    ├── xmlbeans-5.0.2.jar
    ├── xmlenc-0.52.jar
    ├── xz-1.0.jar
    ├── xz-1.9.jar
    ├── zookeeper-3.4.10.jar
    ├── zookeeper-3.4.13.jar
    ├── zookeeper-3.4.6.jar
    ├── zookeeper-3.6.0.jar
    ├── zookeeper-jute-3.6.0.jar
    └── zstd-jni-1.5.1-1.jar

在执行命令时提示:java.lang.NoSuchMethodError，下面是完整的错误信息：

➜  12489 bash 26624_12489.command                 
java -server -Xms64m -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/lanyu/addax -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -Daddax.home=/Users/lanyu/addax -Dlogback.configurationFile=/Users/lanyu/addax/conf/logback.xml -classpath .:/etc/hbase/conf:/Users/lanyu/addax/lib/*:/Users/lanyu/addax/shared/* -DprocessInstanceId=26624 -DtaskInstanceId=12489 -DsourceDataSourceId=213 -DtargetDataSourceId=215 -DtaskType=OSDB -DserviceHost=dmp-api:12344 -DdslogPath=null -DtaskId=1736563022724796418 -Dloglevel=info -Daddax.log=/Users/lanyu/addax/log -Dlog.file.name=addax_26624_12489_job_20231218_180100_72383.log com.wgzhao.addax.core.Engine -job /tmp/dolphinscheduler/exec/process/0/1111111111111_1/26624/12489/26624_12489_job.json
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/lanyu/addax/lib/logback-classic-1.2.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lanyu/addax/shared/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lanyu/addax/shared/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lanyu/addax/shared/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lanyu/addax/shared/phoenix-client-hbase-2.1-5.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lanyu/addax/shared/logback-classic-1.2.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/lanyu/addax/shared/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]

  ___      _     _            
 / _ \    | |   | |           
/ /_\ \ __| | __| | __ ___  __
|  _  |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |>  < 
\_| |_/\__,_|\__,_|\__,_/_/\_\

:: Addax version ::    (v4.0.8)

2023-12-18 18:01:01.086 [        main] INFO  VMInfo               - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-12-18 18:01:01.100 [        main] INFO  Engine               - 
{
        "content":{
                "reader":{
                        "parameter":{
                                "password":"*****",
                                "autoPk":false,
                                "column":[
                                        "name",
                                        "age",
                                        "school"
                                ],
                                "connection":[
                                        {
                                                "driver":"com.mysql.cj.jdbc.Driver",
                                                "jdbcUrl":[
                                                        "jdbc:mysql://******:****/"
                                                ],
                                                "table":[
                                                        "radondb.table_stu"
                                                ]
                                        }
                                ],
                                "username":"root"
                        },
                        "name":"mysqlreader"
                },
                "writer":{
                        "parameter":{
                                "path":"/user/hive/warehouse/ods.db/cus_stu",
                                "fileName":"stu",
                                "column":[
                                        {
                                                "name":"name",
                                                "type":"STRING"
                                        },
                                        {
                                                "name":"age",
                                                "type":"INT"
                                        },
                                        {
                                                "name":"begin_time",
                                                "type":"TIMESTAMP"
                                        }
                                ],
                                "defaultFS":"hdfs://hdp01:8020",
                                "hiveConfig":{
                                        "jdbcUrl":"jdbc:hive2://*****:10000/",
                                        "table":"ods.cus_stu",
                                        "username":"root"
                                },
                                "writeMode":"overwrite",
                                "encoding":"utf-8",
                                "fieldDelimiter":"\u0001",
                                "fileType":"text"
                        },
                        "name":"hdfswriter"
                }
        },
        "setting":{
                "errorLimit":{
                        "record":"0",
                        "percentage":"0"
                },
                "speed":{
                        "byte":"-1",
                        "channel":"1"
                }
        }
}

2023-12-18 18:01:01.150 [        main] INFO  PerfTrace            - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-12-18 18:01:01.150 [        main] INFO  JobContainer         - Addax jobContainer starts job.
2023-12-18 18:01:01.155 [        main] INFO  JobContainer         - Set jobId = 0
2023-12-18 18:01:01.192 [       job-0] WARN  OriginalConfPretreatmentUtil - use specified driver class: com.mysql.cj.jdbc.Driver
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
2023-12-18 18:01:03.246 [       job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://172.16.11.164:32444/?yearIsDateType=false&zeroDateTimeBehavior=CONVERT_TO_NULL&tinyInt1isBit=false&rewriteBatchedStatements=true&useSSL=false.
2023-12-18 18:01:03.455 [       job-0] INFO  OriginalConfPretreatmentUtil - table:[radondb.table_stu] has columns:[name,age,school].
2023-12-18 18:01:03.722 [       job-0] INFO  JobContainer         - PerfTrace not enable!
java.lang.NoSuchMethodError: org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(Ljava/lang/String;)V
        at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:84)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:319)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:304)
        at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1828)
        at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
        at com.wgzhao.addax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:300)
        at com.wgzhao.addax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:74)
        at com.wgzhao.addax.core.job.JobContainer.initJobWriter(JobContainer.java:645)
        at com.wgzhao.addax.core.job.JobContainer.init(JobContainer.java:309)
        at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:129)
        at com.wgzhao.addax.core.Engine.start(Engine.java:75)
        at com.wgzhao.addax.core.Engine.entry(Engine.java:132)
        at com.wgzhao.addax.core.Engine.main(Engine.java:163)
lanyugod commented 6 months ago

我如果把hadoop-apache-3.2.0-14.jar 复制到${ADDAX_HOME}/lib 下，可以加载此方法。

lanyugod commented 6 months ago

编译 4.0.12版本也是出现了这样的问题，因为我们生产环境中使用的是 4.0.8 版本，我打算升级结果发现了这个问题，想寻求帮助

wgzhao commented 6 months ago

解压后，去 plugin/writer/hdfswriter/libs 目录下，执行 ls 后名，麻烦贴出输出结果，看看是否有 hadoop-apache-3.2.0-14.jar 文件的软连接

lanyugod commented 6 months ago

解压后，去 目录下，执行 ls 后名，麻烦贴出输出结果，看看是否有 文件的软连接plugin/writer/hdfswriter/libs``hadoop-apache-3.2.0-14.jar

是有的：

➜  libs ls -ltr |grep hadoop-apache-3.2.0-14.jar     
lrwxr-xr-x  1 lanyu  staff     45 Dec 18 18:14 hadoop-apache-3.2.0-14.jar -> ../../../../shared/hadoop-apache-3.2.0-14.jar
➜  libs pwd
/Users/lanyu/addax/plugin/writer/hdfswriter/libs
➜  libs cd ../../../../shared/    
➜  shared ls -ltr |grep hadoop-apache-3.2.0-14.jar
-rw-r--r--  1 lanyu  staff   37302576 Feb 14  2023 hadoop-apache-3.2.0-14.jar
➜  shared pwd
/Users/lanyu/addax/shared
➜  shared

我之前还通过仅打包，未压缩的 addax包测试 发现也不行。

mvn clean package
mvn package assembly:single
wgzhao commented 6 months ago

存在软连接，不应该找不到这个 jar 包，我晚一些测试下当前 master 分支上的代码，看能否重现这个问题。 hdfswriter 插件的打包指令 package.xml 这个几个版本都没有改动过。

感谢反馈！

lanyugod commented 6 months ago

抱歉，找到了原因，好像 是因为hdfswriter 我们自己引入依赖冲突导致的。