search

DTStack / chunjun

A data integration framework
https://dtstack.github.io/chunjun/
Apache License 2.0
4.01k stars 1.69k forks source link

[Question][chunjun-connector-iceberg] java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration #1717

Closed liuzx8888 closed 1 year ago

liuzx8888 commented 1 year ago

Search before asking

Description

在本地打包的环境里执行测试找不到 org/apache/hadoop/conf/Configuration 类

D:\work\Java\jdk1.8.0_101\bin\java.exe -javaagent:C:\Users\Administrator\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\231.8109.175\lib\idea_rt.jar=51058:C:\Users\Administrator\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\231.8109.175\bin -Dfile.encoding=UTF-8 -classpath D:\work\Java\jdk1.8.0_101\jre\lib\charsets.jar;D:\work\Java\jdk1.8.0_101\jre\lib\deploy.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\access-bridge-64.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\cldrdata.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\dnsns.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\jaccess.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\jfxrt.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\localedata.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\nashorn.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\sunec.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\sunjce_provider.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\sunmscapi.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\sunpkcs11.jar;D:\work\Java\jdk1.8.0_101\jre\lib\ext\zipfs.jar;D:\work\Java\jdk1.8.0_101\jre\lib\javaws.jar;D:\work\Java\jdk1.8.0_101\jre\lib\jce.jar;D:\work\Java\jdk1.8.0_101\jre\lib\jfr.jar;D:\work\Java\jdk1.8.0_101\jre\lib\jfxswt.jar;D:\work\Java\jdk1.8.0_101\jre\lib\jsse.jar;D:\work\Java\jdk1.8.0_101\jre\lib\management-agent.jar;D:\work\Java\jdk1.8.0_101\jre\lib\plugin.jar;D:\work\Java\jdk1.8.0_101\jre\lib\resources.jar;D:\work\Java\jdk1.8.0_101\jre\lib\rt.jar;D:\work\Java\Demo\chunmaster\chunjun-local-test\target\classes;D:\work\Java\Demo\chunmaster\chunjun-core\target\classes;C:\Users\Administrator\.m2\repository\cn\hutool\hutool-all\5.8.4\hutool-all-5.8.4.jar;C:\Users\Administrator\.m2\repository\com\alibaba\fastjson\1.2.83\fastjson-1.2.83.jar;C:\Users\Administrator\.m2\repository\com\google\guava\guava\19.0\guava-19.0.jar;C:\Users\Administrator\.m2\repository\com\google\code\gson\gson\2.7\gson-2.7.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-core\1.16.1\flink-core-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-annotations\1.16.1\flink-annotations-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-metrics-core\1.16.1\flink-metrics-core-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-shaded-asm-9\9.2-15.0\flink-shaded-asm-9-9.2-15.0.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-shaded-jackson\2.12.4-15.0\flink-shaded-jackson-2.12.4-15.0.jar;C:\Users\Administrator\.m2\repository\com\esotericsoftware\kryo\kryo\2.24.0\kryo-2.24.0.jar;C:\Users\Administrator\.m2\repository\com\esotericsoftware\minlog\minlog\1.2\minlog-1.2.jar;C:\Users\Administrator\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-shaded-guava\30.1.1-jre-15.0\flink-shaded-guava-30.1.1-jre-15.0.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-streaming-java\1.16.1\flink-streaming-java-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-file-sink-common\1.16.1\flink-file-sink-common-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-java\1.16.1\flink-java-1.16.1.jar;C:\Users\Administrator\.m2\repository\com\twitter\chill-java\0.7.6\chill-java-0.7.6.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-runtime\1.16.1\flink-runtime-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-rpc-core\1.16.1\flink-rpc-core-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-rpc-akka-loader\1.16.1\flink-rpc-akka-loader-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-queryable-state-client-java\1.16.1\flink-queryable-state-client-java-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-hadoop-fs\1.16.1\flink-hadoop-fs-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-shaded-netty\4.1.70.Final-15.0\flink-shaded-netty-4.1.70.Final-15.0.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-shaded-zookeeper-3\3.5.9-15.0\flink-shaded-zookeeper-3-3.5.9-15.0.jar;C:\Users\Administrator\.m2\repository\commons-cli\commons-cli\1.5.0\commons-cli-1.5.0.jar;C:\Users\Administrator\.m2\repository\org\javassist\javassist\3.24.0-GA\javassist-3.24.0-GA.jar;C:\Users\Administrator\.m2\repository\org\xerial\snappy\snappy-java\1.1.8.3\snappy-java-1.1.8.3.jar;C:\Users\Administrator\.m2\repository\org\lz4\lz4-java\1.8.0\lz4-java-1.8.0.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-table-api-java-bridge\1.16.1\flink-table-api-java-bridge-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-table-api-java\1.16.1\flink-table-api-java-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-table-api-bridge-base\1.16.1\flink-table-api-bridge-base-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-statebackend-rocksdb\1.16.1\flink-statebackend-rocksdb-1.16.1.jar;C:\Users\Administrator\.m2\repository\com\ververica\frocksdbjni\6.20.3-ververica-1.0\frocksdbjni-6.20.3-ververica-1.0.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-table-common\1.16.1\flink-table-common-1.16.1.jar;C:\Users\Administrator\.m2\repository\com\ibm\icu\icu4j\67.1\icu4j-67.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-table-runtime\1.16.1\flink-table-runtime-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-cep\1.16.1\flink-cep-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-table-planner_2.12\1.16.1\flink-table-planner_2.12-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\codehaus\janino\commons-compiler\3.0.11\commons-compiler-3.0.11.jar;C:\Users\Administrator\.m2\repository\org\codehaus\janino\janino\3.0.11\janino-3.0.11.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-scala_2.12\1.16.1\flink-scala_2.12-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\scala-lang\scala-reflect\2.12.7\scala-reflect-2.12.7.jar;C:\Users\Administrator\.m2\repository\org\scala-lang\scala-library\2.12.17\scala-library-2.12.17.jar;C:\Users\Administrator\.m2\repository\org\scala-lang\scala-compiler\2.12.7\scala-compiler-2.12.7.jar;C:\Users\Administrator\.m2\repository\org\scala-lang\modules\scala-xml_2.12\1.0.6\scala-xml_2.12-1.0.6.jar;C:\Users\Administrator\.m2\repository\com\twitter\chill_2.12\0.7.6\chill_2.12-0.7.6.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-hadoop-compatibility_2.12\1.16.1\flink-hadoop-compatibility_2.12-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-yarn\1.16.1\flink-yarn-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-clients\1.16.1\flink-clients-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-optimizer\1.16.1\flink-optimizer-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-queryable-state-runtime\1.16.1\flink-queryable-state-runtime-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\httpcomponents\httpclient\4.5.3\httpclient-4.5.3.jar;C:\Users\Administrator\.m2\repository\org\apache\httpcomponents\httpcore\4.4.6\httpcore-4.4.6.jar;C:\Users\Administrator\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.12.6\jackson-databind-2.12.6.jar;C:\Users\Administrator\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.12.6\jackson-annotations-2.12.6.jar;C:\Users\Administrator\.m2\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-lang3\3.12.0\commons-lang3-3.12.0.jar;C:\Users\Administrator\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-stream\target\classes;C:\Users\Administrator\.m2\repository\com\github\jsonzou\jmockdata\4.3.0\jmockdata-4.3.0.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-binlog\target\classes;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.parse\1.1.6\canal.parse-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.protocol\1.1.6\canal.protocol-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.meta\1.1.6\canal.meta-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.sink\1.1.6\canal.sink-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.store\1.1.6\canal.store-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.parse.dbsync\1.1.6\canal.parse.dbsync-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.parse.driver\1.1.6\canal.parse.driver-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\h2database\h2\1.4.196\h2-1.4.196.jar;C:\Users\Administrator\.m2\repository\com\lmax\disruptor\3.4.4\disruptor-3.4.4.jar;C:\Users\Administrator\.m2\repository\javax\annotation\javax.annotation-api\1.3.2\javax.annotation-api-1.3.2.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.common\1.1.6\canal.common-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\101tec\zkclient\0.10\zkclient-0.10.jar;C:\Users\Administrator\.m2\repository\com\alibaba\fastjson2\fastjson2\2.0.4\fastjson2-2.0.4.jar;C:\Users\Administrator\.m2\repository\org\springframework\spring-core\5.0.5.RELEASE\spring-core-5.0.5.RELEASE.jar;C:\Users\Administrator\.m2\repository\org\springframework\spring-jcl\5.0.5.RELEASE\spring-jcl-5.0.5.RELEASE.jar;C:\Users\Administrator\.m2\repository\com\alibaba\otter\canal.filter\1.1.6\canal.filter-1.1.6.jar;C:\Users\Administrator\.m2\repository\com\googlecode\aviator\aviator\2.2.1\aviator-2.2.1.jar;C:\Users\Administrator\.m2\repository\oro\oro\2.0.8\oro-2.0.8.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-iceberg\target\classes;C:\Users\Administrator\.m2\repository\org\apache\iceberg\iceberg-flink-runtime-1.16\1.2.1\iceberg-flink-runtime-1.16-1.2.1.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-client\3.3.0\hadoop-client-3.3.0.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-jobclient\3.3.0\hadoop-mapreduce-client-jobclient-3.3.0.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-common\3.3.0\hadoop-mapreduce-client-common-3.3.0.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-mongodb\target\classes;C:\Users\Administrator\.m2\repository\org\mongodb\mongodb-driver\3.12.11\mongodb-driver-3.12.11.jar;C:\Users\Administrator\.m2\repository\org\mongodb\mongodb-driver-core\3.12.11\mongodb-driver-core-3.12.11.jar;C:\Users\Administrator\.m2\repository\org\mongodb\bson\3.12.11\bson-3.12.11.jar;C:\Users\Administrator\.m2\repository\org\mongodb\mongodb-driver-async\3.12.11\mongodb-driver-async-3.12.11.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-emqx\target\classes;C:\Users\Administrator\.m2\repository\org\eclipse\paho\org.eclipse.paho.client.mqttv3\1.2.5\org.eclipse.paho.client.mqttv3-1.2.5.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-jdbc-base\target\classes;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-connector-jdbc\1.16.1\flink-connector-jdbc-1.16.1.jar;C:\Users\Administrator\.m2\repository\io\vertx\vertx-jdbc-client\3.9.7\vertx-jdbc-client-3.9.7.jar;C:\Users\Administrator\.m2\repository\io\vertx\vertx-sql-common\3.9.7\vertx-sql-common-3.9.7.jar;C:\Users\Administrator\.m2\repository\com\mchange\c3p0\0.9.5.4\c3p0-0.9.5.4.jar;C:\Users\Administrator\.m2\repository\com\mchange\mchange-commons-java\0.2.15\mchange-commons-java-0.2.15.jar;C:\Users\Administrator\.m2\repository\io\vertx\vertx-core\3.9.7\vertx-core-3.9.7.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-common\4.1.60.Final\netty-common-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-buffer\4.1.60.Final\netty-buffer-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-transport\4.1.60.Final\netty-transport-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-handler\4.1.60.Final\netty-handler-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-codec\4.1.60.Final\netty-codec-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-handler-proxy\4.1.60.Final\netty-handler-proxy-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-codec-socks\4.1.60.Final\netty-codec-socks-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-codec-http\4.1.60.Final\netty-codec-http-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-codec-http2\4.1.60.Final\netty-codec-http2-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-resolver\4.1.60.Final\netty-resolver-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-resolver-dns\4.1.60.Final\netty-resolver-dns-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\io\netty\netty-codec-dns\4.1.60.Final\netty-codec-dns-4.1.60.Final.jar;C:\Users\Administrator\.m2\repository\com\alibaba\druid\1.2.11\druid-1.2.11.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-mysql\target\classes;C:\Users\Administrator\.m2\repository\mysql\mysql-connector-java\5.1.49\mysql-connector-java-5.1.49.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-kafka\target\classes;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-connector-kafka\1.16.1\flink-connector-kafka-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-connector-base\1.16.1\flink-connector-base-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\kafka\kafka-clients\3.3.2\kafka-clients-3.3.2.jar;C:\Users\Administrator\.m2\repository\com\github\luben\zstd-jni\1.5.2-1\zstd-jni-1.5.2-1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-avro-confluent-registry\1.16.1\flink-avro-confluent-registry-1.16.1.jar;C:\Users\Administrator\.m2\repository\io\confluent\kafka-schema-registry-client\6.2.2\kafka-schema-registry-client-6.2.2.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-avro\1.16.1\flink-avro-1.16.1.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-kingbase\target\classes;C:\Users\Administrator\.m2\repository\com\kingbase8\kingbase8\8.2.0\kingbase8-8.2.0.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-ftp\target\classes;C:\Users\Administrator\.m2\repository\com\jcraft\jsch\0.1.55\jsch-0.1.55.jar;C:\Users\Administrator\.m2\repository\net\sourceforge\javacsv\javacsv\2.0\javacsv-2.0.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-csv\1.16.1\flink-csv-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\poi\poi-ooxml\4.1.2\poi-ooxml-4.1.2.jar;C:\Users\Administrator\.m2\repository\org\apache\poi\poi\4.1.2\poi-4.1.2.jar;C:\Users\Administrator\.m2\repository\com\zaxxer\SparseBitSet\1.2\SparseBitSet-1.2.jar;C:\Users\Administrator\.m2\repository\org\apache\poi\poi-ooxml-schemas\4.1.2\poi-ooxml-schemas-4.1.2.jar;C:\Users\Administrator\.m2\repository\org\apache\xmlbeans\xmlbeans\3.1.0\xmlbeans-3.1.0.jar;C:\Users\Administrator\.m2\repository\com\github\virtuald\curvesapi\1.06\curvesapi-1.06.jar;C:\Users\Administrator\.m2\repository\com\alibaba\easyexcel\3.0.1\easyexcel-3.0.1.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-csv\1.8\commons-csv-1.8.jar;C:\Users\Administrator\.m2\repository\cglib\cglib\3.3.0\cglib-3.3.0.jar;C:\Users\Administrator\.m2\repository\org\ow2\asm\asm\7.1\asm-7.1.jar;C:\Users\Administrator\.m2\repository\org\ehcache\ehcache\3.8.1\ehcache-3.8.1.jar;C:\Users\Administrator\.m2\repository\org\glassfish\jaxb\jaxb-runtime\2.3.1\jaxb-runtime-2.3.1.jar;C:\Users\Administrator\.m2\repository\org\glassfish\jaxb\txw2\2.3.1\txw2-2.3.1.jar;C:\Users\Administrator\.m2\repository\com\sun\istack\istack-commons-runtime\3.0.7\istack-commons-runtime-3.0.7.jar;C:\Users\Administrator\.m2\repository\org\jvnet\staxex\stax-ex\1.8\stax-ex-1.8.jar;C:\Users\Administrator\.m2\repository\com\sun\xml\fastinfoset\FastInfoset\1.2.15\FastInfoset-1.2.15.jar;C:\Users\Administrator\.m2\repository\javax\activation\javax.activation-api\1.2.0\javax.activation-api-1.2.0.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-json\1.16.1\flink-json-1.16.1.jar;C:\Users\Administrator\.m2\repository\org\apache\flink\flink-shaded-force-shading\15.0\flink-shaded-force-shading-15.0.jar;C:\Users\Administrator\.m2\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;D:\work\Java\Demo\chunmaster\chunjun-connectors\chunjun-connector-oceanbase\target\classes;C:\Users\Administrator\.m2\repository\com\oceanbase\oceanbase-client\2.4.0\oceanbase-client-2.4.0.jar;C:\Users\Administrator\.m2\repository\org\apache\avro\avro\1.10.1\avro-1.10.1.jar;C:\Users\Administrator\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.12.6\jackson-core-2.12.6.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-compress\1.20\commons-compress-1.20.jar;C:\Users\Administrator\.m2\repository\org\apache\parquet\parquet-avro\1.12.2\parquet-avro-1.12.2.jar;C:\Users\Administrator\.m2\repository\org\apache\parquet\parquet-column\1.12.2\parquet-column-1.12.2.jar;C:\Users\Administrator\.m2\repository\org\apache\parquet\parquet-common\1.12.2\parquet-common-1.12.2.jar;C:\Users\Administrator\.m2\repository\org\apache\yetus\audience-annotations\0.12.0\audience-annotations-0.12.0.jar;C:\Users\Administrator\.m2\repository\org\apache\parquet\parquet-encoding\1.12.2\parquet-encoding-1.12.2.jar;C:\Users\Administrator\.m2\repository\org\apache\parquet\parquet-hadoop\1.12.2\parquet-hadoop-1.12.2.jar;C:\Users\Administrator\.m2\repository\org\apache\parquet\parquet-jackson\1.12.2\parquet-jackson-1.12.2.jar;C:\Users\Administrator\.m2\repository\commons-pool\commons-pool\1.6\commons-pool-1.6.jar;C:\Users\Administrator\.m2\repository\org\apache\parquet\parquet-format-structures\1.12.2\parquet-format-structures-1.12.2.jar;C:\Users\Administrator\.m2\repository\org\slf4j\slf4j-api\1.7.36\slf4j-api-1.7.36.jar;C:\Users\Administrator\.m2\repository\org\apache\logging\log4j\log4j-slf4j-impl\2.17.1\log4j-slf4j-impl-2.17.1.jar;C:\Users\Administrator\.m2\repository\org\apache\logging\log4j\log4j-api\2.17.1\log4j-api-2.17.1.jar;C:\Users\Administrator\.m2\repository\org\apache\logging\log4j\log4j-core\2.17.1\log4j-core-2.17.1.jar;C:\Users\Administrator\.m2\repository\org\slf4j\jcl-over-slf4j\1.7.36\jcl-over-slf4j-1.7.36.jar;C:\Users\Administrator\.m2\repository\org\apache\logging\log4j\log4j-1.2-api\2.17.1\log4j-1.2-api-2.17.1.jar;C:\Users\Administrator\.m2\repository\org\projectlombok\lombok\1.18.24\lombok-1.18.24.jar;C:\Users\Administrator\.m2\repository\org\objenesis\objenesis\2.6\objenesis-2.6.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-annotations\2.8.5\hadoop-annotations-2.8.5.jar;D:\work\Java\jdk1.8.0_101\lib\tools.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;C:\Users\Administrator\.m2\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;C:\Users\Administrator\.m2\repository\commons-io\commons-io\2.11.0\commons-io-2.11.0.jar;C:\Users\Administrator\.m2\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;C:\Users\Administrator\.m2\repository\javax\xml\stream\stax-api\1.0-2\stax-api-1.0-2.jar;C:\Users\Administrator\.m2\repository\javax\activation\activation\1.1\activation-1.1.jar;C:\Users\Administrator\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-hdfs-client\2.8.5\hadoop-hdfs-client-2.8.5.jar;C:\Users\Administrator\.m2\repository\com\squareup\okhttp\okhttp\2.4.0\okhttp-2.4.0.jar;C:\Users\Administrator\.m2\repository\com\squareup\okio\okio\1.4.0\okio-1.4.0.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-collections4\4.4\commons-collections4-4.4.jar com.dtstack.chunjun.local.test.LocalTest
2023-06-09 00:16:17,141 INFO  com.dtstack.chunjun.Main - ------------program params-------------------------
2023-06-09 00:16:17,154 INFO  com.dtstack.chunjun.Main - -mode
2023-06-09 00:16:17,154 INFO  com.dtstack.chunjun.Main - localTest
2023-06-09 00:16:17,154 INFO  com.dtstack.chunjun.Main - -jobType
2023-06-09 00:16:17,154 INFO  com.dtstack.chunjun.Main - sync
2023-06-09 00:16:17,154 INFO  com.dtstack.chunjun.Main - -job
2023-06-09 00:16:17,154 INFO  com.dtstack.chunjun.Main - %7B%0A++%22job%22%3A+%7B%0A++++%22content%22%3A+%5B%0A++++++%7B%0A++++++++%22reader%22%3A+%7B%0A++++++++++%22parameter%22%3A+%7B%0A++++++++++++%22column%22%3A+%5B%0A++++++++++++++%7B%0A++++++++++++++++%22name%22%3A+%22id%22%2C%0A++++++++++++++++%22type%22%3A+%22id%22%0A++++++++++++++%7D%0A++++++++++++%5D%2C%0A++++++++++++%22sliceRecordCount%22%3A+%5B%2210%22%5D%2C%0A++++++++++++%22permitsPerSecond%22%3A+1%0A++++++++++%7D%2C%0A++++++++++%22name%22%3A+%22streamreader%22%0A++++++++%7D%2C%0A++++++++%22writer%22%3A+%7B%0A++++++++++%22parameter%22%3A+%7B%0A++++++++++++%22column%22%3A+%5B+%7B%0A++++++++++++++%22name%22%3A+%22id%22%2C%0A++++++++++++++%22index%22%3A+0%2C%0A++++++++++++++%22resourceName%22%3A+%22%22%2C%0A++++++++++++++%22type%22%3A+%22INT%22%2C%0A++++++++++++++%22key%22%3A+%22id%22%0A++++++++++++%7D%5D%2C%0A++++++++++++%22uri%22%3A+%22thrift%3A%2F%2F192.168.1.181%3A9083%22%2C%0A++++++++++++%22warehouse%22%3A+%22hdfs%3A%2F%2F192.168.1.180%3A8020%2F%2Fuser%2Fhive%2Fwarehouse%22%2C%0A++++++++++++%22database%22%3A+%22iceberg_test%22%2C%0A++++++++++++%22table%22%3A+%22test%22%2C%0A++++++++++++%22hadoopConfig%22%3A+%7B%0A++++++++++++++%22fs.defaultFS%22%3A%22hdfs%3A%2F%2F192.168.1.180%3A9000%22%2C%0A++++++++++++++%22dfs.nameservices%22%3A%22192.168.1.180%22%2C%0A++++++++++++++%22dfs.ha.namenodes.ns1%22%3A%22192.168.1.180%22%2C%0A++++++++++++++%22dfs.namenode.rpc-address.ns1.nn1%22%3A%22192.168.1.180%3A9000%22%2C%0A++++++++++++++%22dfs.client.failover.proxy.provider.ns1%22%3A+%22org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider%22%0A++++++++++++%7D%2C%0A++++++++++++%22writeMode%22%3A+%22overwrite%22%0A++++++++++%7D%2C%0A++++++++++%22name%22%3A+%22icebergwriter%22%0A++++++++%7D%0A++++++%7D%0A++++%5D%2C%0A++++%22setting%22%3A+%7B%0A++++++%22errorLimit%22%3A+%7B%0A++++++++%22record%22%3A+100%0A++++++%7D%2C%0A++++++%22speed%22%3A+%7B%0A++++++++%22bytes%22%3A+0%2C%0A++++++++%22channel%22%3A+1%2C%0A++++++++%22readerChannel%22%3A+1%2C%0A++++++++%22writerChannel%22%3A+1%0A++++++%7D%0A++++%7D%0A++%7D%0A%7D%0A
2023-06-09 00:16:17,155 INFO  com.dtstack.chunjun.Main - {}
2023-06-09 00:16:17,155 INFO  com.dtstack.chunjun.Main - -------------------------------------------
2023-06-09 00:16:20,252 INFO  com.dtstack.chunjun.Main - Register to table configuration:{table.dynamic-table-options.enabled=true, pipeline.name=Flink_Job}
2023-06-09 00:16:20,336 INFO  com.dtstack.chunjun.util.PluginUtil - ChunJun executionMode: localTest
2023-06-09 00:16:20,337 INFO  com.dtstack.chunjun.util.PluginUtil - ChunJun reset pipeline.jars: []
2023-06-09 00:16:20,344 INFO  com.dtstack.chunjun.classloader.ClassLoaderManager - jarUrl: create ClassLoad successful...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
    at com.dtstack.chunjun.connector.iceberg.sink.IcebergSinkFactory.buildTableLoader(IcebergSinkFactory.java:101)
    at com.dtstack.chunjun.connector.iceberg.sink.IcebergSinkFactory.createSink(IcebergSinkFactory.java:86)
    at com.dtstack.chunjun.Main.exeSyncJob(Main.java:236)
    at com.dtstack.chunjun.Main.main(Main.java:138)
    at com.dtstack.chunjun.local.test.LocalTest.main(LocalTest.java:172)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 5 more

依赖包也导入正常，POM的排除已经注释。打包生成好的文件在有flink hdp的测试环境是可以正常的跑通。目前调试需要Debug非常不方便，只能在jar包里面打一些提示信息，在log日志里面看对应信息 image

Code of Conduct

ll076110 commented 1 year ago

在localtest模块中的pom文件里添加hadoop-common依赖试试

liuzx8888 commented 1 year ago

@ll076110 可以了，十分感谢！！！！！