search

jaykelin / clickhouse-hdfs-loader

loading hdfs data to clickhouse
MIT License
73 stars 42 forks source link

JDBC java.lang.NoSuchFieldError: INSTANCE #3

Open HadoopChen opened 6 years ago

HadoopChen commented 6 years ago

命令

hadoop_lt jar clickhouse-hdfs-loader-2.0.3-jar-with-dependencies.jar com.kugou.loader.clickhouse.ClickhouseHdfsLoader \ -Dmapreduce.job.queuename=root.cj.default \ -i orc \ --connect jdbc:clickhouse:/1.2.3.4:8123/default \ --table distributed_tag \ --dt 2018-06-23 \ --export-dir /user/cj/hive_db/cj.db/tag2

错误

18/06/23 17:47:19 INFO clickhouse.ClickhouseClientHolder: Clickhouse Loader : get clickhouse client[jdbc:clickhouse://1.2.3.4:8123/default] for user=null 18/06/23 17:47:19 INFO clickhouse.ClickHouseDriver: Driver registered 18/06/23 17:47:19 INFO clickhouse.ClickHouseDriver: Creating connection Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:52) at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:56) at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:46) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:82) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:95) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:104) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:62) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.(PoolingHttpClientConnectionManager.java:572) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:174) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:158) at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.getConnectionManager(ClickHouseHttpClientBuilder.java:77) at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.buildClient(ClickHouseHttpClientBuilder.java:54) at ru.yandex.clickhouse.ClickHouseConnectionImpl.(ClickHouseConnectionImpl.java:71) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:58) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:50) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:32) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:233) at com.kugou.loader.clickhouse.ClickhouseClient.(ClickhouseClient.java:47) at com.kugou.loader.clickhouse.ClickhouseClient.(ClickhouseClient.java:39) at com.kugou.loader.clickhouse.ClickhouseClientHolder.getClickhouseClient(ClickhouseClientHolder.java:49) at com.kugou.loader.clickhouse.ClickhouseClientHolder.getClickhouseClient(ClickhouseClientHolder.java:57) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.initClickhouseParameters(ClickhouseHdfsLoader.java:223) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.run(ClickhouseHdfsLoader.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.main(ClickhouseHdfsLoader.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

排查

telnet 1.2.3.4 8123 ==> OK default数据库存在、分布式表distributed_tag也存在； clickhouseServer正常，无报错日志；

求助！！！

jaykelin commented 6 years ago

缺少参数：--username xxx --password xxx

HadoopChen commented 6 years ago

哦哦 感谢！

HadoopChen commented 6 years ago

貌似依然不行

CH本机可以通过clickhouse-client --user default --password default登入CH；但使用JDBC依然不行；报同样的错；均是默认配置，只是给default用户加了一个明文的密码。

<password>default</password>
<networks incl="networks" replace="replace">
                <ip>::/0</ip>
</networks>
<profile>default</profile>
<quota>default</quota>
 </default>

命令

hadoop_lt jar clickhouse-hdfs-loader-2.0.3-jar-with-dependencies.jar com.kugou.loader.clickhouse.ClickhouseHdfsLoader -Dmapreduce.job.queuename=root.cj.default -i orc --connect jdbc:clickhouse:/1.2.3.4:8123/default --username default --password default --table distributed_tag --dt 2018-06-23 --export-dir /user/cj/hive_db/cj.db/tag2

错误

18/06/26 10:35:06 INFO clickhouse.ClickhouseClientHolder: Clickhouse Loader : get clickhouse client[jdbc:clickhouse://1.2.3.4:8123/default] for user=default 18/06/26 10:35:06 INFO clickhouse.ClickHouseDriver: Driver registered 18/06/26 10:35:06 INFO clickhouse.ClickHouseDriver: Creating connection Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:52) at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:56) at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:46) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:82) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:95) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:104) at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:62) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.(PoolingHttpClientConnectionManager.java:572) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:174) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:158) at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.getConnectionManager(ClickHouseHttpClientBuilder.java:77) at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.buildClient(ClickHouseHttpClientBuilder.java:54) at ru.yandex.clickhouse.ClickHouseConnectionImpl.(ClickHouseConnectionImpl.java:71) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:58) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:50) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:32) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at com.kugou.loader.clickhouse.ClickhouseClient.(ClickhouseClient.java:45) at com.kugou.loader.clickhouse.ClickhouseClientHolder.getClickhouseClient(ClickhouseClientHolder.java:65) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.initClickhouseParameters(ClickhouseHdfsLoader.java:223) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.run(ClickhouseHdfsLoader.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.main(ClickhouseHdfsLoader.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

jaykelin commented 6 years ago

@HadoopChen http客户端的jar包是不是有冲突

HadoopChen commented 6 years ago

image

咋搞，把jdbc用4.5.5改一遍。旧的类都替换掉 ？

jaykelin commented 6 years ago

试试用这个jdbc客户端 https://github.com/jaykelin/clickhouse-jdbc/tree/0.1-SNAPSHOT

panda403 commented 5 years ago

username 和password是clickhouse的用户名密码吗？我默认单机装的没有用户名密码怎么弄

slimtom95 commented 5 years ago

use shade relocation to solve this conflicts ref to

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.1</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <relocations>
                    <relocation>
                        <pattern>org.apache.http</pattern>
                        <shadedPattern>shaded.org.apache.http</shadedPattern>
                    </relocation>
                </relocations>
            </configuration>
        </execution>
    </executions>
</plugin>
Balro commented 4 years ago

将冲突包设置为compile，提交作业前设置变量调整hadoop类路径加载顺序，后提交作业。 export HADOOP_CLASSPATH=xxx.jar export HADOOP_USER_CLASSPATH_FIRST=true 提交作业时，再加参 -Dmapreduce.job.user.classpath.first=true