java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager

sdlcwangsong commented 6 years ago

当我执行任务时报错：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager at ru.yandex.clickhouse.ClickHouseConnectionImpl.(ClickHouseConnectionImpl.java:68) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:58) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:50) at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:32) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at com.kugou.loader.clickhouse.ClickhouseClient.(ClickhouseClient.java:45) at com.kugou.loader.clickhouse.ClickhouseClientHolder.getClickhouseClient(ClickhouseClientHolder.java:65) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.initClickhouseParameters(ClickhouseHdfsLoader.java:223) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.run(ClickhouseHdfsLoader.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.main(ClickhouseHdfsLoader.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.HttpClientConnectionManager at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 18 more

我看到有人说是httpclient版本问题，我修改pom.xml中版本为4.5.5还是不行:

org.apache.httpcomponents httpclient 4.5.5 provided ru.yandex.clickhouse clickhouse-jdbc 0.1.40

请问是什么原因呢？求解答下

jaykelin commented 6 years ago

我现在用的jdbc： https://github.com/jaykelin/clickhouse-jdbc/tree/0.1-SNAPSHOT 用的httpclient版本：

       <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.5</version>
            <scope>provided</scope>
        </dependency>

sdlcwangsong commented 6 years ago

@jaykelin 你好，我按你说的设置pom.xml,还是报同样的错误，麻烦看下我的pom.xml与你的有什么不同：

`<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

4.0.0

<groupId>com.kugou</groupId>
<artifactId>clickhouse-hdfs-loader</artifactId>
<version>2.0.3</version>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <hadoop.version>2.7.2</hadoop.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
        <!--<exclusions>
            <exclusion>
                <groupId>org.apache.httpcomponents</groupId>
                <artifactId>httpclient</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.apache.httpcomponents</groupId>
                <artifactId>httpcore</artifactId>
            </exclusion>
        </exclusions>-->
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-common</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-core</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.orc</groupId>
        <artifactId>orc-mapreduce</artifactId>
        <version>1.2.1</version>
    </dependency>

    <dependency>
        <groupId>args4j</groupId>
        <artifactId>args4j</artifactId>
        <version>2.33</version>
    </dependency>

    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.2.5</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpmime</artifactId>
        <version>4.5.2</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>ru.yandex.clickhouse</groupId>
        <artifactId>clickhouse-jdbc</artifactId>
        <version>0.1.40</version>
    </dependency>

    <dependency>
        <groupId>io.dropwizard.metrics</groupId>
        <artifactId>metrics-core</artifactId>
        <version>3.1.2</version>
    </dependency>

    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.2</version>
            <configuration>
                <source>1.7</source>
                <target>1.7</target>
                <!--<showDeprecation>true</showDeprecation>
                <showWarnings>true</showWarnings>
                <fork>true</fork>-->
                <encoding>UTF-8</encoding>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.5</version>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
            </configuration>
        </plugin>
    </plugins>
</build>

`

jaykelin commented 6 years ago

建议使用我提供的那个jdbc

<dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.2.5</version>
        <scope>provided</scope>
    </dependency>

这个jar的版本是没有 HttpClientConnectionManager 的

sdlcwangsong commented 6 years ago

@jaykelin 我用的就是你说的那个jdbc，是不是还需要修改代码呢

`

ru.yandex.clickhouse

<artifactId>clickhouse-jdbc</artifactId>
<version>0.1.40</version>

`

jaykelin commented 6 years ago

可以下载jdbc的代码，install到本地mvn仓库，完成jar依赖

fibersel commented 4 years ago

Can you please help me? I am getting this error too, I tried a lot of pom.xml configurations. Can you please show valid pom.xml and commands for jar building?

jaykelin commented 4 years ago

@fibersel clone clickhouse-jdbc from https://github.com/jaykelin/clickhouse-jdbc/tree/0.1-SNAPSHOT cd ...../../clickhouse-jdbc && make install cd .../clickhouse-hdfs-loader && mvn clean package assembly:assembly -DskipTests

fibersel commented 4 years ago

sorry, but i can't do make install, there is no any Makefile

jaykelin commented 4 years ago

my foult. it's mvn install

fibersel commented 4 years ago

Here is full step-by-step guide for jar building:

git clone https://github.com/jaykelin/clickhouse-jdbc.git
git clone https://github.com/jaykelin/clickhouse-hdfs-loader.git
cd clickhouse-jdbs
git checkout 0.1-SNAPSHOT
mvn install -DskipTests

(tests run on clickhouse instance, running on your machine at port 8123, so we skip it)

cd ../clickhouse-hdfs-loader 
mvn clean package assembly:assembly -DskipTests

after these commands you will get jars at "target" folder So you can submit hadoop job:

hadoop jar target/clickhouse-hdfs-loader-2.1.2-jar-with-dependencies.jar com.kugou.loader.clickhouse.ClickhouseHdfsLoader ...

@jaykelin , put these instruction to README please, it will make our life easier :)

jaykelin commented 4 years ago

@fibersel Thanks for the advice

jaykelin / clickhouse-hdfs-loader

java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager #4