search

cindysz110 / blog

8 stars 1 forks source link

[Hadoop] Sqoop 从MySQL导入数据到hive表报错“Could not load org.apache.hadoop.hive.conf.HiveConf”解决 #22

Open cindygl opened 6 years ago

cindygl commented 6 years ago

Sqoop从MySQL导入数据到Hive表报错：ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.

[hadoop@hadoop01 ~]$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/db_1 --username root --password root --table city_info --hive-table city_info --fields-terminated-by ',' --hive-overwrite;
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/26 22:39:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/06/26 22:39:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/26 22:39:46 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 22:39:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 22:39:48 INFO hive.HiveImport: Loading uploaded data into Hive
18/06/26 22:39:48 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
18/06/26 22:39:48 ERROR tool.CreateHiveTableTool: Encountered IOException running create table job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
    at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
    at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
    at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
    at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
    at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
    at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
    ... 11 more

[hadoop@hadoop01 ~]$

解决：

# 查看HiveConf.class类是否存在
[hadoop@hadoop01 app]$ cd apache-hive-1.1.0-cdh5.7.0-bin/lib/
[hadoop@hadoop01 lib]$ jar tf hive-common-1.1.0-cdh5.7.0.jar  | grep HiveConf.class
org/apache/hadoop/hive/conf/HiveConf.class     
[hadoop@hadoop01 lib]$ 
查看到HiveConf.class类明明存在，只是环境没有找到。

修改环境配置，将hive的lib添加HADOOP_CLASSPATH中

[root@hadoop01 ~]# vi /etc/profile
export HADOOP_CLASSPATH=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/*
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/hadoop/app/apache-hive-1.1.0-cdh5.7.0-bin/lib/*

再次执行，执行成功。

[hadoop@hadoop01 ~]$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/db_1 --username root --password root --table city_info --hive-table city_info --fields-terminated-by ',' --hive-overwrite;
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/06/26 23:19:58 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/06/26 23:19:58 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/06/26 23:19:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/06/26 23:19:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 23:19:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `city_info` AS t LIMIT 1
18/06/26 23:20:00 INFO hive.HiveImport: Loading uploaded data into Hive

Logging initialized using configuration in jar:file:/home/hadoop/app/apache-hive-1.1.0-cdh5.7.0-bin/lib/hive-common-1.1.0-cdh5.7.0.jar!/hive-log4j.properties
OK
Time taken: 1.427 seconds
[hadoop@hadoop01 ~]$