search

apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.3k stars 1.17k forks source link

when I upgrap hive to 3.0.0, rebuild links, query in Scripts an warn occurs :java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.CommandNeedRetryException #338

Closed Rjiliang closed 4 years ago

Rjiliang commented 4 years ago

Describe the bug when I upgrap hive to 3.0.0, rebuild links, query in Scripts an warn occurs: WARN [hiveEngineEngine-Thread-2] com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$ 84 apply - java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.CommandNeedRetryException

To Reproduce upgrade hadoop version (HDP-3.1) hadoop-3.1.1 hive-3.0.0 rebulid DSS and links, Query in Scripts :select * from demo.dept limit 100

Screenshots error log: 44385: 2020-03-31 23:10:38-957 INFO [hiveEngineEngine-Thread-2] com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor.executeLine(HiveEngineExecutor.scala:108) executeLine - hive client begins to run hql code: 44385: select * from demo.dept limit 100 44385: 2020-03-31 23:10:38-994 WARN [hiveEngineEngine-Thread-2] com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$ com.webank.wedatasphere.linkis.common.utils.Utils$$anonfun$tryAndWarn$1.apply(Utils.scala:84) apply - java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.CommandNeedRetryException 44385: at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_201] 44385: at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_201] 44385: at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_201] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$$anonfun$2.apply(HiveEngineExecutor.scala:374) ~[linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$$anonfun$2.apply(HiveEngineExecutor.scala:374) ~[linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:74) [linkis-common-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$.(HiveEngineExecutor.scala:372) [linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$.(HiveEngineExecutor.scala) [linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor$$anon$1.run(HiveEngineExecutor.scala:124) [linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor$$anon$1.run(HiveEngineExecutor.scala:121) [linkis-hive-engine-0.9.3.jar:?] 44385: at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_201] 44385: at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_201] 44385: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) [hadoop-common-3.1.1.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor.executeLine(HiveEngineExecutor.scala:121) [linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9$$anonfun$apply$10.apply(EngineExecutor.scala:141) [linkis-ujes-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9$$anonfun$apply$10.apply(EngineExecutor.scala:140) [linkis-ujes-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9.apply(EngineExecutor.scala:141) [linkis-ujes-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9.apply(EngineExecutor.scala:136) [linkis-ujes-engine-0.9.3.jar:?] 44385: at scala.collection.immutable.Range.foreach(Range.scala:160) [scala-library-2.11.8.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1.apply(EngineExecutor.scala:136) [linkis-ujes-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1.apply(EngineExecutor.scala:118) [linkis-ujes-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:62) [linkis-common-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.scheduler.executer.AbstractExecutor.ensureIdle(AbstractExecutor.scala:60) [linkis-scheduler-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.scheduler.executer.AbstractExecutor.ensureIdle(AbstractExecutor.scala:54) [linkis-scheduler-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor.ensureOp$1(EngineExecutor.scala:117) [linkis-ujes-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor.execute(EngineExecutor.scala:118) [linkis-ujes-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$3.apply(Job.scala:254) [linkis-scheduler-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$3.apply(Job.scala:254) [linkis-scheduler-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.scheduler.queue.Job.run(Job.scala:254) [linkis-scheduler-0.9.3.jar:?] 44385: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201] 44385: at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201] 44385: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201] 44385: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201] 44385: at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

44385: Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14 Table not found 'dept' 44385: at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2118) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2038) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11954) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12049) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:334) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:644) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1684) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1631) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1626) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214) ~[hive-exec-3.0.0.jar:3.0.0] 44385: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201] 44385: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_201] 44385: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201] 44385: at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$$anonfun$run$3.apply(HiveEngineExecutor.scala:348) ~[linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$$anonfun$run$3.apply(HiveEngineExecutor.scala:348) ~[linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) ~[linkis-common-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:74) ~[linkis-common-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy.run(HiveEngineExecutor.scala:347) ~[linkis-hive-engine-0.9.3.jar:?] 44385: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor$$anon$1.run(HiveEngineExecutor.scala:137) ~[linkis-hive-engine-0.9.3.jar:?] 44385: ... 27 more

chaogefeng commented 4 years ago

@renrr This error does not affect the execution of the task. If the table does not exist, it is likely that the hive conf configuration is not configured properly.

Rjiliang commented 4 years ago

@renrr This error does not affect the execution of the task. If the table does not exist, it is likely that the hive conf configuration is not configured properly.

@chaogefeng Thank you for your reply. Which path should I check for hive configuration