Closed ittechblog closed 3 years ago
这里是为了做Hive的兼容,并不是问题所在,这个提示后面会改掉,https://github.com/WeBankFinTech/Linkis/pull/349/commits/bc48e799c220a8dccd8a69c7bded6937c97a6f0a
具体原因需要看一下其他的 Error 日志
具体原因需要看一下其他的 Error 日志
2020-06-05 10:46:44,858 INFO (background-preinit) INFO Version - HV000001: Hibernate Validator 5.1.2.Final 2020-06-05 10:46:45,140 INFO (main) INFO AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@4b6579e8: startup date [Fri Jun 05 10:46:45 CST 2020]; root of context hierarchy 2020-06-05 10:46:45,371 INFO (main) INFO AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring 2020-06-05 10:46:45,405 INFO (main) INFO PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$a94202bb] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
. _ _ /\ / '_ () \ \ \ \ ( ( )\ | ' | '| | ' \/ ` | \ \ \ \ \/ _)| |)| | | | | || (| | ) ) ) ) ' |__| .|| ||| |\, | / / / / =========|_|==============|__/=//// :: Spring Boot :: (v2.0.3.RELEASE)
2020-06-05 10:46:45,653 INFO (main) INFO ConfigServicePropertySourceLocator - Fetching config from server at : http://localhost:8888
2020-06-05 10:46:45,749 INFO (main) INFO ConfigServicePropertySourceLocator - Connect Timeout Exception on Url - http://localhost:8888. Will be trying the next url if available
2020-06-05 10:46:45,749 WARN (main) WARN ConfigServicePropertySourceLocator - Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/hiveEngineManager/default": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
2020-06-05 10:46:45,751 INFO (main) INFO DataWorkCloudApplication - No active profile set, falling back to default profiles: default
2020-06-05 10:46:45,770 INFO (main) INFO DataWorkCloudApplication - add config from config server...
2020-06-05 10:46:45,770 INFO (main) INFO DataWorkCloudApplication - initialize DataWorkCloud spring application...
2020-06-05 10:46:45,773 INFO (main) INFO AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@444548a0: startup date [Fri Jun 05 10:46:45 CST 2020]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@4b6579e8
2020-06-05 10:46:46,479 INFO (main) INFO DefaultListableBeanFactory - Overriding bean definition for bean 'resources' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=engineManagerSpringConfiguration; factoryMethodName=createResource; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/webank/wedatasphere/linkis/enginemanager/impl/EngineManagerSpringConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=hiveEngineManagerSpringConfiguration; factoryMethodName=createResource; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/webank/wedatasphere/linkis/enginemanager/hive/conf/HiveEngineManagerSpringConfiguration.class]]
2020-06-05 10:46:46,479 INFO (main) INFO DefaultListableBeanFactory - Overriding bean definition for bean 'hooks' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=engineManagerSpringConfiguration; factoryMethodName=createEngineHook; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/webank/wedatasphere/linkis/enginemanager/impl/EngineManagerSpringConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=hiveEngineManagerSpringConfiguration; factoryMethodName=createEngineHook; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/webank/wedatasphere/linkis/enginemanager/hive/conf/HiveEngineManagerSpringConfiguration.class]]
2020-06-05 10:46:46,999 INFO (main) INFO GenericScope - BeanFactory id=572d531d-6d22-33ef-9a3b-6fb666d9b9e3
2020-06-05 10:46:47,015 INFO (main) INFO AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2020-06-05 10:46:47,243 INFO (main) INFO PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$a94202bb] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-06-05 10:46:47.436 INFO [main] org.eclipse.jetty.util.log 193 initialized - Logging initialized @3678ms to org.eclipse.jetty.util.log.Slf4jLog
2020-06-05 10:46:47,545 INFO (main) INFO JettyServletWebServerFactory - Server initialized with port: 10099
2020-06-05 10:46:47.575 INFO [main] org.eclipse.jetty.server.Server 374 doStart - jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_181-b13
2020-06-05 10:46:47.687 INFO [main] org.eclipse.jetty.server.session 365 doStart - DefaultSessionIdManager workerName=node0
2020-06-05 10:46:47.687 INFO [main] org.eclipse.jetty.server.session 370 doStart - No SessionScavenger set, using defaults
2020-06-05 10:46:47.689 INFO [main] org.eclipse.jetty.server.session 149 startScavenging - node0 Scavenging every 660000ms
2020-06-05 10:46:47.694 INFO [main] org.eclipse.jetty.server.handler.ContextHandler.application 2318 log - Initializing Spring embedded WebApplicationContext
2020-06-05 10:46:47,694 INFO (main) INFO ContextLoader - Root WebApplicationContext: initialization completed in 1921 ms
2020-06-05 10:46:47.910 WARN [main] com.netflix.config.sources.URLConfigurationSource 121
这是完整日志,请帮忙查看一下。
创建 hbase 链接失败了,可以看看 hbase 相关的配置是不是有问题,Hbase Zookeeper 地址有没有配置?
创建 hbase 链接失败了,可以看看 hbase 相关的配置是不是有问题,Hbase Zookeeper 地址有没有配置?
这台机器上也安装了hdfs,hbase,hive,spark,直接通过hbase命令和hive命令查询外部表也是没问题的。只有通过dss和links查询hive外部表才有问题,直接查询hive内部表也是没问题的。
把日志级别调成 debug,可以看一下具体的错误信息
创建 hbase 链接失败了,可以看看 hbase 相关的配置是不是有问题,Hbase Zookeeper 地址有没有配置?
这台机器上也安装了hdfs,hbase,hive,spark,直接通过hbase命令和hive命令查询外部表也是没问题的。只有通过dss和links查询hive外部表才有问题,直接查询hive内部表也是没问题的。
这个问题最后怎么解决的?求教
错误信息如下: 59469: 2020-06-05 15:03:29-915 WARN [hiveEngineEngine-Thread-2] com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$ com.webank.wedatasphere.linkis.common.utils.Utils$$anonfun$tryAndWarn$1.apply(Utils.scala:84) apply - java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.IDriver 59469: at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_181] 59469: at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_181] 59469: at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181] 59469: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$$anonfun$3.apply(HiveEngineExecutor.scala:378) ~[linkis-hive-engine-0.9.3.jar:?] 59469: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$$anonfun$3.apply(HiveEngineExecutor.scala:378) ~[linkis-hive-engine-0.9.3.jar:?] 59469: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?] 59469: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:74) [linkis-common-0.9.3.jar:?] 59469: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$.(HiveEngineExecutor.scala:376) [linkis-hive-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveDriverProxy$.(HiveEngineExecutor.scala) [linkis-hive-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor$$anon$1.run(HiveEngineExecutor.scala:124) [linkis-hive-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor$$anon$1.run(HiveEngineExecutor.scala:121) [linkis-hive-engine-0.9.3.jar:?]
59469: at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_181]
59469: at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_181]
59469: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.2.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.hive.executor.HiveEngineExecutor.executeLine(HiveEngineExecutor.scala:121) [linkis-hive-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9$$anonfun$apply$10.apply(EngineExecutor.scala:141) [linkis-ujes-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9$$anonfun$apply$10.apply(EngineExecutor.scala:140) [linkis-ujes-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9.apply(EngineExecutor.scala:141) [linkis-ujes-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9.apply(EngineExecutor.scala:136) [linkis-ujes-engine-0.9.3.jar:?]
59469: at scala.collection.immutable.Range.foreach(Range.scala:160) [scala-library-2.11.8.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1.apply(EngineExecutor.scala:136) [linkis-ujes-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1.apply(EngineExecutor.scala:118) [linkis-ujes-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:62) [linkis-common-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.scheduler.executer.AbstractExecutor.ensureIdle(AbstractExecutor.scala:60) [linkis-scheduler-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.scheduler.executer.AbstractExecutor.ensureIdle(AbstractExecutor.scala:54) [linkis-scheduler-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor.ensureOp$1(EngineExecutor.scala:117) [linkis-ujes-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor.execute(EngineExecutor.scala:118) [linkis-ujes-engine-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$3.apply(Job.scala:254) [linkis-scheduler-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$3.apply(Job.scala:254) [linkis-scheduler-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?]
59469: at com.webank.wedatasphere.linkis.scheduler.queue.Job.run(Job.scala:254) [linkis-scheduler-0.9.3.jar:?]
59469: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
59469: at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
59469: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
59469: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
59469: at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
59469:
org.apache.hadoop.hive.ql.IDriver在hive-exec.jar里并没有这个类只有org.apache.hadoop.hive.ql.Driver