search

WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
https://github.com/WeBankFinTech/DataSphereStudio-Doc
Apache License 2.0
3.09k stars 1k forks source link

DSS0.9.1 with Linkis0.9.4 not compatible with HDP3.1 #311

Open lordk911 opened 3 years ago

lordk911 commented 3 years ago

When I run HSQL I got ERROR:

2021-04-25 09:44:40.044 INFO Your job is accepted, jobID is IDE_bigtop_0 and taskID is 1. Please wait it to be scheduled 2021-04-25 09:44:40.044 INFO Your job is Scheduled. Please wait it to run. 2021-04-25 09:44:43.044 INFO Background is starting a new engine for you, it may take several seconds, please wait 2021-04-25 09:44:56.044 INFO Congratulations! Your new engine has started successfully ClassNotFoundException: org.apache.tez.dag.api.TezConfiguration 2021-04-25 09:44:56.044 ERROR Request engine failed, possibly due to insufficient resources or background process error(请求引擎失败，可能是由于资源不足或后台进程错误)! 2021-04-25 09:44:57.044 ERROR DWCException{errCode=20010, desc='ClassNotFoundException: org.apache.tez.dag.api.TezConfiguration', ip='host-10-0-105-243', port=9108, serviceKind='hiveEntrance'} at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester$EngineInitThread.notifyThread(EngineRequester.scala:108) at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester$EngineInitThread.notifyThread(EngineRequester.scala:116) at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester.reportNewEngineStatus(EngineRequester.scala:83) at com.webank.wedatasphere.linkis.entrance.execute.EntranceReceiver.receive(EntranceReceiver.scala:104) at com.webank.wedatasphere.linkis.rpc.RPCReceiveRestful$$anon$1$$anonfun$onEvent$2.apply(RPCReceiveRestful.scala:92) at com.webank.wedatasphere.linkis.rpc.RPCReceiveRestful$$anon$1$$anonfun$onEvent$2.apply(RPCReceiveRestful.scala:92) at scala.Option.fold(Option.scala:158)

When I run sparksql I got ERROR:

2021-04-25 09:55:51.266 INFO [qtp1455871253-28] com.webank.wedatasphere.linkis.resourcemanager.utils.YarnUtil$ 42 info - Hadoop version is JString(3.1.1.3.1.4.0-315) 2021-04-25 09:55:51.286 ERROR [qtp1455871253-28] com.webank.wedatasphere.linkis.rpc.RPCReceiveRestful 72 apply - error code(错误码): 11006, error message(错误信息): Get the Yarn queue information exception.(获取Yarn队列信息异常). com.webank.wedatasphere.linkis.resourcemanager.exception.RMErrorException: errCode: 11006 ,desc: Get the Yarn queue information exception.(获取Yarn队列信息异常) ,ip: host-10-0-105-243 ,port: 9104 ,serviceKind: ResourceManager

Caused by: java.lang.ClassCastException: org.json4s.JsonAST$JDouble cannot be cast to org.json4s.JsonAST$JDecimal at com.webank.wedatasphere.linkis.resourcemanager.utils.YarnUtil$$anonfun$maxEffectiveHandle$1$1.apply(YarnUtil.scala:136) ~[linkis-resourcemanager-server-0.9.4.jar:?] at com.webank.wedatasphere.linkis.resourcemanager.utils.YarnUtil$$anonfun$maxEffectiveHandle$1$1.apply(YarnUtil.scala:135) ~[linkis-resourcemanager-server-0.9.4.jar:?] at scala.Option.map(Option.scala:146) ~[scala-library-2.11.8.jar:?] at com.webank.wedatasphere.linkis.resourcemanager.utils.YarnUtil$.maxEffectiveHandle$1(YarnUtil.scala:135) ~[linkis-resourcemanager-server-0.9.4.jar:?] at com.webank.wedatasphere.linkis.resourcemanager.utils.YarnUtil$.com$webank$wedatasphere$linkis$resourcemanager$utils$YarnUtil$$getResources$1(YarnUtil.scala:203) ~[linkis-resourcemanager-server-0.9.4.jar:?] at com.webank.wedatasphere.linkis.resourcemanager.utils.YarnUtil$$anonfun$getQueueInfo$1.apply(YarnUtil.scala:219) ~[linkis-resourcemanager-server-0.9.4.jar:?] at com.webank.wedatasphere.linkis.resourcemanager.utils.YarnUtil$$anonfun$getQueueInfo$1.apply(YarnUtil.scala:219) ~[linkis-resourcemanager-server-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) ~[linkis-common-0.9.4.jar:?] ... 121 more

you10906 commented 3 years ago

i got the same error when i executed hsql.
Linkis version is 0.9.4 , HDP version is 3.1.4

2021-07-15 13:35:34.035 INFO Your job is accepted,  jobID is IDE_hadoop_11 and taskID is 14. Please wait it to be scheduled
2021-07-15 13:35:34.035 INFO Your job is Scheduled. Please wait it to run.
2021-07-15 13:35:35.035 INFO Background is starting a new engine for you, it may take several seconds, please wait
2021-07-15 13:35:45.035 INFO Congratulations! Your new engine has started successfully
ClassNotFoundException: org.apache.tez.dag.api.TezConfiguration
2021-07-15 13:35:45.035 ERROR Request engine failed, possibly due to insufficient resources or background process error(请求引擎失败，可能是由于资源不足或后台进程错误)!
2021-07-15 13:35:45.035 ERROR DWCException{errCode=20010, desc='ClassNotFoundException: org.apache.tez.dag.api.TezConfiguration', ip='nn01', port=9108, serviceKind='hiveEntrance'}
    at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester$EngineInitThread.notifyThread(EngineRequester.scala:108)
    at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester$EngineInitThread.notifyThread(EngineRequester.scala:116)
    at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester.reportNewEngineStatus(EngineRequester.scala:83)
    at com.webank.wedatasphere.linkis.entrance.execute.EntranceReceiver.receive(EntranceReceiver.scala:104)
gangly commented 3 years ago

same error with hdp.