Closed RustRw closed 3 years ago
This is a bug introduced by scala 2.12 and will be fixed in version 0.11.0
how to fix it? I'm using linkis-0.11.0, but also got the same error. @peacewong
@RustRw 请问是如何解决spark3的问题的?
Hello, in linkis 0.X version, if scala 2.12 needs to comment this line,
Utils.waitUntil(() => scalaExecutor.sparkILoopInited == true && scalaExecutor.sparkILoop.intp != null, new TimeType("120s").toDuration)
//Thread.currentThread().setContextClassLoader(scalaExecutor.sparkILoop.intp.classLoader)
the circular reference problem caused by classLoader SparkEngineExecutorFactory
with HDP3.4.1 as base environment, I've can use spark3.1.1 with linkis 0.11.0 , we should not use spark3's json4s version to compile linkis, but use the default 3.5.3 version . after install linkis replace linkis-ujes-spark-enginemanager/lib/json4s-ast_2.12-3.5.3.jar with spark3's same name jar, then will be fine.
Describe the bug use spark 3.00 , scala2.12.10 , but spark-engine start error:
s:0; Last connection made:Thu Jan 01 00:00:00 UTC 1970; First connection made: Thu Jan 01 00:00:00 UTC 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0] 44487: ]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@4c51fc9a 44487: 20/11/11 11:48:24 INFO PyUdfEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO ScalaUdfEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO PyFunctionEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO ScalaFunctionEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO SparkEngineExecutorFactory: Ready to create a Spark EngineExecutor 44487: 20/11/11 11:48:24 INFO SparkEngineExecutorFactory: ------ Create new SparkContext {yarn} ------- 44487: 20/11/11 11:48:24 INFO SparkEngineExecutorFactory: outputDir====> /mnt/tmp/repl-9b2ebdb1-7d87-45ef-9810-278fe426d499 44487: 20/11/11 11:48:24 INFO SparkScalaExecutor: outputDir====> /mnt/tmp/repl-9b2ebdb1-7d87-45ef-9810-278fe426d499 44487: 20/11/11 11:48:24 INFO ChainedDynamicProperty: Flipping property: sparkEngineManager.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647 44487: 44487: scala> 20/11/11 11:48:25 INFO ChainedDynamicProperty: Flipping property: cloud-publicservice.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647 44487: 20/11/11 11:48:25 ERROR EngineReceiver: init engine failed! 44487: scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving class Object
To Reproduce Steps to reproduce the behavior: