search

apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.3k stars 1.17k forks source link

scala2.12 found illegal cyclic reference involving class Object #506

Closed RustRw closed 3 years ago

RustRw commented 3 years ago

Describe the bug use spark 3.00 , scala2.12.10 , but spark-engine start error:

s:0; Last connection made:Thu Jan 01 00:00:00 UTC 1970; First connection made: Thu Jan 01 00:00:00 UTC 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0] 44487: ]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@4c51fc9a 44487: 20/11/11 11:48:24 INFO PyUdfEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO ScalaUdfEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO PyFunctionEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO ScalaFunctionEngineHook: start loading UDFs 44487: 20/11/11 11:48:24 INFO SparkEngineExecutorFactory: Ready to create a Spark EngineExecutor 44487: 20/11/11 11:48:24 INFO SparkEngineExecutorFactory: ------ Create new SparkContext {yarn} ------- 44487: 20/11/11 11:48:24 INFO SparkEngineExecutorFactory: outputDir====> /mnt/tmp/repl-9b2ebdb1-7d87-45ef-9810-278fe426d499 44487: 20/11/11 11:48:24 INFO SparkScalaExecutor: outputDir====> /mnt/tmp/repl-9b2ebdb1-7d87-45ef-9810-278fe426d499 44487: 20/11/11 11:48:24 INFO ChainedDynamicProperty: Flipping property: sparkEngineManager.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647 44487: 44487: scala> 20/11/11 11:48:25 INFO ChainedDynamicProperty: Flipping property: cloud-publicservice.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647 44487: 20/11/11 11:48:25 ERROR EngineReceiver: init engine failed! 44487: scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving class Object

To Reproduce Steps to reproduce the behavior:

  1. run sample sql: select * from xxx limit 10, error.
  2. found the spark-engine-manager linkis.out found error :
peacewong commented 3 years ago

This is a bug introduced by scala 2.12 and will be fixed in version 0.11.0

lordk911 commented 3 years ago

how to fix it? I'm using linkis-0.11.0, but also got the same error. @peacewong

lordk911 commented 3 years ago

@RustRw 请问是如何解决spark3的问题的？

peacewong commented 3 years ago

Hello, in linkis 0.X version, if scala 2.12 needs to comment this line,

Utils.waitUntil(() => scalaExecutor.sparkILoopInited == true && scalaExecutor.sparkILoop.intp != null, new TimeType("120s").toDuration)    
//Thread.currentThread().setContextClassLoader(scalaExecutor.sparkILoop.intp.classLoader)

the circular reference problem caused by classLoader SparkEngineExecutorFactory

lordk911 commented 3 years ago

with HDP3.4.1 as base environment, I've can use spark3.1.1 with linkis 0.11.0 , we should not use spark3's json4s version to compile linkis, but use the default 3.5.3 version . after install linkis replace linkis-ujes-spark-enginemanager/lib/json4s-ast_2.12-3.5.3.jar with spark3's same name jar, then will be fine.