search

apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.3k stars 1.17k forks source link

executors.SparkEngineExecutor 59 error #385

Closed liuzhimindluter closed 2 years ago

liuzhimindluter commented 4 years ago

启动一个pyspark引擎，尝试运行以下命令 rdd = sc.parallelize([1,2,3,4,5])

日志如下： 36904: 2020-05-24 16:01:59-965 INFO [sparkEngineEngine-Thread-4] com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor com.webank.wedatasphere.linkis.common.utils.Logging$class.info(Logging.scala:42) info - pyspark builder command:/opt/cslc/anaconda3/bin/python /tmp/3263790786606819816.py 36608 230cloudera3 /opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/lib/py4j-0.10.7-src.zip:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/lib/pyspark.zip:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/jars/spark-core_2.11-2.3.0.cloudera3.jar 36904: 2020-05-24 16:02:00-555 INFO [ForkJoinPool-1-worker-29] com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor com.webank.wedatasphere.linkis.common.utils.Logging$class.info(Logging.scala:42) info - Pyspark process has stopped with exit code 1 36904: 2020-05-24 16:02:30-971 ERROR [sparkEngineEngine-Thread-4] com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor com.webank.wedatasphere.linkis.common.utils.Logging$class.error(Logging.scala:59) error - initialize python executor failed, please ask administrator for help! java.util.concurrent.TimeoutException: null 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.aux$1(Utils.scala:199) ~[linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.waitUntil(Utils.scala:204) ~[linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.waitUntil(Utils.scala:207) ~[linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor.com$webank$wedatasphere$linkis$engine$executors$SparkPythonExecutor$$initGateway(SparkPythonExecutor.scala:153) ~[linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor$$anonfun$lazyInitGageWay$1.apply$mcV$sp(SparkPythonExecutor.scala:170) ~[linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor$$anonfun$lazyInitGageWay$1.apply(SparkPythonExecutor.scala:170) ~[linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor$$anonfun$lazyInitGageWay$1.apply(SparkPythonExecutor.scala:170) ~[linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryThrow(Utils.scala:58) [linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor.lazyInitGageWay(SparkPythonExecutor.scala:170) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkPythonExecutor.execute(SparkPythonExecutor.scala:163) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkEngineExecutor$$anonfun$executeLine$2$$anonfun$2$$anonfun$apply$9.apply(SparkEngineExecutor.scala:133) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkEngineExecutor$$anonfun$executeLine$2$$anonfun$2$$anonfun$apply$9.apply(SparkEngineExecutor.scala:133) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at scala.Option.map(Option.scala:146) [scala-library-2.11.8.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkEngineExecutor$$anonfun$executeLine$2$$anonfun$2.apply(SparkEngineExecutor.scala:133) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkEngineExecutor$$anonfun$executeLine$2$$anonfun$2.apply(SparkEngineExecutor.scala:133) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:62) [linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkEngineExecutor$$anonfun$executeLine$2.apply(SparkEngineExecutor.scala:133) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkEngineExecutor$$anonfun$executeLine$2.apply(SparkEngineExecutor.scala:105) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:62) [linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.executors.SparkEngineExecutor.executeLine(SparkEngineExecutor.scala:141) [linkis-ujes-spark-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9$$anonfun$apply$10.apply(EngineExecutor.scala:141) [linkis-ujes-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9$$anonfun$apply$10.apply(EngineExecutor.scala:140) [linkis-ujes-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9.apply(EngineExecutor.scala:141) [linkis-ujes-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1$$anonfun$apply$9.apply(EngineExecutor.scala:136) [linkis-ujes-engine-0.9.3.jar:?] 36904: at scala.collection.immutable.Range.foreach(Range.scala:160) [scala-library-2.11.8.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1.apply(EngineExecutor.scala:136) [linkis-ujes-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor$$anonfun$execute$1.apply(EngineExecutor.scala:118) [linkis-ujes-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryFinally(Utils.scala:62) [linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.scheduler.executer.AbstractExecutor.ensureIdle(AbstractExecutor.scala:60) [linkis-scheduler-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.scheduler.executer.AbstractExecutor.ensureIdle(AbstractExecutor.scala:54) [linkis-scheduler-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor.ensureOp$1(EngineExecutor.scala:117) [linkis-ujes-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.engine.execute.EngineExecutor.execute(EngineExecutor.scala:118) [linkis-ujes-engine-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$3.apply(Job.scala:254) [linkis-scheduler-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.scheduler.queue.Job$$anonfun$3.apply(Job.scala:254) [linkis-scheduler-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.3.jar:?] 36904: at com.webank.wedatasphere.linkis.scheduler.queue.Job.run(Job.scala:254) [linkis-scheduler-0.9.3.jar:?] 36904: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151] 36904: at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151] 36904: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151] 36904: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151] 36904: at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

尝试在spark节点单独执行日志里面提示的pyspark的构建命令： /opt/cslc/anaconda3/bin/python /tmp/3263790786606819816.py 36608 230cloudera3 /opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/lib/py4j-0.10.7-src.zip:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/lib/pyspark.zip:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/jars/spark-core_2.11-2.3.0.cloudera3.jar

报错： Unable to load inline matplotlib backend, falling back to Agg

yangzhiyue commented 4 years ago

What's your python version? Do you use anaconda or pure python?