Closed xiaohei-info closed 2 years ago
有两个解决办法。
第一个, byzer 的主包使用 --jars 也带上。 第二个,添加一个启动参数试试:
-streaming.ps.cluster.enable false
问题原因: Byzer 需要在 Executor侧也做一些初始化,需要对应的类通过 --jars 带上。或者说不初始化这些功能,那么就可以使用第二个方法关掉。
另外,Byzer 官方测试过的是Spark 2.4.3 和 3.1.1。
另外,如果你使用 byzer 自带的启动脚本启动的,能贴一个完整的启动脚本的日志么?
另外,如果你使用 byzer 自带的启动脚本启动的,能贴一个完整的启动脚本的日志么?
可以提供 logs/shell.stderr
, logs/byzer.out
,logs/byzer.log
, 以及 conf/byzer.properties
或 conf/byzer.properties.override
来方便排查问题
另外,如果你使用 byzer 自带的启动脚本启动的,能贴一个完整的启动脚本的日志么?
可以提供
logs/shell.stderr
,logs/byzer.out
,logs/byzer.log
, 以及conf/byzer.properties
或conf/byzer.properties.override
来方便排查问题
conf/byzer.properties.override
byzer.server.mode=server streaming.master=yarn streaming.name=Byzer-lang-server streaming.rest=true streaming.thrift=false streaming.platform=spark streaming.spark.service=true streaming.driver.port=9003 streaming.enableHiveSupport=true streaming.datalake.path=./delta spark.driver.memory=2g spark.executor.memory=1g spark.driver.cores=1 spark.executor.cores=1 spark.executor.instances=10
bin/byzer.sh start
Starting Byzer engine...
Byzer-lang is checking installation environment, log is at /opt/soft/byzer-lang-2.4.3-2.3.0.1/logs/check-env.out
Checking OS ...................................................[PASS] Checking Java Version ...................................................[PASS] Checking Ports Availability ...................................................[PASS]
Checking environment finished successfully. To check again, run 'bin/check-env.sh' manually.
SPARK_HOME is: /opt/cloudera/parcels/CDH/lib/spark BYZER_HOME is: /opt/soft/byzer-lang-2.4.3-2.3.0.1 BYZER_CONFIG_FILE is: /opt/soft/byzer-lang-2.4.3-2.3.0.1/conf/byzer.properties Starting Byzer engine in server mode...
[Spark Config] --conf spark.kryoserializer.buffer=256k --conf spark.executor.memory=1g --conf spark.driver.memory=2g --conf spark.kryoserializer.buffer.max=1024m --conf spark.sql.hive.thriftServer.singleSession=true --conf spark.master=local[] --conf spark.scheduler.mode=FAIR --conf spark.executor.cores=1 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.executor.instances=10 --conf spark.driver.cores=1 [Byzer Config] -streaming.spark.service true -streaming.driver.port 9003 -streaming.platform spark -streaming.name Byzer-lang-server -streaming.thrift false -streaming.master yarn -streaming.rest true -streaming.datalake.path /warehouse/deltalake -streaming.enableHiveSupport true [Extra Config] /opt/soft/byzer-lang-2.4.3-2.3.0.1/plugin/.jar:/opt/soft/byzer-lang-2.4.3-2.3.0.1/libs/a_guava-28.1.jre.jar:/opt/soft/byzer-lang-2.4.3-2.3.0.1/libs/ansj_seg-5.1.6.jar:/opt/soft/byzer-lang-2.4.3-2.3.0.1/libs/nlp-lang-1.7.8.jar:/opt/soft/byzer-lang-2.4.3-2.3.0.1/main/byzer-lang-2.4.3-2.11-2.3.0.1.jar
Byzer engine is starting. It may take a while. For status, please visit http://127.0.0.1:9003.
You may also check status via: PID:3325, or Log: /opt/soft/byzer-lang-2.4.3-2.3.0.1/logs/byzer-lang.log.
logs/byzer-lang.log
22/05/16 21:21:03 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 7449 for reason Container from a bad node: container_e06_1651462646567_0164_01_011189 on host: findataplat-worker01.tg.mt.com. Exit status: 1. Diagnostics: [2022-05-16 21:21:03.581]Exception from container-launch.
Container id: container_e06_1651462646567_0164_01_011189
Exit code: 1
Exception message: Launch container failed
Shell output: main : command provided 1
main : run as user is hive
main : requested yarn user is hive
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /opt/yarn/nm/nmPrivate/application_1651462646567_0164/container_e06_1651462646567_0164_01_011189/container_e06_1651462646567_0164_01_011189.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
[2022-05-16 21:21:03.582]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ocal directory at /opt/yarn/nm/usercache/hive/appcache/application_1651462646567_0164/blockmgr-37586980-ff61-4917-94a3-3aab884ef123
22/05/16 21:21:02 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@findataplat-utility.tg.mt.com:30157
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
22/05/16 21:21:03 INFO executor.Executor: Starting executor ID 7449 on host findataplat-worker01.tg.mt.com
22/05/16 21:21:03 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 15269.
22/05/16 21:21:03 INFO netty.NettyBlockTransferService: Server created on findataplat-worker01.tg.mt.com:15269
22/05/16 21:21:03 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManager: external shuffle service port = 7337
22/05/16 21:21:03 INFO storage.BlockManager: Registering executor with local external shuffle service.
22/05/16 21:21:03 INFO client.TransportClientFactory: Successfully created connection to findataplat-worker01.tg.mt.com/10.199.132.216:7337 after 1 ms (0 ms spent in bootstraps)
22/05/16 21:21:03 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 ERROR executor.CoarseGrainedExecutorBackend: Executor self-exiting due to : Unable to create executor due to org.apache.spark.ps.cluster.PSExecutorPlugin
java.lang.ClassNotFoundException: org.apache.spark.ps.cluster.PSExecutorPlugin
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:194)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2660)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2658)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2658)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:148)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:147)
at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:205)
at org.apache.spark.executor.Executor.
[2022-05-16 21:21:03.583]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ocal directory at /opt/yarn/nm/usercache/hive/appcache/application_1651462646567_0164/blockmgr-37586980-ff61-4917-94a3-3aab884ef123
22/05/16 21:21:02 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@findataplat-utility.tg.mt.com:30157
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
22/05/16 21:21:03 INFO executor.Executor: Starting executor ID 7449 on host findataplat-worker01.tg.mt.com
22/05/16 21:21:03 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 15269.
22/05/16 21:21:03 INFO netty.NettyBlockTransferService: Server created on findataplat-worker01.tg.mt.com:15269
22/05/16 21:21:03 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManager: external shuffle service port = 7337
22/05/16 21:21:03 INFO storage.BlockManager: Registering executor with local external shuffle service.
22/05/16 21:21:03 INFO client.TransportClientFactory: Successfully created connection to findataplat-worker01.tg.mt.com/10.199.132.216:7337 after 1 ms (0 ms spent in bootstraps)
22/05/16 21:21:03 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 ERROR executor.CoarseGrainedExecutorBackend: Executor self-exiting due to : Unable to create executor due to org.apache.spark.ps.cluster.PSExecutorPlugin
java.lang.ClassNotFoundException: org.apache.spark.ps.cluster.PSExecutorPlugin
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:194)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2660)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2658)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2658)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:148)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:147)
at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:205)
at org.apache.spark.executor.Executor.
logs/shell.stderr
22/05/16 21:21:03 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 7449 for reason Container from a bad node: container_e06_1651462646567_0164_01_011189 on host: findataplat-worker01.tg.mt.com. Exit status: 1. Diagnostics: [2022-05-16 21:21:03.581]Exception from container-launch.
Container id: container_e06_1651462646567_0164_01_011189
Exit code: 1
Exception message: Launch container failed
Shell output: main : command provided 1
main : run as user is hive
main : requested yarn user is hive
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /opt/yarn/nm/nmPrivate/application_1651462646567_0164/container_e06_1651462646567_0164_01_011189/container_e06_1651462646567_0164_01_011189.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
[2022-05-16 21:21:03.582]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ocal directory at /opt/yarn/nm/usercache/hive/appcache/application_1651462646567_0164/blockmgr-37586980-ff61-4917-94a3-3aab884ef123
22/05/16 21:21:02 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@findataplat-utility.tg.mt.com:30157
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
22/05/16 21:21:03 INFO executor.Executor: Starting executor ID 7449 on host findataplat-worker01.tg.mt.com
22/05/16 21:21:03 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 15269.
22/05/16 21:21:03 INFO netty.NettyBlockTransferService: Server created on findataplat-worker01.tg.mt.com:15269
22/05/16 21:21:03 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManager: external shuffle service port = 7337
22/05/16 21:21:03 INFO storage.BlockManager: Registering executor with local external shuffle service.
22/05/16 21:21:03 INFO client.TransportClientFactory: Successfully created connection to findataplat-worker01.tg.mt.com/10.199.132.216:7337 after 1 ms (0 ms spent in bootstraps)
22/05/16 21:21:03 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 ERROR executor.CoarseGrainedExecutorBackend: Executor self-exiting due to : Unable to create executor due to org.apache.spark.ps.cluster.PSExecutorPlugin
java.lang.ClassNotFoundException: org.apache.spark.ps.cluster.PSExecutorPlugin
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:194)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2660)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2658)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2658)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:148)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:147)
at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:205)
at org.apache.spark.executor.Executor.
[2022-05-16 21:21:03.583]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ocal directory at /opt/yarn/nm/usercache/hive/appcache/application_1651462646567_0164/blockmgr-37586980-ff61-4917-94a3-3aab884ef123
22/05/16 21:21:02 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@findataplat-utility.tg.mt.com:30157
22/05/16 21:21:03 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
22/05/16 21:21:03 INFO executor.Executor: Starting executor ID 7449 on host findataplat-worker01.tg.mt.com
22/05/16 21:21:03 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 15269.
22/05/16 21:21:03 INFO netty.NettyBlockTransferService: Server created on findataplat-worker01.tg.mt.com:15269
22/05/16 21:21:03 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 INFO storage.BlockManager: external shuffle service port = 7337
22/05/16 21:21:03 INFO storage.BlockManager: Registering executor with local external shuffle service.
22/05/16 21:21:03 INFO client.TransportClientFactory: Successfully created connection to findataplat-worker01.tg.mt.com/10.199.132.216:7337 after 1 ms (0 ms spent in bootstraps)
22/05/16 21:21:03 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(7449, findataplat-worker01.tg.mt.com, 15269, None)
22/05/16 21:21:03 ERROR executor.CoarseGrainedExecutorBackend: Executor self-exiting due to : Unable to create executor due to org.apache.spark.ps.cluster.PSExecutorPlugin
java.lang.ClassNotFoundException: org.apache.spark.ps.cluster.PSExecutorPlugin
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:194)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2660)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2658)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2658)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:148)
at org.apache.spark.executor.Executor$$anonfun$6.apply(Executor.scala:147)
at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:205)
at org.apache.spark.executor.Executor.
看起来很奇怪;主包应该通过 --jars 戴上了;要不你把在配置文件里加个 streaming.ps.cluster.enable=false 吧
看起来很奇怪;主包应该通过 --jars 戴上了;要不你把在配置文件里加个 streaming.ps.cluster.enable=false 吧
有其他异常: 22/05/18 19:41:54 INFO Configuration: resource-types.xml not found 22/05/18 19:41:54 INFO ResourceUtils: Unable to find 'resource-types.xml'. 22/05/18 19:41:55 INFO YarnClientImpl: Submitted application application_1651462646567_0207 Exception in thread "main" java.lang.VerifyError: Stack map does not match the one at exception handler 77 Exception Details: Location: com/fasterxml/jackson/databind/deser/std/StdDeserializer._parseDate(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/util/Date; @77: astore Reason: Type 'com/fasterxml/jackson/core/JsonParseException' (current frame, stack[0]) is not assignable to 'com/fasterxml/jackson/core/exc/StreamReadException' (stack map, stack[0]) Current Frame: bci: @69 flags: { } locals: { 'com/fasterxml/jackson/databind/deser/std/StdDeserializer', 'com/fasterxml/jackson/core/JsonParser', 'com/fasterxml/jackson/databind/DeserializationContext' } stack: { 'com/fasterxml/jackson/core/JsonParseException' } Stackmap Frame: bci: @77 flags: { } locals: { 'com/fasterxml/jackson/databind/deser/std/StdDeserializer', 'com/fasterxml/jackson/core/JsonParser', 'com/fasterxml/jackson/databind/DeserializationContext' } stack: { 'com/fasterxml/jackson/core/exc/StreamReadException' } Bytecode: 0x0000000: 2bb6 0035 aa00 0000 0000 0081 0000 0003 0x0000010: 0000 000b 0000 007a 0000 0081 0000 0081 0x0000020: 0000 0034 0000 0041 0000 0081 0000 0081 0x0000030: 0000 0081 0000 0071 2a2b b600 11b6 0012 0x0000040: 2cb6 006b b02b b600 4742 a700 223a 052c 0x0000050: 2ab4 0002 2bb6 006e 126f 03bd 0004 b600 0x0000060: 70c0 002d 3a06 1906 b600 4c42 bb00 7159 0x0000070: 21b7 0072 b02a 2cb6 0073 c000 71b0 2a2b 0x0000080: 2cb6 0074 b02c 2ab4 0002 2bb6 0025 c000 0x0000090: 71b0 Exception Handler Table: bci [69, 74] => handler: 77 bci [69, 74] => handler: 77 Stackmap Table: same_frame(@56) same_frame(@69) same_locals_1_stack_item_frame(@77,Object[#367]) append_frame(@108,Long) chop_frame(@117,1) same_frame(@126) same_frame(@133)
at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<init>(ScalaNumberDeserializersModule.scala:48)
at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<clinit>(ScalaNumberDeserializersModule.scala)
at com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule$class.$init$(ScalaNumberDeserializersModule.scala:60)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:18)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:36)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:349)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:131)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:533)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2547)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:939)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:930)
at streaming.core.strategy.platform.SparkRuntime.createRuntime(SparkRuntime.scala:185)
at streaming.core.strategy.platform.SparkRuntime.<init>(SparkRuntime.scala:63)
at streaming.core.strategy.platform.SparkRuntime$.getOrCreate(SparkRuntime.scala:330)
at streaming.core.strategy.platform.SparkRuntime.getOrCreate(SparkRuntime.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at streaming.core.strategy.platform.PlatformManager$.createRuntimeByPlatform(PlatformManager.scala:237)
at streaming.core.strategy.platform.PlatformManager$.getRuntime(PlatformManager.scala:252)
at streaming.core.strategy.platform.PlatformManager.run(PlatformManager.scala:117)
at streaming.core.StreamingApp$.main(StreamingApp.scala:45)
at streaming.core.StreamingApp.main(StreamingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:847)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
看起来很奇怪;主包应该通过 --jars 戴上了;要不你把在配置文件里加个 streaming.ps.cluster.enable=false 吧
有其他异常: 22/05/18 19:41:54 INFO Configuration: resource-types.xml not found 22/05/18 19:41:54 INFO ResourceUtils: Unable to find 'resource-types.xml'. 22/05/18 19:41:55 INFO YarnClientImpl: Submitted application application_1651462646567_0207 Exception in thread "main" java.lang.VerifyError: Stack map does not match the one at exception handler 77 Exception Details: Location: com/fasterxml/jackson/databind/deser/std/StdDeserializer._parseDate(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/util/Date; @77: astore Reason: Type 'com/fasterxml/jackson/core/JsonParseException' (current frame, stack[0]) is not assignable to 'com/fasterxml/jackson/core/exc/StreamReadException' (stack map, stack[0]) Current Frame: bci: @69 flags: { } locals: { 'com/fasterxml/jackson/databind/deser/std/StdDeserializer', 'com/fasterxml/jackson/core/JsonParser', 'com/fasterxml/jackson/databind/DeserializationContext' } stack: { 'com/fasterxml/jackson/core/JsonParseException' } Stackmap Frame: bci: @77 flags: { } locals: { 'com/fasterxml/jackson/databind/deser/std/StdDeserializer', 'com/fasterxml/jackson/core/JsonParser', 'com/fasterxml/jackson/databind/DeserializationContext' } stack: { 'com/fasterxml/jackson/core/exc/StreamReadException' } Bytecode: 0x0000000: 2bb6 0035 aa00 0000 0000 0081 0000 0003 0x0000010: 0000 000b 0000 007a 0000 0081 0000 0081 0x0000020: 0000 0034 0000 0041 0000 0081 0000 0081 0x0000030: 0000 0081 0000 0071 2a2b b600 11b6 0012 0x0000040: 2cb6 006b b02b b600 4742 a700 223a 052c 0x0000050: 2ab4 0002 2bb6 006e 126f 03bd 0004 b600 0x0000060: 70c0 002d 3a06 1906 b600 4c42 bb00 7159 0x0000070: 21b7 0072 b02a 2cb6 0073 c000 71b0 2a2b 0x0000080: 2cb6 0074 b02c 2ab4 0002 2bb6 0025 c000 0x0000090: 71b0 Exception Handler Table: bci [69, 74] => handler: 77 bci [69, 74] => handler: 77 Stackmap Table: same_frame(@56) same_frame(@69) same_locals_1_stack_item_frame(@77,Object[#367]) append_frame(@108,Long) chop_frame(@117,1) same_frame(@126) same_frame(@133)
at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<init>(ScalaNumberDeserializersModule.scala:48) at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<clinit>(ScalaNumberDeserializersModule.scala) at com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule$class.$init$(ScalaNumberDeserializersModule.scala:60) at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:18) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:36) at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala) at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60) at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala) at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:349) at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:131) at org.apache.spark.SparkContext.<init>(SparkContext.scala:533) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2547) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:939) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:930) at streaming.core.strategy.platform.SparkRuntime.createRuntime(SparkRuntime.scala:185) at streaming.core.strategy.platform.SparkRuntime.<init>(SparkRuntime.scala:63) at streaming.core.strategy.platform.SparkRuntime$.getOrCreate(SparkRuntime.scala:330) at streaming.core.strategy.platform.SparkRuntime.getOrCreate(SparkRuntime.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at streaming.core.strategy.platform.PlatformManager$.createRuntimeByPlatform(PlatformManager.scala:237) at streaming.core.strategy.platform.PlatformManager$.getRuntime(PlatformManager.scala:252) at streaming.core.strategy.platform.PlatformManager.run(PlatformManager.scala:117) at streaming.core.StreamingApp$.main(StreamingApp.scala:45) at streaming.core.StreamingApp.main(StreamingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:847) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
应该还是spark版本的问题,使用243后启动正常
byzer version: byzer-lang-2.4.3-2.3.0.1 spark version: 2.11-2.4.7.7.1.7.0-551(Cloudera CDH) scala version: 2.11.12