what's problem caused the stream program shut down ?

[zhengqiangtan]

hi ,I'm sorry to bothered you, I met some issue like this , I can not find the reason why the stream program shutdown , This is the detail error log .

2020-01-14 16:55:00 ERROR MicroBatchExecution:91 - Query [id = 1e58880f-be83-47c3-b837-9d7c64a90982, runId = 9eecfa85-29c5-4246-963e-90fd09f6731c] terminated with error java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.spark.sql.mlsql.sources.mysql.binlog.BinLogSocketServerSerDer$class.readResponse(servers.scala:163) at org.apache.spark.sql.mlsql.sources.MLSQLBinLogSource.readResponse(MLSQLBinLogDataSource.scala:243) at org.apache.spark.sql.mlsql.sources.MLSQLBinLogSource.getLatestOffset(MLSQLBinLogDataSource.scala:324) at org.apache.spark.sql.mlsql.sources.MLSQLBinLogSource.getOffset(MLSQLBinLogDataSource.scala:334) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1$$anonfun$5$$anonfun$apply$9.apply(MicroBatchExecution.scala:345) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1$$anonfun$5$$anonfun$apply$9.apply(MicroBatchExecution.scala:345) at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351) at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1$$anonfun$5.apply(MicroBatchExecution.scala:344) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1$$anonfun$5.apply(MicroBatchExecution.scala:341) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1.apply$mcZ$sp(MicroBatchExecution.scala:341) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1.apply(MicroBatchExecution.scala:337) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1.apply(MicroBatchExecution.scala:337) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.withProgressLocked(MicroBatchExecution.scala:554) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch(MicroBatchExecution.scala:337) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:183) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166) at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351) at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58) at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166) at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160) at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:279) at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189) 2020-01-14 16:55:00 INFO DAGScheduler:54 - Asked to cancel job group 8913a720-df1b-427b-abfc-18cae73935dc 2020-01-14 16:55:00 INFO YarnScheduler:54 - Cancelling stage 3 2020-01-14 16:55:00 INFO YarnScheduler:54 - Killing all running tasks in stage 3: Stage cancelled 2020-01-14 16:55:00 INFO YarnScheduler:54 - Stage 3 was cancelled 2020-01-14 16:55:00 INFO DAGScheduler:54 - ResultStage 3 (start at NativeMethodAccessorImpl.java:0) failed in 86450.992 s due to Job 1 cancelled part of cancelled job group 8913a720-df1b-427b-abfc-18cae73935dc 2020-01-14 16:55:00 INFO DAGScheduler:54 - Job 1 failed: start at NativeMethodAccessorImpl.java:0, took 86450.997796 s Exception in thread "launch-binlog-socket-server-in-spark-job" org.apache.spark.SparkException: Job 1 cancelled part of cancelled job group 8913a720-df1b-427b-abfc-18cae73935dc at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887) at org.apache.spark.scheduler.DAGScheduler.handleJobCancellation(DAGScheduler.scala:1822) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleJobGroupCancelled$1.apply$mcVI$sp(DAGScheduler.scala:906) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleJobGroupCancelled$1.apply(DAGScheduler.scala:906) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleJobGroupCancelled$1.apply(DAGScheduler.scala:906) at scala.collection.mutable.HashSet.foreach(HashSet.scala:78) at org.apache.spark.scheduler.DAGScheduler.handleJobGroupCancelled(DAGScheduler.scala:906) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2077) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:363) at org.apache.spark.rdd.RDD.collect(RDD.scala:944) at org.apache.spark.sql.mlsql.sources.MLSQLBinLogDataSource.org$apache$spark$sql$mlsql$sources$MLSQLBinLogDataSource$$launchBinlogServer$1(MLSQLBinLogDataSource.scala:207) at org.apache.spark.sql.mlsql.sources.MLSQLBinLogDataSource$$anon$4.run(MLSQLBinLogDataSource.scala:214) Traceback (most recent call last): File "/data/sync/ods_users_df/stream2hdfs.py", line 36, in query.awaitTermination() File "/opt/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/streaming.py", line 103, in awaitTermination File "/opt/spark-2.4.0-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in call File "/opt/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/utils.py", line 75, in deco

[allwefantasy]

I guess the executor who runs binlog server may be killed by yarn. Try to increase the spark executor overhead memory.

[zhengqiangtan]

[zhengqiangtan]

2020-01-15 12:09:26 ERROR MLSQLBinLogDataSource:91 - next on empty iterator 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 2 on zmbd-pm07: Executor heartbeat timed out after 155288 ms 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 5 on zmbd-pm13: Executor heartbeat timed out after 152209 ms 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 8 on zmbd-pm11: Executor heartbeat timed out after 152101 ms 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 7 on zmbd-pm07: Executor heartbeat timed out after 150332 ms 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 1 on zmbd-pm07: Executor heartbeat timed out after 156505 ms 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 4 on zmbd-pm13: Executor heartbeat timed out after 147583 ms 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 6 on zmbd-pm16: Executor heartbeat timed out after 150414 ms 2020-01-16 01:31:49 ERROR YarnScheduler:70 - Lost executor 3 on zmbd-pm07: Executor heartbeat timed out after 150221 ms 2020-01-16 01:33:49 ERROR TransportChannelHandler:144 - Connection to /172.20.1.10:36940 has been quiet for 120000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.network.timeout if this is wrong. 2020-01-16 01:33:50 ERROR TransportResponseHandler:144 - Still have 1 requests outstanding when connection from /172.20.1.10:36940 is closed 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(2)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:33:50 ERROR TransportClient:331 - Failed to send RPC RPC 8516244618252785373 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(5)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:33:50 ERROR TransportClient:331 - Failed to send RPC RPC 5604667397559748996 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(8)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:33:50 ERROR TransportClient:331 - Failed to send RPC RPC 4795019310054221062 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(7)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:33:50 ERROR TransportClient:331 - Failed to send RPC RPC 5234086875584424262 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(1)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:33:50 ERROR TransportClient:331 - Failed to send RPC RPC 5740243668631548947 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(4)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:33:50 ERROR TransportClient:331 - Failed to send RPC RPC 7509468406493835515 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(6)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:33:50 ERROR TransportClient:331 - Failed to send RPC RPC 4790295982241523280 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:33:50 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending KillExecutors(List(3)) to AM was unsuccessful 2020-01-16 01:33:50 ERROR Utils:91 - Uncaught exception in thread kill-executor-thread 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:06 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:34 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:34 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:38 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:38 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:41 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:41 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:41 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:41 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:42 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:42 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:42 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:42 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:44 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:44 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:44 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:39:44 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:45:34 ERROR MicroBatchExecution:91 - Query [id = c4492e8a-a982-4774-a19b-6aa5a2ac9c9b, runId = 5aa1e3bd-2f26-4e96-bd26-c88d0efa9176] terminated with error 2020-01-16 01:45:34 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:45:34 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:45:34 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:45:34 ERROR AsyncEventQueue:91 - Listener EventLoggingListener threw an exception 2020-01-16 01:45:34 ERROR TransportClient:331 - Failed to send RPC RPC 6764822998199901153 to /172.20.1.10:36940: java.nio.channels.ClosedChannelException 2020-01-16 01:45:34 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful 2020-01-16 01:45:34 ERROR Utils:91 - Uncaught exception in thread Thread-1 2020-01-16 01:45:34 ERROR Utils:91 - Uncaught exception in thread Thread-1