search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.58k stars 1.67k forks source link

[Bug] SeaTunnel job executed failed #6854

Open arthur1989Ng opened 1 month ago

arthur1989Ng commented 1 month ago

Search before asking

What happened

When I use batch mode to synchronize a table whose size is 4.5g, I always get failed.

Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed

SeaTunnel Version

2.3.5

SeaTunnel Config

env {
     parallelism = 1
     job.mode = "BATCH"

}

source {
    Jdbc {
        url="jdbc:mysql:///uac?rewriteBatchedStatements=true"
        driver = "com.mysql.cj.jdbc.Driver"
        user=""
        password=""
        query = "SELECT * from tb_village_people"
    }
}

sink {

    Doris {
      fenodes = "192.168.xx.xx:8030"
      username = 
      password = ""
      table.identifier = "uac.tb_village_people_uac"
      sink.label-prefix = "uac_tb_village_people_uac"
      sink.enable-2pc = "true"
      sink.enable-delete = "true"
      doris.config {
        format = "json"
        read_json_by_line = "true"
      }
  }
}

Running Command

sh /data/apache-seatunnel-2.3.5/bin/seatunnel.sh --config userconf/bigdata.conf

Error Exception

2024-05-15 11:44:38,493 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (842967299441819650) end with state FAILED
2024-05-15 11:44:38,494 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2024-05-15 11:44:38,500 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [192.168.27.230]:5801:5c48b28f-0161-40ba-a2bc-151e8066499e, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/192.168.27.99:33330->/192.168.27.230:5801}, remoteAddress=[192.168.27.230]:5801, lastReadTime=2024-05-15 11:44:35.619, lastWriteTime=2024-05-15 11:44:35.618, closedTime=2024-05-15 11:44:38.497, connected server version=5.1}
2024-05-15 11:44:38,501 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [192.168.27.99]:5801:ac07fda7-009f-423a-8b10-9a66500c6eb7, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/192.168.27.99:56017->/192.168.27.99:5801}, remoteAddress=[192.168.27.99]:5801, lastReadTime=2024-05-15 11:44:38.478, lastWriteTime=2024-05-15 11:44:30.618, closedTime=2024-05-15 11:44:38.500, connected server version=5.1}
2024-05-15 11:44:38,502 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2024-05-15 11:44:38,504 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2024-05-15 11:44:38,505 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2024-05-15 11:44:38,505 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2024-05-15 11:44:38,505 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -

===============================================================================

2024-05-15 11:44:38,505 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error,

2024-05-15 11:44:38,505 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-05-15 11:44:38,505 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed

2024-05-15 11:44:38,507 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:262)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:68)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50)
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78)
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703)
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.subSinkErrorCheck(MultiTableSinkWriter.java:121)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:158)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:43)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:252)
        ... 16 more
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:57)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62)
        ... 5 more
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordBuffer.write(RecordBuffer.java:93)
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:55)
        ... 9 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:194)
        ... 2 more

2024-05-15 11:44:38,507 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -
===============================================================================

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:262)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:68)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50)
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78)
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703)
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.subSinkErrorCheck(MultiTableSinkWriter.java:121)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:158)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:43)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:252)
        ... 16 more
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:57)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62)
        ... 5 more
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordBuffer.write(RecordBuffer.java:93)
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:55)
        ... 9 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:194)
        ... 2 more
2024-05-15 11:44:38,509 INFO  [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-11] - run shutdown hook because get close signal

Zeta or Flink or Spark Version

No response

Java or Scala Version

java version "1.8.0_202"

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

EricJoy2048 commented 1 month ago

It seems doris streamloader failed, but seatunnel is not show the error message in streamloader. After https://github.com/apache/seatunnel/pull/6688, we improve it and the streamloader error message will return to job client.

Now, you need find the streamloader error message from the seatunnel-engine-server log.

arthur1989Ng commented 1 month ago

Here is error message.


2024-05-15 11:44:07,412 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-6-thread-1] - [xx.xx.xx.230]:5801 [seatunnel] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2024-05-15 11:44:19,116 INFO  [o.a.h.i.e.RetryExec           ] [stream-load-upload] - I/O exception (java.net.SocketException) caught when processing request to {}->http://xx.xx.xx.254:8040: Broken pipe (Write failed)
2024-05-15 11:44:25,709 ERROR [o.a.s.c.d.s.w.DorisSinkWriter ] [stream-load-check] - stream load finished unexpectedly, interrupt worker thread! org.apache.http.client.ClientProtocolException
2024-05-15 11:44:25,709 ERROR [s.c.m.MultiTableWriterRunnable] [st-multi-table-sink-writer-1] - MultiTableWriterRunnable error
java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:57) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62) ~[seatunnel-transforms-v2.jar:2.3.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_202]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_202]
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) ~[?:1.8.0_202]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordBuffer.write(RecordBuffer.java:93) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:55) ~[?:?]
        ... 9 more
2024-05-15 11:44:26,136 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@5128cc0
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:262) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:68) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004) ~[seatunnel-starter.jar:2.3.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.subSinkErrorCheck(MultiTableSinkWriter.java:121) ~[seatunnel-transforms-v2.jar:2.3.5]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:158) ~[seatunnel-transforms-v2.jar:2.3.5]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:43) ~[seatunnel-transforms-v2.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:252) ~[seatunnel-starter.jar:2.3.5]
        ... 16 more
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:57) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62) ~[seatunnel-transforms-v2.jar:2.3.5]
        ... 5 more
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_202]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_202]
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) ~[?:1.8.0_202]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordBuffer.write(RecordBuffer.java:93) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:55) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62) ~[seatunnel-transforms-v2.jar:2.3.5]
        ... 5 more
2024-05-15 11:44:26,136 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] taskDone, taskId = 50000, taskGroup = TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}
2024-05-15 11:44:38,433 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@266a1b60
java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.received(IntermediateBlockingQueue.java:41) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.received(IntermediateQueueFlowLifeCycle.java:46) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.received(IntermediateQueueFlowLifeCycle.java:28) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.sendRecordToNext(SeaTunnelSourceCollector.java:204) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:119) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceReader.pollNext(JdbcSourceReader.java:70) ~[?:?]
        at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004) ~[seatunnel-starter.jar:2.3.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_202]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_202]
        at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) ~[?:1.8.0_202]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.received(IntermediateBlockingQueue.java:39) ~[seatunnel-starter.jar:2.3.5]
        ... 16 more
2024-05-15 11:44:38,433 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}
2024-05-15 11:44:38,434 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - Release classloader for job 842967299441819650 with jars [file:/data/apache-seatunnel-2.3.5/connectors/connector-doris-2.3.5.jar, file:/data/apache-seatunnel-2.3.5/plugins/jdbc/lib/mysql-connector-java-8.0.27.jar, file:/data/apache-seatunnel-2.3.5/connectors/connector-jdbc-2.3.5.jar]
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread mysql-cj-abandoned-connection-cleanup
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread ForkJoinPool.commonPool-worker-13
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread hz.main.seaTunnel.task.thread-71
2024-05-15 11:44:38,442 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-71] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000} complete with state FAILED
2024-05-15 11:44:38,448 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.generic-operation.thread-39] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=1}) need cancel.
zxc2817427 commented 1 month ago

In Doris data synchronization, I also encountered the same problem. According to the logs, every time I synchronized 37 million, an error occurred, and the error message could not provide valid error information

EricJoy2048 commented 2 weeks ago

Here is error message.


2024-05-15 11:44:07,412 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-6-thread-1] - [xx.xx.xx.230]:5801 [seatunnel] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2024-05-15 11:44:19,116 INFO  [o.a.h.i.e.RetryExec           ] [stream-load-upload] - I/O exception (java.net.SocketException) caught when processing request to {}->http://xx.xx.xx.254:8040: Broken pipe (Write failed)
2024-05-15 11:44:25,709 ERROR [o.a.s.c.d.s.w.DorisSinkWriter ] [stream-load-check] - stream load finished unexpectedly, interrupt worker thread! org.apache.http.client.ClientProtocolException
2024-05-15 11:44:25,709 ERROR [s.c.m.MultiTableWriterRunnable] [st-multi-table-sink-writer-1] - MultiTableWriterRunnable error
java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:57) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62) ~[seatunnel-transforms-v2.jar:2.3.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_202]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_202]
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) ~[?:1.8.0_202]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordBuffer.write(RecordBuffer.java:93) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:55) ~[?:?]
        ... 9 more
2024-05-15 11:44:26,136 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@5128cc0
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:262) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:68) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004) ~[seatunnel-starter.jar:2.3.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.subSinkErrorCheck(MultiTableSinkWriter.java:121) ~[seatunnel-transforms-v2.jar:2.3.5]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:158) ~[seatunnel-transforms-v2.jar:2.3.5]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter.write(MultiTableSinkWriter.java:43) ~[seatunnel-transforms-v2.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:252) ~[seatunnel-starter.jar:2.3.5]
        ... 16 more
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:57) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62) ~[seatunnel-transforms-v2.jar:2.3.5]
        ... 5 more
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_202]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_202]
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) ~[?:1.8.0_202]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordBuffer.write(RecordBuffer.java:93) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.RecordStream.write(RecordStream.java:55) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.writeRecord(DorisStreamLoad.java:194) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:135) ~[?:?]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.write(DorisSinkWriter.java:52) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableWriterRunnable.run(MultiTableWriterRunnable.java:62) ~[seatunnel-transforms-v2.jar:2.3.5]
        ... 5 more
2024-05-15 11:44:26,136 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] taskDone, taskId = 50000, taskGroup = TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}
2024-05-15 11:44:38,433 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@266a1b60
java.lang.RuntimeException: java.lang.InterruptedException
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.received(IntermediateBlockingQueue.java:41) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.received(IntermediateQueueFlowLifeCycle.java:46) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.received(IntermediateQueueFlowLifeCycle.java:28) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.sendRecordToNext(SeaTunnelSourceCollector.java:204) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:119) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceReader.pollNext(JdbcSourceReader.java:70) ~[?:?]
        at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004) ~[seatunnel-starter.jar:2.3.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_202]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_202]
        at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) ~[?:1.8.0_202]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75) ~[seatunnel-starter.jar:2.3.5]
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.received(IntermediateBlockingQueue.java:39) ~[seatunnel-starter.jar:2.3.5]
        ... 16 more
2024-05-15 11:44:38,433 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}
2024-05-15 11:44:38,434 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - Release classloader for job 842967299441819650 with jars [file:/data/apache-seatunnel-2.3.5/connectors/connector-doris-2.3.5.jar, file:/data/apache-seatunnel-2.3.5/plugins/jdbc/lib/mysql-connector-java-8.0.27.jar, file:/data/apache-seatunnel-2.3.5/connectors/connector-jdbc-2.3.5.jar]
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread mysql-cj-abandoned-connection-cleanup
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread ForkJoinPool.commonPool-worker-13
2024-05-15 11:44:38,442 INFO  [.c.c.DefaultClassLoaderService] [BlockingWorker-TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread hz.main.seaTunnel.task.thread-71
2024-05-15 11:44:38,442 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-71] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=30000} complete with state FAILED
2024-05-15 11:44:38,448 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.generic-operation.thread-39] - [xx.xx.xx.230]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=842967299441819650, pipelineId=1, taskGroupId=1}) need cancel.

This is not the reason of the error, java.lang.InterruptedException is the result, not the reason. You need found other error in the log.

singleWalker commented 1 week ago

I want to migrate 60G of data in one-table from MySQL 5.7 to Doris 2.0.5，I encountered the same problem，but I found some logs as follow:

<FE.WARN> LOG OF DORIS-FE :

2024-06-25 18:37:02,277 WARN (thrift-server-pool-21|14222) [WorkloadRuntimeStatusMgr.updateBeQueryStats():125] be report workload runtime status but without beid 2024-06-25 18:37:02,277 WARN (thrift-server-pool-21|14222) [QeProcessorImpl.reportExecStatus():205] ReportExecStatus() runtime error, query 344a27a4638ec577-e78ba743337a84b8 with type LOAD does not exist

<be.INFO.log.20240625-152518> LOG OF DORIS-BE

I0625 18:36:51.372577 3406 olap_server.cpp:1065] cooldown producer get tablet num: 0 I0625 18:36:52.233999 3339 tablet.cpp:3168] skip to construct delete bitmap tablet: 35095413 rowset: 02000000000000043d4f2a8a22dd7f765092f14ca4cbe596 I0625 18:36:52.234027 3339 memtable.cpp:486] [Memtable Flush] construct delete bitmap tablet: 35095413, rowset_ids: 0, cur max_version: 1, transaction_id: 156, cost: 48(us), total rows: 1365504 W0625 18:37:02.276039 2772 status.h:395] meet error status: [INTERNAL_ERROR]cancelled: sender is gone

    0#  doris::io::StreamLoadPipe::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
    1#  non-virtual thunk to doris::io::StreamLoadPipe::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) at /home/zcp/repo_center/doris_release/doris/be/src/io/fs/stream_load_pipe.cpp:0
    2#  doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:445