search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.78k stars 1.74k forks source link

[Bug] [elastic search sink] es Real time data write interrupt #6336

Open lovezhou1990 opened 7 months ago

lovezhou1990 commented 7 months ago

Search before asking

What happened

Scenario: Real time writing of MySQL changed data to the ES end (MySQL cdc>>ES) Problem description: If the change data does not reach 10 pieces (default batch submission number for ES sink), it will not be written to ES. If the change reaches 10 pieces, it will be written to ES. (There is no problem at the beginning of the task, even if one piece of data is changed, it will be synchronized to ES) Upon self investigation, it was found that: 1> As long as there is a data change, it will enter the code: org. apache. seatunnel. connectors. seatunnel. Elasticsearch. sink. ElasticsearchSinkWriter # write. However, it is determined that if the submission count is not reached, the org. apache. seatunnel. connectors. seatunnel. Elasticsearch. sink. ElasticsearchSinkWriter # bulkEsWithTry method will not be executed

2> Local debugging found that when the task starts, the RetryUtils. retryWithException in the org. apache. seatunnel. connectors. seatunnel. Elasticsearch. link. Elasticsearch. SinkWriter # bulkEsWithRetry method will loop and execute until there is a problem

SeaTunnel Version

2.3.3

SeaTunnel Config

env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.name = "st功能调试"
  job.mode = "STREAMING"
  checkpoint.interval = 2000
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
  MySQL-CDC {
    result_table_name = "company"
    parallelism = 1
    username = "xxxx"
    password = "xxxx"
    startup.mode = "latest"
    database-names =["test3_bigdata"]
    table-names = ["test3_bigdata.company"]
    base-url = "jdbc:mysql://xxx:3306/test3_bigdata?useUnicode=true&characterEncoding=utf8&serverTimezone=GMT%2B8&rewriteBatchedStatements=true&zeroDateTimeBehavior=CONVERT_TO_NULL"
  }
}

sink {
  Elasticsearch {
      source_table_name = "company"
      hosts = ["http://xxx:19200"]
      index = "changelog"
    }
}

Running Command

org.apache.seatunnel.example.engine.SeaTunnelEngineServerExample

org.apache.seatunnel.example.engine.SeaTunnelEngineExample

by source execution

Error Exception

no

Zeta or Flink or Spark Version

zeta

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

lovezhou1990 commented 7 months ago

When will the checkpoint task stop? Currently, debugging has found that the possible reason is that the checkpoint task stopped midway. Code cannot enter method:org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle#received 'if (record.getData() instanceof Barrier) check

lovezhou1990 commented 6 months ago

Discovering new anomalies:

2024-02-22 08:58:56,392 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-27] - wait checkpoint completed: 27782 2024-02-22 08:58:56,395 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-27] - pending checkpoint(27782/2@812617486796062721) notify finished! 2024-02-22 08:58:56,395 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-27] - start notify checkpoint completed, job id: 812617486796062721, pipeline id: 2, checkpoint id:27782 2024-02-22 08:58:57,874 INFO [o.a.s.e.s.d.p.PhysicalPlan ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721) turned from state RUNNING to CANCELING. 2024-02-22 08:58:57,874 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-27] - start clean pending checkpoint cause Pipeline turn to end state. 2024-02-22 08:58:57,874 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-27] - Turn checkpoint_state_812617486796062721_1 state from RUNNING to CANCELED 2024-02-22 08:58:57,875 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(1/2)] turned from state RUNNING to CANCELING. 2024-02-22 08:58:57,875 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] state process is start 2024-02-22 08:58:57,875 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] turned from state RUNNING to CANCELING. 2024-02-22 08:58:57,875 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Send cancel Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] operator to member [localhost]:5801 2024-02-22 08:58:57,875 INFO [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-27] - [localhost]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}) need cancel. 2024-02-22 08:58:57,875 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SourceTask (1/1)] state process is start 2024-02-22 08:58:57,875 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel] [5.1] Interrupted task 20000 - org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask@2544bd88 2024-02-22 08:58:57,875 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1} 2024-02-22 08:58:57,875 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}] - recycle classloader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader@132557f 2024-02-22 08:58:57,876 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SourceTask (1/1)] turned from state RUNNING to CANCELING. 2024-02-22 08:58:57,876 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Send cancel Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SourceTask (1/1)] operator to member [localhost]:5801 2024-02-22 08:58:57,876 INFO [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-27] - [localhost]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}) need cancel. 2024-02-22 08:58:57,876 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-27] - start clean pending checkpoint cause Pipeline turn to end state. 2024-02-22 08:58:57,877 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] Interrupted task 50000 - org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@2508776d 2024-02-22 08:58:57,877 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 50000, taskGroup = TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000} 2024-02-22 08:58:57,877 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@3dd4c1c6 org.apache.seatunnel.common.utils.SeaTunnelException: java.lang.InterruptedException: sleep interrupted at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:166) ~[seatunnel-transforms-v2.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:92) ~[seatunnel-transforms-v2.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:110) ~[?:?] at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:150) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:643) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:944) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_144] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) ~[?:1.8.0_144] at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:164) ~[seatunnel-transforms-v2.jar:2.3.4-SNAPSHOT] ... 13 more 2024-02-22 08:58:57,877 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000} 2024-02-22 08:58:57,877 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader@21f134cb 2024-02-22 08:58:57,882 INFO [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-27] - Turn checkpoint_state_812617486796062721_2 state from RUNNING to CANCELED 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}] - recycle classloader for thread connection-pool-192.168.1.188:3306 housekeeper 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}] - recycle classloader for thread BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1} 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 8 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 5 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 17 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 9 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 15 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 14 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-1] - [localhost]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1} complete with state CANCELED 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 7 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 19 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 12 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 18 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 16 2024-02-22 08:58:57,883 INFO [.e.IncrementalSourceEnumerator] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}] - Closing enumerator... 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.CoordinatorService ] [hz.main.seaTunnel.task.thread-1] - [localhost]:5801 [seatunnel] [5.1] Received task end from execution TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=1}, state CANCELED 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 10 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 6 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000} 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000} 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread pool-28-thread-1 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 1 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 20 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 2 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 11 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread ForkJoinPool.commonPool-worker-8 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 13 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 3 2024-02-22 08:58:57,883 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - recycle classloader for thread I/O dispatcher 4 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(2/2)] turned from state RUNNING to CANCELING. 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(2/2)], task: [pipeline-2 [Source[1]-MySQL-CDC]-SplitEnumerator (1/1)] state process is start 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-1] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] turned from state CANCELING to CANCELED. 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-1] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] state process is stopped 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-13] - [localhost]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000} complete with state CANCELED 2024-02-22 08:58:57,883 INFO [a.s.c.s.c.s.r.SourceReaderBase] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - Closing Source Reader. 2024-02-22 08:58:57,883 INFO [o.a.s.e.s.CoordinatorService ] [hz.main.seaTunnel.task.thread-13] - [localhost]:5801 [seatunnel] [5.1] Received task end from execution TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}, state CANCELED 2024-02-22 08:58:57,884 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-29] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SplitEnumerator (1/1)] future complete with state CANCELED 2024-02-22 08:58:57,884 INFO [o.a.s.c.s.c.s.r.f.SplitFetcher] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=1, taskGroupId=30000}] - Shutting down split fetcher 0 2024-02-22 08:58:57,884 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-13] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SourceTask (1/1)] turned from state CANCELING to CANCELED. 2024-02-22 08:58:57,884 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.seaTunnel.task.thread-13] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SourceTask (1/1)] state process is stopped 2024-02-22 08:58:57,884 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(2/2)], task: [pipeline-2 [Source[1]-MySQL-CDC]-SplitEnumerator (1/1)] turned from state RUNNING to CANCELING. 2024-02-22 08:58:57,884 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-29] - Job datamonitor (812617486796062721), Pipeline: [(1/2)], task: [pipeline-1 [Source[0]-MySQL-CDC]-SourceTask (1/1)] future complete with state CANCELED 2024-02-22 08:58:57,884 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-29] - Job datamonitor (812617486796062721), Pipeline: [(1/2)] will end with state CANCELED 2024-02-22 08:58:57,885 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Send cancel Job datamonitor (812617486796062721), Pipeline: [(2/2)], task: [pipeline-2 [Source[1]-MySQL-CDC]-SplitEnumerator (1/1)] operator to member [localhost]:5801 2024-02-22 08:58:57,885 INFO [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-27] - [localhost]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}) need cancel. 2024-02-22 08:58:57,885 INFO [o.a.s.e.s.d.p.SubPlan ] [seatunnel-coordinator-service-29] - Job datamonitor (812617486796062721), Pipeline: [(1/2)] turned from state CANCELING to CANCELED. 2024-02-22 08:58:57,885 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(2/2)], task: [pipeline-2 [Source[1]-MySQL-CDC]-SourceTask (1/1)] state process is start 2024-02-22 08:58:57,885 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}] - [localhost]:5801 [seatunnel] [5.1] Interrupted task 70000 - org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask@5468c66a 2024-02-22 08:58:57,885 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 70000, taskGroup = TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6} 2024-02-22 08:58:57,885 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}] - recycle classloader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader@47ab2e18 2024-02-22 08:58:57,886 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Job datamonitor (812617486796062721), Pipeline: [(2/2)], task: [pipeline-2 [Source[1]-MySQL-CDC]-SourceTask (1/1)] turned from state RUNNING to CANCELING. 2024-02-22 08:58:57,886 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [seatunnel-coordinator-service-27] - Send cancel Job datamonitor (812617486796062721), Pipeline: [(2/2)], task: [pipeline-2 [Source[1]-MySQL-CDC]-SourceTask (1/1)] operator to member [localhost]:5801 2024-02-22 08:58:57,886 INFO [o.a.s.e.s.TaskExecutionService] [seatunnel-coordinator-service-27] - [localhost]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}) need cancel. 2024-02-22 08:58:57,886 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - [localhost]:5801 [seatunnel] [5.1] Interrupted task 100000 - org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@22830a7f 2024-02-22 08:58:57,886 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 100000, taskGroup = TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000} 2024-02-22 08:58:57,886 WARN [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - [localhost]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@15b0e614 org.apache.seatunnel.common.utils.SeaTunnelException: java.lang.InterruptedException: sleep interrupted at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:166) ~[seatunnel-transforms-v2.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:92) ~[seatunnel-transforms-v2.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:110) ~[?:?] at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:112) ~[seatunnel-transforms-v2.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:110) ~[?:?] at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:150) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:643) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:944) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_144] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) ~[?:1.8.0_144] at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:164) ~[seatunnel-transforms-v2.jar:2.3.4-SNAPSHOT] ... 15 more 2024-02-22 08:58:57,886 INFO [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 90000, taskGroup = TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000} 2024-02-22 08:58:57,886 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader@77add642 2024-02-22 08:58:57,888 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}] - recycle classloader for thread BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6} 2024-02-22 08:58:57,888 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}] - recycle classloader for thread connection-pool-192.168.1.188:3306 housekeeper 2024-02-22 08:58:57,888 INFO [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-4] - [localhost]:5801 [seatunnel] [5.1] Task TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6} complete with state CANCELED 2024-02-22 08:58:57,888 INFO [.e.IncrementalSourceEnumerator] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}] - Closing enumerator... 2024-02-22 08:58:57,888 INFO [o.a.s.e.s.CoordinatorService ] [hz.main.seaTunnel.task.thread-4] - [localhost]:5801 [seatunnel] [5.1] Received task end from execution TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=6}, state CANCELED 2024-02-22 08:58:57,889 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader for thread I/O dispatcher 14 2024-02-22 08:58:57,889 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader for thread I/O dispatcher 13 2024-02-22 08:58:57,889 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader for thread I/O dispatcher 16 2024-02-22 08:58:57,889 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader for thread I/O dispatcher 15 2024-02-22 08:58:57,889 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader for thread I/O dispatcher 3 2024-02-22 08:58:57,889 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader for thread pool-17-thread-1 2024-02-22 08:58:57,889 INFO [o.a.s.e.c.l.ClassLoaderUtil ] [BlockingWorker-TaskGroupLocation{jobId=812617486796062721, pipelineId=2, taskGroupId=80000}] - recycle classloader for thread I/O dispatcher 9

github-actions[bot] commented 5 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.