search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.86k stars 1.77k forks source link

[Bug] [zeta] seatunnel cluster failed to submit the osstoclickhouse task #4202

Closed john8628 closed 1 year ago

john8628 commented 1 year ago

Search before asking

What happened

firstly i setup three nodes of seatunnel-cluster (10..109;10...110,10...111) ; then i submit the etl task to the cluster ; then failed to run ;

SeaTunnel Version

dev

SeaTunnel Config

env {
  # You can set spark configuration here
  execution.parallelism = 8
  job.mode = "BATCH"
}
source {
      OSSJindoFile {
            path = "/adm/test/dt=20230112/label=country"
            bucket = "oss://test"
            type = "parquet"
            access_key="*"
            access_secret="*"
            endpoint="*.aliyuncs.com"
           result_table_name="fake"
           parallelism=3
          }
}
sink{
   ClickhouseFile {
        source_table_name="fake"
    host = "10.*.*.*:8123"
        database = "test"
        table = "table1"
        username = "default"
        password = ""
        clickhouse_local_path = "/etc/clickhouse local"
        node_free_password = true
    node_pass=[]
    }
}

Running Command

$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/oss2clickhousefile.template  > /tmp/client0217_4.log

Error Exception

2023-02-24 09:28:32,684 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-24 09:30:22,472 INFO  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Received auth from Connection[id=1, /10.8.4.111:5801->/10.8.4.111:42967, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:42967, remoteUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: e92a3cf5-1db6-4961-8d87-b6e7408c7469, client name: hz.client_1, client version: 5.1
2023-02-24 09:30:24,379 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681311931096629254, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=1, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:30:24,386 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681311931096629254, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=2, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:30:24,393 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681311931096629254, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=3, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:30:24,493 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1}
2023-02-24 09:30:24,508 INFO  org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: OssJindoFile
2023-02-24 09:30:24,533 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002}
2023-02-24 09:30:24,533 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000}
2023-02-24 09:30:24,543 INFO  org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0
2023-02-24 09:30:24,543 INFO  org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 2
2023-02-24 09:30:24,563 INFO  org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 2
2023-02-24 09:30:24,563 INFO  org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0
2023-02-24 09:30:32,683 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-24 09:30:48,285 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=1, /10.8.4.111:5801->/10.8.4.111:42967, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:42967, remoteUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-02-24 09:30:48,286 INFO  com.hazelcast.client.impl.ClientEndpointManager - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /10.8.4.111:5801->/10.8.4.111:42967, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:42967, remoteUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1677202222468, latest clientAttributes=lastStatisticsCollectionTime=1677202237517,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1677202222463,clientAddress=10.8.4.111,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=14666620928,os.freePhysicalMemorySize=4834521088,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=255350,os.openFileDescriptorCount=53,os.processCpuTime=7580000000,os.systemLoadAverage=0.97,os.totalPhysicalMemorySize=33019609088,os.totalSwapSpaceSize=0,runtime.availableProcessors=8,runtime.freeMemory=483194304,runtime.maxMemory=7339507712,runtime.totalMemory=581435392,runtime.uptime=16390,runtime.usedMemory=98241088, labels=[]}
2023-02-24 09:31:24,525 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000}) need cancel.
2023-02-24 09:31:24,525 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1}) need cancel.
2023-02-24 09:31:24,525 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1}
2023-02-24 09:31:24,525 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60000, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000}
2023-02-24 09:31:24,525 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70000, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000}
2023-02-24 09:31:24,525 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1} complete with state CANCELED
2023-02-24 09:31:24,526 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000} complete with state CANCELED
2023-02-24 09:31:24,526 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002}) need cancel.
2023-02-24 09:31:24,526 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60002, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002}
2023-02-24 09:31:24,526 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70002, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002}
2023-02-24 09:31:24,526 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002} complete with state CANCELED
2023-02-24 09:31:24,609 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 681311931096629254, slot: SlotProfile{worker=[10.8.4.111]:6800, slotID=2, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:31:24,610 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 681311931096629254, slot: SlotProfile{worker=[10.8.4.111]:6800, slotID=1, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:31:24,610 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 681311931096629254, slot: SlotProfile{worker=[10.8.4.111]:6800, slotID=3, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:32:21,815 INFO  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Received auth from Connection[id=2, /10.8.4.111:5801->/10.8.4.111:59371, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:59371, remoteUuid=6277892b-a6b9-4315-bd9f-73533697d2c8, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 6277892b-a6b9-4315-bd9f-73533697d2c8, client name: hz.client_1, client version: 5.1
2023-02-24 09:32:23,037 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=4, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:32:23,051 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=5, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:32:23,073 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}
2023-02-24 09:32:23,075 INFO  org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: OssJindoFile
2023-02-24 09:32:23,123 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}
2023-02-24 09:32:23,126 INFO  org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 2
2023-02-24 09:32:23,132 INFO  org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 2
2023-02-24 09:32:32,683 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-24 09:33:23,110 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}) need cancel.
2023-02-24 09:33:23,110 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}
2023-02-24 09:33:23,110 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} complete with state CANCELED
2023-02-24 09:33:23,113 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}) need cancel.
2023-02-24 09:33:23,113 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}
2023-02-24 09:33:23,113 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}
2023-02-24 09:33:23,113 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} complete with state CANCELED
2023-02-24 09:34:32,683 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-24 09:35:03,746 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=6, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:35:03,803 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}
2023-02-24 09:35:03,805 INFO  org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 1
2023-02-24 09:35:03,823 INFO  org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 1
2023-02-24 09:36:03,941 WARN  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=1, /10.8.4.111:44409->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Exception in Connection[id=1, /10.8.4.111:44409->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=true, connectionType=MEMBER, planeIndex=0], thread=hz.main.IO.thread-in-2
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_362]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_362]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_362]
    at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_362]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.8.0_362]
    at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:114) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
2023-02-24 09:36:03,942 INFO  com.hazelcast.internal.server.tcp.TcpServerConnector - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connecting to /10.8.4.110:6800, timeout: 0, bind-any: true
2023-02-24 09:36:03,943 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:43465 and /10.8.4.110:6800
2023-02-24 09:36:03,944 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:6800 and /10.8.4.110:43925
2023-02-24 09:36:03,947 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}) need cancel.
2023-02-24 09:36:03,947 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}
2023-02-24 09:36:03,947 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}
2023-02-24 09:36:03,947 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} complete with state CANCELED
2023-02-24 09:36:32,683 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-24 09:38:04,733 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=7, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:38:04,738 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=8, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:38:04,742 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=9, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:38:04,753 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=10, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:38:04,780 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}
2023-02-24 09:38:04,784 INFO  org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: OssJindoFile
2023-02-24 09:38:04,806 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3}
2023-02-24 09:38:04,808 INFO  org.apache.seatunnel.engine.server.task.SinkAggregatedCommitterTask - starting seatunnel sink aggregated committer task, sink name: ClickhouseFile
2023-02-24 09:38:04,809 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000}
2023-02-24 09:38:04,813 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}
2023-02-24 09:38:04,815 INFO  org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0
2023-02-24 09:38:04,817 INFO  org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 2
2023-02-24 09:38:04,831 INFO  org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 2
2023-02-24 09:38:04,831 INFO  org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0
2023-02-24 09:38:13,050 INFO  com.hazelcast.internal.diagnostics.HealthMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] processors=8, physical.memory.total=30.8G, physical.memory.free=4.2G, swap.space.total=0, swap.space.free=0, heap.memory.used=1.6G, heap.memory.free=7.5G, heap.memory.total=9.1G, heap.memory.max=14.2G, heap.memory.used/total=17.61%, heap.memory.used/max=11.30%, minor.gc.count=2, minor.gc.time=39ms, major.gc.count=2, major.gc.time=255ms, load.process=50.00%, load.system=75.29%, load.systemAverage=0.66, thread.count=68, thread.peakCount=77, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=15068, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=13, clientEndpoint.count=1, connection.active.count=0, client.connection.count=0, connection.count=0
2023-02-24 09:38:32,683 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-24 09:38:33,057 INFO  com.hazelcast.internal.diagnostics.HealthMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] processors=8, physical.memory.total=30.8G, physical.memory.free=4.2G, swap.space.total=0, swap.space.free=0, heap.memory.used=1.6G, heap.memory.free=7.5G, heap.memory.total=9.1G, heap.memory.max=14.2G, heap.memory.used/total=17.69%, heap.memory.used/max=11.35%, minor.gc.count=2, minor.gc.time=39ms, major.gc.count=2, major.gc.time=255ms, load.process=80.00%, load.system=75.47%, load.systemAverage=2.19, thread.count=67, thread.peakCount=77, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=15406, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=13, clientEndpoint.count=1, connection.active.count=0, client.connection.count=0, connection.count=0
2023-02-24 09:38:53,059 INFO  com.hazelcast.internal.diagnostics.HealthMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] processors=8, physical.memory.total=30.8G, physical.memory.free=4.1G, swap.space.total=0, swap.space.free=0, heap.memory.used=1.6G, heap.memory.free=7.5G, heap.memory.total=9.1G, heap.memory.max=14.2G, heap.memory.used/total=17.80%, heap.memory.used/max=11.42%, minor.gc.count=2, minor.gc.time=39ms, major.gc.count=2, major.gc.time=255ms, load.process=0.00%, load.system=75.37%, load.systemAverage=3.20, thread.count=67, thread.peakCount=77, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=15709, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=13, clientEndpoint.count=1, connection.active.count=0, client.connection.count=0, connection.count=0
2023-02-24 09:39:04,820 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}) need cancel.
2023-02-24 09:39:04,820 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}
2023-02-24 09:39:04,820 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} complete with state CANCELED
2023-02-24 09:39:04,820 WARN  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Exception in Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=true, connectionType=MEMBER, planeIndex=0], thread=hz.main.seaTunnel.task.thread-8
java.nio.channels.ClosedByInterruptException: null
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_362]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477) ~[?:1.8.0_362]
    at com.hazelcast.internal.networking.nio.NioOutboundPipeline.flushToSocket(NioOutboundPipeline.java:439) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioOutboundPipeline.process(NioOutboundPipeline.java:324) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioOutboundPipeline.executePipeline(NioOutboundPipeline.java:240) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioOutboundPipeline.write(NioOutboundPipeline.java:218) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioChannel.write(NioChannel.java:79) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.server.tcp.TcpServerConnection.write(TcpServerConnection.java:222) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.OutboundOperationHandler.send(OutboundOperationHandler.java:59) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeRemote(Invocation.java:612) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:582) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:341) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_362]
    at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
    at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
2023-02-24 09:39:04,823 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000}) need cancel.
2023-02-24 09:39:04,823 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000}
2023-02-24 09:39:04,823 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000}
2023-02-24 09:39:04,823 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000} complete with state CANCELED
2023-02-24 09:39:04,824 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:6800 and /10.8.4.110:37786
2023-02-24 09:39:04,824 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:36663 and /10.8.4.110:6800
2023-02-24 09:39:04,921 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3}) need cancel.
2023-02-24 09:39:04,921 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3}
2023-02-24 09:39:04,922 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3} complete with state CANCELED
2023-02-24 09:39:04,923 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}) need cancel.
2023-02-24 09:39:04,923 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}
2023-02-24 09:39:04,923 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}
2023-02-24 09:39:04,923 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} complete with state CANCELED
2023-02-24 09:40:32,683 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-24 09:41:05,704 INFO  com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Invocations:1 timeouts:1 backup-timeouts:0
2023-02-24 09:41:05,710 WARN  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]}
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:121)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617)
    at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343)
    at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322)
    at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172)
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
    at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697)
    at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]}
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.newOperationTimeoutException(InvocationFuture.java:194)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:136)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:99)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617)
    at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343)
    at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322)
    at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172)
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
    at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697)
    at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
    at ------ submitted from ------.()
    at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:336)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:112)
    ... 16 more

2023-02-24 09:41:05,710 WARN  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] notify the job of the task(TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}) status failed, retry in 1000 millis
2023-02-24 09:41:05,710 ERROR org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]}
java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]}
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:121) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_362]
    at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
    at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]}
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.newOperationTimeoutException(InvocationFuture.java:194) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:136) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:99) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_362]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_362]
    at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
    at ------ submitted from ------.() ~[?:?]
    at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:336) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:112) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    ... 16 more
2023-02-24 09:41:05,739 INFO  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=11, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'}
2023-02-24 09:41:05,785 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}
2023-02-24 09:41:05,786 INFO  org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 1
2023-02-24 09:41:05,790 INFO  org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 1
2023-02-24 09:42:05,824 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}) need cancel.
2023-02-24 09:42:05,824 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}
2023-02-24 09:42:05,824 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}
2023-02-24 09:42:05,824 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} complete with state CANCELED
2023-02-24 09:42:06,128 WARN  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=8, /10.8.4.111:36663->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Exception in Connection[id=8, /10.8.4.111:36663->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=true, connectionType=MEMBER, planeIndex=0], thread=hz.main.IO.thread-in-2
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_362]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_362]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_362]
    at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_362]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.8.0_362]
    at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:114) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
    at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT]
2023-02-24 09:42:06,129 INFO  com.hazelcast.internal.server.tcp.TcpServerConnector - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connecting to /10.8.4.110:6800, timeout: 0, bind-any: true
2023-02-24 09:42:06,130 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:39939 and /10.8.4.110:6800
2023-02-24 09:42:32,683 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1]

Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 year ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.