search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.03k stars 1.82k forks source link

[Bug] [Zeta Engine] cluster service abnormal termination #7089

Closed leijian812 closed 4 months ago

leijian812 commented 4 months ago

Search before asking

What happened

The service stopped abnormally after running for a period of time. Manually restarting the server service still reported an error

SeaTunnel Version

dev

SeaTunnel Config

hazelcast.yaml
hazelcast:
  cluster-name: seatunnel
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_WRITE:
          enabled: true
        DATA:
          enabled: true
    join:
      tcp-ip:
        enabled: true
        member-list:
          - localhost
    port:
      auto-increment: false
      port: 5801
  properties:
    hazelcast.invocation.max.retry.count: 20
    hazelcast.tcp.join.port.try.count: 30
    hazelcast.logging.type: log4j2
    hazelcast.operation.generic.thread.count: 50 

hazelcast-client.yaml
hazelcast-client:
  cluster-name: seatunnel
  properties:
    hazelcast.logging.type: log4j2
  connection-strategy:
    connection-retry:
      cluster-connect-timeout-millis: 3000
  network:
    cluster-members:
      - localhost:5801

seatunnel.yaml
seatunnel:
  engine:
    history-job-expire-minutes: 1440
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: true
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          namespace: /tmp/seatunnel/checkpoint_snapshot
          storage.type: hdfs
          fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission

Running Command

sh bin/seatunnel-cluster.sh

Error Exception

2024-06-28 18:00:06,485 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-17] - wait checkpoint completed: 1239
2024-06-28 18:00:06,493 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-17] - pending checkpoint(1239/1@123456789) notify finished!
2024-06-28 18:00:06,493 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-17] - start notify checkpoint completed, job id: 123456789, pipeline id: 1, checkpoint id:1239
2024-06-28 18:00:06,952 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Running shutdown hook... Current state: ACTIVE
2024-06-28 18:00:06,952 INFO  [c.h.c.LifecycleService        ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is SHUTTING_DOWN
2024-06-28 18:00:06,955 WARN  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Terminating forcefully...
2024-06-28 18:00:06,955 INFO  [o.a.s.e.s.NodeExtension$1     ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Stopping text command service...
2024-06-28 18:00:06,956 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Shutting down connection manager...
2024-06-28 18:00:06,957 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Connection[id=4, /127.0.0.1:5801->/127.0.0.1:39713, qualifier=null, endpoint=[127.0.0.1]:39713, remoteUuid=629837b3-41b6-45b5-9d86-3ed47fe1a7d6, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: TcpServer is stopping
2024-06-28 18:00:06,958 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Shutting down node engine...
2024-06-28 18:00:06,970 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel] [5.1] Interrupted task 20000 - org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask@b87cc8a
2024-06-28 18:00:06,970 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] Interrupted task 50000 - org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@3c25df07
2024-06-28 18:00:06,970 INFO  [.c.c.DefaultClassLoaderService] [hz.ShutdownThread] - close classloader service
2024-06-28 18:00:06,970 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=1}
2024-06-28 18:00:06,970 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 50000, taskGroup = TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}
2024-06-28 18:00:06,971 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel] [5.1] The Node is not ready yet, Node state PASSIVE,looking forward to the next scheduling
2024-06-28 18:00:06,971 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@2d776696
org.apache.seatunnel.common.utils.SeaTunnelException: java.lang.InterruptedException: sleep interrupted
    at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:174) ~[seatunnel-transforms-v2.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:93) ~[seatunnel-transforms-v2.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119) ~[?:?]
    at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1005) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_312]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_312]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method) ~[?:1.8.0_312]
    at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:172) ~[seatunnel-transforms-v2.jar:2.3.6-SNAPSHOT]
    ... 13 more
2024-06-28 18:00:06,972 INFO  [o.a.s.e.s.TaskExecutionService] [event-forwarder-0] - [localhost]:5801 [seatunnel] [5.1] Event forward thread interrupted
2024-06-28 18:00:06,972 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}
2024-06-28 18:00:06,972 WARN  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}] - [localhost]:5801 [seatunnel] [5.1] The Node is not ready yet, Node state PASSIVE,looking forward to the next scheduling
2024-06-28 18:00:06,974 INFO  [.e.IncrementalSourceEnumerator] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=1}] - Closing enumerator...
2024-06-28 18:00:06,974 INFO  [a.s.c.s.c.s.r.SourceReaderBase] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}] - Closing Source Reader 0.
2024-06-28 18:00:06,974 INFO  [o.a.s.c.s.c.s.r.f.SplitFetcher] [BlockingWorker-TaskGroupLocation{jobId=123456789, pipelineId=1, taskGroupId=30000}] - Shutting down split fetcher 0
2024-06-28 18:00:06,979 WARN  [c.h.s.i.o.i.Invocation        ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Future.complete(Object) on completed future. Request: Invocation{op=org.apache.seatunnel.engine.server.operation.WaitForJobCompleteOperation{serviceName='st:impl:seaTunnelServer', identityHash=2071972437, partitionId=-1, replicaIndex=0, callId=-3405, invocationTime=1719556416082 (2024-06-28 14:33:36.082), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1719556416082, firstInvocationTime='2024-06-28 14:33:36.082', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[localhost]:5801, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=null}, current value: ExceptionalResult{cause=com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!}, offered value: ExceptionalResult{cause=java.lang.InterruptedException}
java.lang.Exception: null
    at com.hazelcast.spi.impl.AbstractInvocationFuture.warnIfSuspiciousDoubleCompletion(AbstractInvocationFuture.java:1253) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1230) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.completeExceptionally(Invocation.java:680) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyThrowable(Invocation.java:386) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyError(Invocation.java:330) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.sendResponse(Invocation.java:230) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.Operation.sendResponse(Operation.java:483) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.operation.AsyncOperation.doSendResponse(AsyncOperation.java:80) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.operation.AsyncOperation.lambda$run$0(AsyncOperation.java:59) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_312]
    at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.internalCompleteExceptionally(PassiveCompletableFuture.java:74) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.lambda$new$0(PassiveCompletableFuture.java:31) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_312]
    at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.internalCompleteExceptionally(PassiveCompletableFuture.java:74) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.lambda$new$0(PassiveCompletableFuture.java:31) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_312]
    at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.internalCompleteExceptionally(PassiveCompletableFuture.java:74) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.lambda$new$0(PassiveCompletableFuture.java:31) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_312]
    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_312]
    at org.apache.seatunnel.engine.server.master.JobMaster.interrupt(JobMaster.java:831) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_312]
    at org.apache.seatunnel.engine.server.CoordinatorService.clearCoordinatorService(CoordinatorService.java:401) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.CoordinatorService.shutdown(CoordinatorService.java:687) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.SeaTunnelServer.shutdown(SeaTunnelServer.java:151) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.servicemanager.impl.ServiceManagerImpl.shutdownService(ServiceManagerImpl.java:308) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.servicemanager.impl.ServiceManagerImpl.shutdown(ServiceManagerImpl.java:299) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.NodeEngineImpl.shutdown(NodeEngineImpl.java:549) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.instance.impl.Node.shutdownServices(Node.java:605) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.instance.impl.Node.shutdown(Node.java:544) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.instance.impl.LifecycleServiceImpl.shutdown(LifecycleServiceImpl.java:101) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.instance.impl.LifecycleServiceImpl.terminate(LifecycleServiceImpl.java:89) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.instance.impl.Node$NodeShutdownHookThread.run(Node.java:766) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
2024-06-28 18:00:06,987 INFO  [c.h.i.i.NodeExtension         ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Destroying node NodeExtension.
2024-06-28 18:00:06,987 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Hazelcast Shutdown is completed in 32 ms.
2024-06-28 18:00:06,988 INFO  [c.h.c.LifecycleService        ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is SHUTDOWN

restart the server service error:

2024-07-01 08:57:54,769 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2024-07-01 08:57:54,769 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following: 
  - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-07-01 08:57:54,771 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2024-07-01 08:57:55,179 INFO  [c.h.s.security                ] [main] - [localhost]:5801 [seatunnel] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2024-07-01 08:57:55,237 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel] [5.1] Using TCP/IP discovery
2024-07-01 08:57:55,238 WARN  [c.h.c.CPSubsystem             ] [main] - [localhost]:5801 [seatunnel] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2024-07-01 08:57:55,391 INFO  [.c.c.DefaultClassLoaderService] [main] - start classloader service
2024-07-01 08:57:55,417 WARN  [o.a.s.e.s.TaskExecutionService] [pool-2-thread-1] - [localhost]:5801 [seatunnel] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2024-07-01 08:57:55,419 INFO  [o.a.s.e.s.TaskExecutionService] [main] - [localhost]:5801 [seatunnel] [5.1] Created new BusWork : 2030707995
2024-07-01 08:57:55,430 WARN  [a.s.e.s.s.s.DefaultSlotService] [hz.main.seaTunnel.slotService.thread] - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
2024-07-01 08:57:55,432 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-6-thread-1] - [localhost]:5801 [seatunnel] [5.1] 
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2024-07-01 08:57:55,463 INFO  [c.h.i.d.Diagnostics           ] [main] - [localhost]:5801 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2024-07-01 08:57:55,469 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is STARTING
2024-07-01 08:57:55,487 INFO  [c.h.i.c.ClusterService        ] [main] - [localhost]:5801 [seatunnel] [5.1] 

Members {size:1, ver:1} [
    Member [localhost]:5801 - 037b0f75-c253-4efb-afc6-1cefccb98f6a this
]

2024-07-01 08:57:55,496 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is STARTED
2024-07-01 08:57:55,510 INFO  [c.h.i.d.HealthMonitor         ] [hz.main.HealthMonitor] - [localhost]:5801 [seatunnel] [5.1] processors=40, physical.memory.total=187.1G, physical.memory.free=109.1G, swap.space.total=4.0G, swap.space.free=4.0G, heap.memory.used=140.9M, heap.memory.free=1.9G, heap.memory.total=2.0G, heap.memory.max=2.0G, heap.memory.used/total=6.84%, heap.memory.used/max=6.84%, minor.gc.count=2, minor.gc.time=77ms, major.gc.count=0, major.gc.time=0ms, load.process=13.04%, load.system=18.92%, load.systemAverage=1.45, thread.count=132, thread.peakCount=132, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=1, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0
2024-07-01 08:57:55,531 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel] [5.1] This node become a new active master node, begin init coordinator service
2024-07-01 08:57:55,723 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel] [5.1] Loaded event handlers: [org.apache.seatunnel.api.event.LoggingEventHandler@3ad379bf]
2024-07-01 08:57:55,736 INFO  [.h.i.p.i.PartitionStateManager] [pool-5-thread-1] - [localhost]:5801 [seatunnel] [5.1] Initializing cluster partition table arrangement...
2024-07-01 08:57:55,757 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Running shutdown hook... Current state: ACTIVE
2024-07-01 08:57:55,757 INFO  [c.h.c.LifecycleService        ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is SHUTTING_DOWN
2024-07-01 08:57:55,759 WARN  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Terminating forcefully...
2024-07-01 08:57:55,759 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Shutting down connection manager...
2024-07-01 08:57:55,762 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Shutting down node engine...
2024-07-01 08:57:55,766 INFO  [.c.c.DefaultClassLoaderService] [hz.ShutdownThread] - close classloader service
2024-07-01 08:57:55,766 INFO  [o.a.s.e.s.TaskExecutionService] [event-forwarder-0] - [localhost]:5801 [seatunnel] [5.1] Event forward thread interrupted
2024-07-01 08:57:55,806 WARN  [c.h.s.i.o.i.Invocation        ] [pool-5-thread-1] - [localhost]:5801 [seatunnel] [5.1] Future.complete(Object) on completed future. Request: Invocation{op=com.hazelcast.map.impl.query.QueryOperation{serviceName='hz:impl:mapService', identityHash=1415507940, partitionId=-1, replicaIndex=0, callId=-2, invocationTime=1719795475727 (2024-07-01 08:57:55.727), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0, name=engine_runningJobInfo}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1719795475727, firstInvocationTime='2024-07-01 08:57:55.727', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[localhost]:5801, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=null}, current value: ExceptionalResult{cause=com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!}, offered value: com.hazelcast.map.impl.query.QueryResult@3a0a8522
java.lang.Exception: null
    at com.hazelcast.spi.impl.AbstractInvocationFuture.warnIfSuspiciousDoubleCompletion(AbstractInvocationFuture.java:1253) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1230) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.AbstractInvocationFuture.complete(AbstractInvocationFuture.java:1219) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.complete(Invocation.java:673) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.sendResponse(Invocation.java:236) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.Operation.sendResponse(Operation.java:483) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:283) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnTarget(OperationServiceImpl.java:365) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.query.QueryEngineImpl.dispatchFullQueryOnAllMembersOnQueryThread(QueryEngineImpl.java:315) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.query.QueryEngineImpl.dispatchFullQueryOnQueryThread(QueryEngineImpl.java:284) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.query.QueryEngineImpl.dispatchOnQueryThreads(QueryEngineImpl.java:171) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.query.QueryEngineImpl.doRunOnQueryThreads(QueryEngineImpl.java:164) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.query.QueryEngineImpl.runOnGivenPartitions(QueryEngineImpl.java:141) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.query.QueryEngineImpl.execute(QueryEngineImpl.java:109) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1415) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1383) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.proxy.MapProxyImpl.executePredicate(MapProxyImpl.java:796) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:745) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:739) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.CoordinatorService.initCoordinatorService(CoordinatorService.java:285) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at org.apache.seatunnel.engine.server.CoordinatorService.checkNewActiveMaster(CoordinatorService.java:384) ~[seatunnel-starter.jar:2.3.6-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_312]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_312]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_312]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_312]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2024-07-01 08:57:55,824 ERROR [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel] [5.1] com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
    at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:140)
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:570)
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541)
    at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241)
    at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnPartition(OperationServiceImpl.java:330)
    at com.hazelcast.map.impl.query.QueryEngineImpl.dispatchPartitionScanQueryOnOwnerMemberOnPartitionThread(QueryEngineImpl.java:341)
    at com.hazelcast.map.impl.query.QueryEngineImpl.lambda$dispatchPartitionScanQueryOnOwnerMemberOnPartitionThread$1(QueryEngineImpl.java:333)
    at java.util.PrimitiveIterator$OfInt.forEachRemaining(PrimitiveIterator.java:115)
    at com.hazelcast.map.impl.query.QueryEngineImpl.dispatchPartitionScanQueryOnOwnerMemberOnPartitionThread(QueryEngineImpl.java:332)
    at com.hazelcast.map.impl.query.QueryEngineImpl.doRunOnPartitionThreads(QueryEngineImpl.java:198)
    at com.hazelcast.map.impl.query.QueryEngineImpl.runOnGivenPartitions(QueryEngineImpl.java:144)
    at com.hazelcast.map.impl.query.QueryEngineImpl.execute(QueryEngineImpl.java:109)
    at com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1415)
    at com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1383)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.executePredicate(MapProxyImpl.java:796)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:745)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:739)
    at org.apache.seatunnel.engine.server.CoordinatorService.initCoordinatorService(CoordinatorService.java:285)
    at org.apache.seatunnel.engine.server.CoordinatorService.checkNewActiveMaster(CoordinatorService.java:384)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at ------ submitted from ------.()
    at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:336)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:112)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:609)
    at com.hazelcast.map.impl.query.QueryEngineImpl.addResultsOfPredicate(QueryEngineImpl.java:214)
    at com.hazelcast.map.impl.query.QueryEngineImpl.doRunOnPartitionThreads(QueryEngineImpl.java:200)
    at com.hazelcast.map.impl.query.QueryEngineImpl.runOnGivenPartitions(QueryEngineImpl.java:144)
    at com.hazelcast.map.impl.query.QueryEngineImpl.execute(QueryEngineImpl.java:109)
    at com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1415)
    at com.hazelcast.map.impl.proxy.MapProxySupport.executeQueryInternal(MapProxySupport.java:1383)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.executePredicate(MapProxyImpl.java:796)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:745)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:739)
    at org.apache.seatunnel.engine.server.CoordinatorService.initCoordinatorService(CoordinatorService.java:285)
    at org.apache.seatunnel.engine.server.CoordinatorService.checkNewActiveMaster(CoordinatorService.java:384)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

2024-07-01 08:57:58,773 INFO  [c.h.i.i.NodeExtension         ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Destroying node NodeExtension.
2024-07-01 08:57:58,773 INFO  [c.h.i.i.Node                  ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] Hazelcast Shutdown is completed in 3014 ms.
2024-07-01 08:57:58,773 INFO  [c.h.c.LifecycleService        ] [hz.ShutdownThread] - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is SHUTDOWN

Zeta or Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

image image

Are you willing to submit PR?

Code of Conduct

magelitedhp commented 4 months ago

As a Cainiao, I also encountered the same problem

chenqianwen commented 1 month ago

怎么解决的？

leijian812 commented 1 month ago

怎么解决的？

cluster服务启动的时候没有加 -d参数，控制台关闭后服务自动退出导致的