search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.01k stars 1.82k forks source link

[Bug] [Zeta] Local tasks are not allowed to run concurrently; only one can be running at a time. #4986

Open hanyouyou13 opened 1 year ago

hanyouyou13 commented 1 year ago

Search before asking

What happened

I would like to run the seatunnel locally multiple times （-e local）where only the first run is successful, and all subsequent runs fail.

SeaTunnel Version

2.3.2

SeaTunnel Config

env {
  execution.parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 2000
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
  FakeSource {
    parallelism = 2
    result_table_name = "fake"
    row.num = 16
    schema = {
      fields {
        name = "string"
        age = "int"
      }
    }
  }
}

sink {
  Console {
  }
}

Running Command

sh bin/seatunnel.sh -c config/v2.streaming.conf.template -e local

Error Exception

2023-06-28 12:53:09,962 INFO  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-06-28 12:53:09,963 WARN  com.hazelcast.logging.impl.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:263) - [localhost]:5802 [seatunnel-551475] [5.1] Received an invalid join request from [localhost]:5801, cause: members part of different cluster
2023-06-28 12:53:09,965 INFO  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-06-28 12:53:09,970 INFO  com.hazelcast.logging.impl.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:263) - [localhost]:5802 [seatunnel-551475] [5.1] Connection[id=5, /127.0.0.1:5802->/127.0.0.1:46982, qualifier=null, endpoint=[localhost]:5801, remoteUuid=e02c61f2-6c78-45f1-abd1-07b0b03b6e3b, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Connection closed by the other side
2023-06-28 12:53:09,991 INFO  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Trying to connect to cluster: seatunnel-551475
2023-06-28 12:53:09,993 INFO  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Trying to connect to [localhost]:5801
2023-06-28 12:53:10,013 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:43672->localhost/127.0.0.1:5801}, remoteAddress=null, lastReadTime=2023-06-28 12:53:10.008, lastWriteTime=2023-06-28 12:53:10.007, closedTime=2023-06-28 12:53:10.010, connected server version=null} closed. Reason: Failed to authenticate connection
com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.checkAuthenticationResponse(TcpClientConnectionManager.java:995) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onAuthenticated(TcpClientConnectionManager.java:898) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.getOrConnectToAddress(TcpClientConnectionManager.java:615) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.lambda$doConnectToCandidateCluster$3(TcpClientConnectionManager.java:505) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connect(TcpClientConnectionManager.java:459) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCandidateCluster(TcpClientConnectionManager.java:505) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:411) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connectToCluster(TcpClientConnectionManager.java:372) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:381) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.constructHazelcastClient(HazelcastClient.java:460) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:416) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:136) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.<init>(SeaTunnelHazelcastClient.java:53) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.engine.client.SeaTunnelClient.<init>(SeaTunnelClient.java:46) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:111) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.2]
2023-06-28 12:53:10,020 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Exception during initial connection to [localhost]:5801: com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
2023-06-28 12:53:10,021 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Unable to get live cluster connection, retry in 1000 ms, attempt: 1, cluster connect timeout: 3000 ms, max backoff: 30000 ms
2023-06-28 12:53:10,426 INFO  com.hazelcast.logging.impl.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:263) - [localhost]:5802 [seatunnel-551475] [5.1] This node become a new active master node, begin init coordinator service
2023-06-28 12:53:10,658 INFO  com.hazelcast.logging.impl.LoggingServiceImpl$DefaultLogger.log(LoggingServiceImpl.java:263) - [localhost]:5802 [seatunnel-551475] [5.1] Initializing cluster partition table arrangement...
2023-06-28 12:53:11,021 INFO  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Trying to connect to [localhost]:5801
2023-06-28 12:53:11,032 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] ClientConnection{alive=false, connectionId=2, channel=NioChannel{/127.0.0.1:54643->localhost/127.0.0.1:5801}, remoteAddress=null, lastReadTime=2023-06-28 12:53:11.028, lastWriteTime=2023-06-28 12:53:11.025, closedTime=2023-06-28 12:53:11.029, connected server version=null} closed. Reason: Failed to authenticate connection
com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.checkAuthenticationResponse(TcpClientConnectionManager.java:995) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onAuthenticated(TcpClientConnectionManager.java:898) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.getOrConnectToAddress(TcpClientConnectionManager.java:615) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.lambda$doConnectToCandidateCluster$3(TcpClientConnectionManager.java:505) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connect(TcpClientConnectionManager.java:459) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCandidateCluster(TcpClientConnectionManager.java:505) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:411) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connectToCluster(TcpClientConnectionManager.java:372) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:381) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.constructHazelcastClient(HazelcastClient.java:460) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:416) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:136) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.<init>(SeaTunnelHazelcastClient.java:53) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.engine.client.SeaTunnelClient.<init>(SeaTunnelClient.java:46) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:111) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.2]
2023-06-28 12:53:11,035 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Exception during initial connection to [localhost]:5801: com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
2023-06-28 12:53:11,036 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Unable to get live cluster connection, retry in 1050 ms, attempt: 2, cluster connect timeout: 3000 ms, max backoff: 30000 ms
2023-06-28 12:53:12,087 INFO  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Trying to connect to [localhost]:5801
2023-06-28 12:53:12,096 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] ClientConnection{alive=false, connectionId=3, channel=NioChannel{/127.0.0.1:35959->localhost/127.0.0.1:5801}, remoteAddress=null, lastReadTime=2023-06-28 12:53:12.094, lastWriteTime=2023-06-28 12:53:12.091, closedTime=2023-06-28 12:53:12.095, connected server version=null} closed. Reason: Failed to authenticate connection
com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.checkAuthenticationResponse(TcpClientConnectionManager.java:995) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onAuthenticated(TcpClientConnectionManager.java:898) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.getOrConnectToAddress(TcpClientConnectionManager.java:615) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.lambda$doConnectToCandidateCluster$3(TcpClientConnectionManager.java:505) ~[seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connect(TcpClientConnectionManager.java:459) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCandidateCluster(TcpClientConnectionManager.java:505) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:411) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connectToCluster(TcpClientConnectionManager.java:372) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:381) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.constructHazelcastClient(HazelcastClient.java:460) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:416) [seatunnel-starter.jar:2.3.2]
    at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:136) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.<init>(SeaTunnelHazelcastClient.java:53) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.engine.client.SeaTunnelClient.<init>(SeaTunnelClient.java:46) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:111) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.2]
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.2]
2023-06-28 12:53:12,099 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Exception during initial connection to [localhost]:5801: com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
2023-06-28 12:53:12,100 WARN  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Unable to get live cluster connection, retry in 892 ms, attempt: 3, cluster connect timeout: 3000 ms, max backoff: 30000 ms
2023-06-28 12:53:12,994 INFO  com.hazelcast.client.impl.clientside.ClientLoggingService$DefaultLogger.log(ClientLoggingService.java:104) - hz.client_1 [seatunnel-551475] [5.1] Trying to connect to [localhost]:5801

Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

ic4y commented 1 year ago

Currently, we can run multiple tasks in local mode using separate JVM instances. The issue likely stems from the hazelcast-client.yaml where cluster-members doesn't include multiple ports. I suggest configuring it like the image shown below:

image

With this, we can run up to 9 jobs simultaneously