search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.94k stars 1.79k forks source link

[Bug] [connector-jdbc] Specify offline mode to execute discovery as a streaming task #4656

Closed zengqinchris closed 1 year ago

zengqinchris commented 1 year ago

Search before asking

What happened

Specify offline mode to execute discovery as a streaming task, Perform approximately 10 tasks, which may occur once

SeaTunnel Version

v2.3.1

SeaTunnel Config

{
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "query" : " select id,use_name,use_pwd,load_ts from xxx where 1=1",
            "connection_check_timeout_sec" : 1000,
            "plugin_name" : "Jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/wcscreen"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "max_retries" : 3,
            "query" : " replace into xxx(id,use_name,use_pwd,load_ts) values (?,?,?,?) ",
            "plugin_name" : "jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/ydtest"
        }
    ]
}

Running Command

./bin/seatunnel.sh --config ./config/bb.conf

Error Exception

2023-04-24 15:41:34.041 INFO Program is substituting variables for you
2023-04-24 15:41:34.041 INFO Variables substitution ended successfully
2023-04-24 15:41:34.041 WARN The code you submit will not be limited by the limit
Job with jobId : nodeexecution_hadoop_ydstnnl_59 and execID : nodeexecution_hadoop_ydstnnl_59 submitted 
2023-04-24 15:41:34.041 INFO You have submitted a new job, script code (after variable substitution) is
************************************SCRIPT CODE************************************
{"taskId":9,"script":"/YdBigData/user/seatunnel/config/9_319_MySQL_MySQL.conf","scriptType":"prod","source":"MySQL","sink":"MySQL","taskName":"m2m4","remark":"r","createTime":"2023-04-20 20:44:34","publishTime":"2023-04-20 20:59:17","projectId":"319","flowId":336,"projectName":"lz_project","flowName":"m2m4","nodeKey":"3ea09f0d-abd8-449d-b2c1-d92040f571ad"}
************************************SCRIPT CODE************************************
2023-04-24 15:41:34.041 INFO Your job is accepted,  jobID is exec_id018014linkis-cg-entrancebigdata08:9104nodeexecution_hadoop_ydstnnl_59 and taskID is 1420 in ServiceInstance(linkis-cg-entrance, bigdata08:9104). Please wait it to be scheduled
job is scheduled.
2023-04-24 15:41:34.041 INFO Your job is Scheduled. Please wait it to run.
Your job is being scheduled by orchestrator.
2023-04-24 15:41:34.041 INFO job is running.
2023-04-24 15:41:34.041 INFO Your job is Running now. Please wait it to complete.
2023-04-24 15:41:34.041 INFO Job with jobGroupId : 1420 and subJobId : 1420 was submitted to Orchestrator.
2023-04-24 15:41:34.041 INFO Background is starting a new engine for you,execId astJob_136_codeExec_136 mark id is mark_136, it may take several seconds, please wait
2023-04-24 15:41:34.041 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, bigdata08:46624) /bigdata/datahorizon/application/linkis/dirs/hadoop/20230424/ydstnnl/14de5239-f2b5-41d5-98d1-e786df3979d1/logs
bigdata08:46624_0 >> /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/bin/seatunnel.sh --config  /appcom/tmp/ydstnnlConfDir/9_319_MySQL_MySQL.conf
JAVA_OPTS: -Dhazelcast.client.config=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast-client.yaml -Dseatunnel.config=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/seatunnel.yaml -Dhazelcast.config=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast.yaml -Dlog4j2.configurationFile=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/log4j2_client.properties -Dseatunnel.logs.path=/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/logs -Dseatunnel.logs.file_name=seatunnel-starter-client -Xms128m -Xmx512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/tmp/seatunnel-6639055162999374831 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/seatunnel/

Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/seatunnel.yaml
Apr 24, 2023 3:41:35 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
INFO: seatunnel.home is /home/hadoop
Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
Apr 24, 2023 3:41:35 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast.yaml
Apr 24, 2023 3:41:36 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
Apr 24, 2023 3:41:36 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/config/hazelcast-client.yaml
2023-04-24 15:41:36,678 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2023-04-24 15:41:36,753 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-04-24 15:41:36,754 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-04-24 15:41:36,777 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2023-04-24 15:41:36,780 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [bigdata04]:5801
2023-04-24 15:41:36,811 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-04-24 15:41:36,812 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [bigdata04]:5801:3ea41eb6-06cd-470a-94fe-f8c485cfa2f2, server version: 5.1, local address: /172.27.10.253:35535
2023-04-24 15:41:36,813 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-04-24 15:41:36,824 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1] 
Members [2] {
    Member [bigdata03]:5801 - 438295ac-d767-43ab-ba13-e79495639ecc
    Member [bigdata04]:5801 - 3ea41eb6-06cd-470a-94fe-f8c485cfa2f2
}
2023-04-24 15:41:36,826 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [bigdata03]:5801:438295ac-d767-43ab-ba13-e79495639ecc, server version: 5.1, local address: /172.27.10.253:53423
2023-04-24 15:41:36,852 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-04-24 15:41:36,892 INFO  org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/sqlite-jdbc-3.39.3.0.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/redshift-jdbc42-2.1.0.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/tablestore-jdbc-5.13.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ojdbc8-12.2.0.1.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/db2jcc-db2jcc4.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/postgresql-42.4.3.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/DmJdbcDriver18-8.1.2.141.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ali-phoenix-shaded-thin-client-5.2.5-HBase-2.x.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mssql-jdbc-9.2.1.jre8.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/terajdbc4-17.20.00.12.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mysql-connector-java-8.0.27.jar]
2023-04-24 15:41:36,898 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /appcom/tmp/ydstnnlConfDir/9_319_MySQL_MySQL.conf
2023-04-24 15:41:36,946 INFO  org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-04-24 15:41:36,989 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "query" : " select id,use_name,use_pwd,load_ts from xxx where 1=1",
            "connection_check_timeout_sec" : 1000,
            "plugin_name" : "Jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/wcscreen"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "max_retries" : 3,
            "query" : " replace into xxx(id,use_name,use_pwd,load_ts) values (?,?,?,?) ",
            "plugin_name" : "jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/ydtest"
        }
    ]
}
2023-04-24 15:41:37,008 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /appcom/tmp/ydstnnlConfDir/9_319_MySQL_MySQL.conf
2023-04-24 15:41:37,013 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "query" : " select id,use_name,use_pwd,load_ts from xxx where 1=1",
            "connection_check_timeout_sec" : 1000,
            "plugin_name" : "Jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/wcscreen"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "password" : "111",
            "driver" : "com.mysql.cj.jdbc.Driver",
            "max_retries" : 3,
            "query" : " replace into xxx(id,use_name,use_pwd,load_ts) values (?,?,?,?) ",
            "plugin_name" : "jdbc",
            "user" : "11",
            "url" : "jdbc:mysql://0.0.0.0:3306/ydtest"
        }
    ]
}
2023-04-24 15:41:37,021 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel
2023-04-24 15:41:37,031 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Jdbc at: file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar
2023-04-24 15:41:37,045 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Jdbc'} from path: file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2023-04-24 15:41:38,450 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource - The partition_column parameter is not configured, and the source parallelism is set to 1
2023-04-24 15:41:38,455 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel
2023-04-24 15:41:38,456 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: jdbc at: file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar
2023-04-24 15:41:38,457 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='jdbc'} from classpath
2023-04-24 15:41:38,496 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - start submit job, job id: 702786240566525958, with plugin jar [file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/sqlite-jdbc-3.39.3.0.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/redshift-jdbc42-2.1.0.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/tablestore-jdbc-5.13.9.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ojdbc8-12.2.0.1.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/db2jcc-db2jcc4.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/connectors/seatunnel/connector-jdbc-2.3.2-SNAPSHOT.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/postgresql-42.4.3.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/DmJdbcDriver18-8.1.2.141.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/ali-phoenix-shaded-thin-client-5.2.5-HBase-2.x.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mssql-jdbc-9.2.1.jre8.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/terajdbc4-17.20.00.12.jar, file:/opt/usdp-srv/srv/udp/2.0.0.0/seatunnel/plugins/jdbc/lib/mysql-connector-java-8.0.27.jar]
2023-04-24 15:41:38,666 WARN  org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
2023-04-24 15:42:38,666 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:41:38
Current Statistic Time    : 2023-04-24 15:42:38
***********************************************
2023-04-24 15:43:38,660 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:42:38
Current Statistic Time    : 2023-04-24 15:43:38
***********************************************
2023-04-24 15:44:38,660 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:43:38
Current Statistic Time    : 2023-04-24 15:44:38
***********************************************
2023-04-24 15:45:38,659 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:44:38
Current Statistic Time    : 2023-04-24 15:45:38
***********************************************
2023-04-24 15:46:38,661 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
***********************************************
           Job Progress Information
***********************************************
Job Id                    :  702786240566525958
Read Count So Far         :                   0
Write Count So Far        :                   0
Average Read Count        :                 0/s
Average Write Count       :                 0/s
Last Statistic Time       : 2023-04-24 15:45:38
Current Statistic Time    : 2023-04-24 15:46:38
***********************************************
2023-04-24 15:46:39,825 INFO  org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (702786240566525958) end with state FAILED
2023-04-24 15:46:39,826 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-04-24 15:46:39,832 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [bigdata04]:5801:3ea41eb6-06cd-470a-94fe-f8c485cfa2f2, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.27.10.253:35535->bigdata04/172.27.10.183:5801}, remoteAddress=[bigdata04]:5801, lastReadTime=2023-04-24 15:46:36.855, lastWriteTime=2023-04-24 15:46:36.855, closedTime=2023-04-24 15:46:39.828, connected server version=5.1}
2023-04-24 15:46:39,833 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [bigdata03]:5801:438295ac-d767-43ab-ba13-e79495639ecc, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/172.27.10.253:53423->bigdata03/172.27.10.152:5801}, remoteAddress=[bigdata03]:5801, lastReadTime=2023-04-24 15:46:39.812, lastWriteTime=2023-04-24 15:46:38.626, closedTime=2023-04-24 15:46:39.832, connected server version=5.1}
2023-04-24 15:46:39,833 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-04-24 15:46:39,835 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-04-24 15:46:39,835 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-04-24 15:46:39,836 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
===============================================================================
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error, 
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
2023-04-24 15:46:39,836 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 
2023-04-24 15:46:39,838 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:215)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:211)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$2(CheckpointCoordinator.java:201)
    at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
    at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1632)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: Execute given execution failed after retry 30 times
    at com.hazelcast.spi.impl.AbstractInvocationFuture.returnOrThrowWithJoinConventions(AbstractInvocationFuture.java:819)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.resolveAndThrowWithJoinConvention(AbstractInvocationFuture.java:835)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.join(AbstractInvocationFuture.java:553)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:250)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$1(CheckpointCoordinator.java:189)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
    ... 3 more
Caused by: java.lang.RuntimeException: Execute given execution failed after retry 30 times
    at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:75)
    at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85)
    at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
    at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.engine.server.exception.TaskGroupContextNotFoundException: task group TaskGroupLocation{jobId=702786240566525958, pipelineId=1, taskGroupId=1} not found.
    at org.apache.seatunnel.engine.server.TaskExecutionService.getExecutionContext(TaskExecutionService.java:168)
    at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:90)
    at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
    ... 9 more
    at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:119)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:174)
    ... 2 more

2023-04-24 15:46:39,838 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
===============================================================================
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:215)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:211)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$2(CheckpointCoordinator.java:201)
    at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
    at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1632)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: Execute given execution failed after retry 30 times
    at com.hazelcast.spi.impl.AbstractInvocationFuture.returnOrThrowWithJoinConventions(AbstractInvocationFuture.java:819)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.resolveAndThrowWithJoinConvention(AbstractInvocationFuture.java:835)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.join(AbstractInvocationFuture.java:553)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:250)
    at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$1(CheckpointCoordinator.java:189)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
    ... 3 more
Caused by: java.lang.RuntimeException: Execute given execution failed after retry 30 times
    at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:75)
    at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85)
    at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
    at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.engine.server.exception.TaskGroupContextNotFoundException: task group TaskGroupLocation{jobId=702786240566525958, pipelineId=1, taskGroupId=1} not found.
    at org.apache.seatunnel.engine.server.TaskExecutionService.getExecutionContext(TaskExecutionService.java:168)
    at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:90)
    at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
    ... 9 more
    at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:119)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:174)
    ... 2 more
2023-04-24 15:46:39,839 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal

Flink or Spark Version

no use flink or spark

Java or Scala Version

jdk8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

zengqinchris commented 1 year ago

If there are more than two seatunnel cluster nodes, this problem is easy to occur. If there are only a single seatunnel node, it will occur less frequently

EricJoy2048 commented 1 year ago

Hi, this is a bug in V2.3.1, we fixed it in https://github.com/apache/incubator-seatunnel/pull/4449

You can use dev branch for test.

zengqinchris commented 1 year ago

Thank you for your support and assistance. The bug in dev code testing is no longer present

zengqinchris commented 1 year ago

The bug in dev code fixed

zengqinchris commented 1 year ago

Thank you for your support

Thank you for your support ，fixed

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 year ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.