search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.93k stars 1.79k forks source link

使用Docker配置集群模式，一个master，两个worker节点，docker2.3.8提交任务的时候，报找不到配置文件 #7878

Open FN20200222 opened 1 day ago

FN20200222 commented 1 day ago

Search before asking

What happened

docker2.3.8提交任务的时候，报找不到配置文件，我已经把配置文件cp到seatunnel_master config目录下 11 22

SeaTunnel Version

docker 2.3.8

SeaTunnel Config

{
    env {
      parallelism = 1
      job.mode = "STREAMING"
      checkpoint.interval = 10000
    }

    source {
      MySQL-CDC {
        base-url = "jdbc:mysql://47.243.170.250:3306/my_test_a"
        username = "root"
        password = "FN920127_z"
        table-names = ["my_test_a.test"]

        startup.mode = "initial"
      }
    }

    sink {
      Console {
      }
    }
}

Running Command

docker run --name seatunnel_client --network seatunnel-network -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 --rm apache/seatunnel ./bin/seatunnel.sh -c /opt/seatunnel/config/mysqlcdc_console.config

Error Exception

C:\Users\lenovo>docker run --name seatunnel_client --network seatunnel-network -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 --rm apache/seatunnel ./bin/seatunnel.sh -c /opt/seatunnel/config/mysqlcdc_console.config
2024-10-20 08:23:54,292 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2024-10-20 08:23:54,296 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2024-10-20 08:23:54,298 INFO  [o.a.s.e.c.c.SeaTunnelConfig   ] [main] - seatunnel.home is /opt/seatunnel
2024-10-20 08:23:54,378 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2024-10-20 08:23:54,378 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2024-10-20 08:23:54,654 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
2024-10-20 08:23:54,654 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast-client.yaml
2024-10-20 08:23:54,877 INFO  [.c.i.s.ClientInvocationService] [main] - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2024-10-20 08:23:54,931 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2024-10-20 08:23:54,932 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2024-10-20 08:23:54,955 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2024-10-20 08:23:54,958 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to [172.18.0.2]:5801
2024-10-20 08:23:54,991 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2024-10-20 08:23:54,992 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:53706372-027d-446a-a233-fd0b0b05912e, server version: 5.1, local address: /172.18.0.5:38287
2024-10-20 08:23:54,994 INFO  [c.h.i.d.Diagnostics           ] [main] - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2024-10-20 08:23:55,008 INFO  [c.h.c.i.s.ClientClusterService] [hz.client_1.event-1] - hz.client_1 [seatunnel] [5.1]

Members [3] {
        Member [localhost]:5801 - 53706372-027d-446a-a233-fd0b0b05912e
        Member [172.18.0.3]:5801 - 19bc49ee-953e-4ad0-9298-aa982814637b worker
        Member [172.18.0.4]:5801 - 56591070-b39f-4c54-ba52-ebbb3dc61a5a worker
}

2024-10-20 08:23:55,011 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Authenticated with server [172.18.0.3]:5801:19bc49ee-953e-4ad0-9298-aa982814637b, server version: 5.1, local address: /172.18.0.5:34667
2024-10-20 08:23:55,018 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Authenticated with server [172.18.0.4]:5801:56591070-b39f-4c54-ba52-ebbb3dc61a5a, server version: 5.1, local address: /172.18.0.5:59851
2024-10-20 08:23:55,050 INFO  [.c.i.s.ClientStatisticsService] [main] - Client statistics is enabled with period 5 seconds.
2024-10-20 08:23:55,164 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2024-10-20 08:23:55,167 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [172.18.0.3]:5801:19bc49ee-953e-4ad0-9298-aa982814637b, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/172.18.0.5:34667->/172.18.0.3:5801}, remoteAddress=[172.18.0.3]:5801, lastReadTime=2024-10-20 08:23:55.039, lastWriteTime=2024-10-20 08:23:55.038, closedTime=2024-10-20 08:23:55.165, connected server version=5.1}
2024-10-20 08:23:55,169 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:53706372-027d-446a-a233-fd0b0b05912e, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.18.0.5:38287->/172.18.0.2:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-10-20 08:23:55.088, lastWriteTime=2024-10-20 08:23:55.087, closedTime=2024-10-20 08:23:55.168, connected server version=5.1}
2024-10-20 08:23:55,172 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [172.18.0.4]:5801:56591070-b39f-4c54-ba52-ebbb3dc61a5a, connection: ClientConnection{alive=false, connectionId=3, channel=NioChannel{/172.18.0.5:59851->/172.18.0.4:5801}, remoteAddress=[172.18.0.4]:5801, lastReadTime=2024-10-20 08:23:55.042, lastWriteTime=2024-10-20 08:23:55.041, closedTime=2024-10-20 08:23:55.170, connected server version=5.1}
2024-10-20 08:23:55,173 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2024-10-20 08:23:55,176 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2024-10-20 08:23:55,177 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2024-10-20 08:23:55,177 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -

===============================================================================

2024-10-20 08:23:55,177 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error,

2024-10-20 08:23:55,177 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-10-20 08:23:55,180 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed

2024-10-20 08:23:55,182 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-22], ErrorDescription:[SeaTunnel read file '/opt/seatunnel/config/mysqlcdc_console.config' failed, because it not existed.]
        at org.apache.seatunnel.common.exception.CommonError.fileNotExistFailed(CommonError.java:86)
        at org.apache.seatunnel.core.starter.utils.FileUtils.checkConfigExist(FileUtils.java:66)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:133)
        ... 2 more

2024-10-20 08:23:55,183 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -
===============================================================================

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-22], ErrorDescription:[SeaTunnel read file '/opt/seatunnel/config/mysqlcdc_console.config' failed, because it not existed.]
        at org.apache.seatunnel.common.exception.CommonError.fileNotExistFailed(CommonError.java:86)
        at org.apache.seatunnel.core.starter.utils.FileUtils.checkConfigExist(FileUtils.java:66)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:133)
        ... 2 more

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

liunaijie commented 8 hours ago

The config file is read from client node, your screenshot is on master node, did you also copy/mount file to the client node?

mount file into this container

docker run --name seatunnel_client --network seatunnel-network -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 --rm apache/seatunnel ./bin/seatunnel.sh -c /opt/seatunnel/config/mysqlcdc_console.config
FN20200222 commented 6 hours ago

The config file is read from client node, your screenshot is on master node, did you also copy/mount file to the client node?

mount file into this container

docker run --name seatunnel_client --network seatunnel-network -e ST_DOCKER_MEMBER_LIST=172.18.0.2:5801 --rm apache/seatunnel ./bin/seatunnel.sh -c /opt/seatunnel/config/mysqlcdc_console.config

yes,master and client node all copy,still report this error