search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.81k stars 1.76k forks source link

[Bug] [Seatunnel Engine] Unable to get S3 Source in local mode #4129

Open xiehanghang opened 1 year ago

xiehanghang commented 1 year ago

Search before asking

What happened

I'm fetching data from S3 with Seatunnel local engine, but I'm facing the following issue,

The log is repeatedly reported ↓

2023-02-15 09:22:23,813 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

SeaTunnel Version

2.3.0

SeaTunnel Config

env {
  execution.parallelism = 1
  job.mode = "STREAMING"
}

source {
  S3File {
        path = "/Database/dfs/news_daily_202302021200"
        type = "parquet"
        access_key = "xxxx"
        secret_key = "xxxx"
        bucket = "s3a://com.xxx.data.prd"
      }

}

transform {

}

sink {
  Console{}
}

Running Command

./bin/seatunnel.sh --config ./config/seatunnel.streaming.conf.template -e local

Error Exception

JAVA_OPTS:  -Dhazelcast.client.config=/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast-client.yaml -Dseatunnel.config=/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/seatunnel.yaml -Dhazelcast.config=/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast.yaml -Dlog4j2.configurationFile=/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/log4j2_client.properties -Dseatunnel.logs.path=/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/logs -Dseatunnel.logs.file_name=seatunnel-starter-client-1676424120000
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/seatunnel.yaml' from System property 'seatunnel.config'
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/seatunnel.yaml
二月 15, 2023 9:22:02 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig
信息: seatunnel.home is /Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast.yaml' from System property 'hazelcast.config'
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast.yaml
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/seatunnel.yaml' from System property 'seatunnel.config'
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/seatunnel.yaml
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast.yaml' from System property 'hazelcast.config'
二月 15, 2023 9:22:02 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast.yaml
2023-02-15 09:22:02,989 WARN  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel_default_cluster-171793] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2023-02-15 09:22:02,994 INFO  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel_default_cluster-171793] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1, 0:0:0:0:0:0:0:1]
2023-02-15 09:22:02,994 INFO  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel_default_cluster-171793] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1]
2023-02-15 09:22:03,023 INFO  org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start...
2023-02-15 09:22:03,024 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2023-02-15 09:22:03,024 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Cluster name: seatunnel_default_cluster-171793
2023-02-15 09:22:03,024 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1]

 _____               _____                             _
/  ___|             |_   _|                           | |
\ `--.   ___   __ _   | |   _   _  _ __   _ __    ___ | |
 `--. \ / _ \ / _` |  | |  | | | || '_ \ | '_ \  / _ \| |
/\__/ /|  __/| (_| |  | |  | |_| || | | || | | ||  __/| |
\____/  \___| \__,_|  \_/   \__,_||_| |_||_| |_| \___||_|

2023-02-15 09:22:03,024 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2023-02-15 09:22:03,024 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:
  - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-02-15 09:22:03,027 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-02-15 09:22:03,583 INFO  com.hazelcast.system.security - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2023-02-15 09:22:03,639 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Using TCP/IP discovery
2023-02-15 09:22:03,639 WARN  com.hazelcast.cp.CPSubsystem - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2023-02-15 09:22:03,802 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Created new BusWork : 1315908533
2023-02-15 09:22:03,806 WARN  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
2023-02-15 09:22:03,807 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-15 09:22:03,814 WARN  org.apache.seatunnel.common.utils.RetryUtils - Failed to execute due to null. Retrying attempt (1/20) after backoff of 1000 ms
2023-02-15 09:22:03,855 INFO  com.hazelcast.internal.diagnostics.Diagnostics - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-02-15 09:22:03,869 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] [localhost]:5801 is STARTING
2023-02-15 09:22:03,911 INFO  com.hazelcast.internal.cluster.ClusterService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1]

Members {size:1, ver:1} [
    Member [localhost]:5801 - db486095-d8c6-4073-9a10-d5339a429d4f this
]

2023-02-15 09:22:03,921 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] [localhost]:5801 is STARTED
二月 15, 2023 9:22:03 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
二月 15, 2023 9:22:03 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/config/hazelcast-client.yaml
2023-02-15 09:22:03,997 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel_default_cluster-171793] [5.1] Running with 2 response threads, dynamic=true
2023-02-15 09:22:04,017 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel_default_cluster-171793] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-02-15 09:22:04,017 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel_default_cluster-171793] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-02-15 09:22:04,026 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel_default_cluster-171793] [5.1] Trying to connect to cluster: seatunnel_default_cluster-171793
2023-02-15 09:22:04,028 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel_default_cluster-171793] [5.1] Trying to connect to [localhost]:5801
2023-02-15 09:22:04,052 INFO  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:54634, qualifier=null, endpoint=[127.0.0.1]:54634, remoteUuid=df25825c-3c65-49a9-9c36-921735b759bd, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: df25825c-3c65-49a9-9c36-921735b759bd, client name: hz.client_1, client version: 5.1
2023-02-15 09:22:04,057 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel_default_cluster-171793] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-02-15 09:22:04,057 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel_default_cluster-171793] [5.1] Authenticated with server [localhost]:5801:db486095-d8c6-4073-9a10-d5339a429d4f, server version: 5.1, local address: /127.0.0.1:54634
2023-02-15 09:22:04,058 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel_default_cluster-171793] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-02-15 09:22:04,065 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel_default_cluster-171793] [5.1]

Members [1] {
    Member [localhost]:5801 - db486095-d8c6-4073-9a10-d5339a429d4f
}

2023-02-15 09:22:04,080 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-02-15 09:22:04,213 INFO  org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/aws-java-sdk-s3-1.11.1000.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/aws-java-sdk-bundle-1.12.262.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/hadoop-client-2.10.2.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/hadoop-aws-3.3.4.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/hadoop-aws-2.10.2.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/jdbc/lib/postgresql-42.5.2.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/hadoop-common-3.3.4.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/hadoop-client-3.3.4.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/aws-java-sdk-core-1.11.1000.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/hadoop-shaded-guava-1.1.1.jar, file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/plugins/hadoop-common-2.10.2.jar]
2023-02-15 09:22:04,215 INFO  org.apache.seatunnel.core.starter.config.ConfigBuilder - Loading config file: ./config/seatunnel.streaming.conf.template
2023-02-15 09:22:04,249 INFO  org.apache.seatunnel.core.starter.config.ConfigBuilder - parsed config file: {
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "STREAMING"
    },
    "source" : [
        {
            "bucket" : "s3a://com.xxxx.data.prd",
            "hadoop_s3_properties" : {
                "fs.s3a.aws.credentials.provider" : "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
            },
            "path" : "/Database/dfs/news_daily_202302021200",
            "secret_key" : "3doxxxxxxxxOH3W",
            "access_key" : "AKxxxxx",
            "type" : "parquet",
            "plugin_name" : "S3File"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "plugin_name" : "Console"
        }
    ]
}

2023-02-15 09:22:04,269 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/connectors/seatunnel
2023-02-15 09:22:04,286 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: S3File at: file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/connectors/seatunnel/connector-file-s3-2.3.0.jar
2023-02-15 09:22:04,300 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='S3File'} from path: file:/Users/xiehang/Desktop/developer/apache-seatunnel-incubating-2.3.0/connectors/seatunnel/connector-file-s3-2.3.0.jar use classloader: sun.misc.Launcher$AppClassLoader
2023-02-15 09:22:04,567 WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-02-15 09:22:04,820 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] This node become a new active master node, begin init coordinator service
2023-02-15 09:22:04,841 WARN  org.apache.hadoop.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2023-02-15 09:22:04,875 INFO  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
2023-02-15 09:22:04,875 INFO  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - s3a-file-system metrics system started
2023-02-15 09:22:05,002 INFO  com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1] Initializing cluster partition table arrangement...
2023-02-15 09:22:05,880 INFO  org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - received new worker register: [localhost]:5801
2023-02-15 09:22:13,818 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-02-15 09:22:23,813 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel_default_cluster-171793] [5.1]
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

TyrantLucifer commented 1 year ago

Change the job.mode from STEAMING to BATCH

xiehanghang commented 1 year ago

Change the job.mode from STEAMING to BATCH

After modifying the job.mode I got another error about package dependency. I want to know which jar packages I need to get the S3 source. this is the jar package I put in the lib directory

aws-java-sdk-bundle-1.12.262.jar                hadoop-aws-2.10.2.jar                           hadoop-shaded-guava-1.1.1.jar
aws-java-sdk-core-1.11.1000.jar                 hadoop-client-2.10.2.jar                        
aws-java-sdk-s3-1.11.1000.jar                   hadoop-common-2.10.2.jar                        seatunnel-hadoop3-3.1.4-uber-2.3.0-optional.jar
Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.http.HttpResponse.getHttpRequest()Lorg/apache/http/client/methods/HttpRequestBase;
    at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:57)
    at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:29)
    at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1519)
    at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:169)
    at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:305)
    at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:330)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at org.apache.seatunnel.shade.connector.file.org.apache.parquet.io.DelegatingSeekableInputStream.read(DelegatingSeekableInputStream.java:61)
    at org.apache.seatunnel.shade.connector.file.org.apache.parquet.bytes.BytesUtils.readIntLittleEndian(BytesUtils.java:83)
    at org.apache.seatunnel.shade.connector.file.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:548)
    at org.apache.seatunnel.shade.connector.file.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:777)
    at org.apache.seatunnel.shade.connector.file.org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:646)
    at org.apache.seatunnel.connectors.seatunnel.file.source.reader.ParquetReadStrategy.getSeaTunnelRowTypeInfo(ParquetReadStrategy.java:209)
    at org.apache.seatunnel.connectors.seatunnel.file.s3.source.S3FileSource.prepare(S3FileSource.java:96)
    at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:60)
    at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:314)
    at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:125)
    at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:129)
    at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:121)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:91)
    at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:31)
TyrantLucifer commented 1 year ago

https://seatunnel.apache.org/docs/2.3.0/connector-v2/source/S3File

xiehanghang commented 1 year ago

https://seatunnel.apache.org/docs/2.3.0/connector-v2/source/S3File

This guide only adds two jar packages( hadoop-aws.jar、hadoop-client.jar), but it is not enough to actually run it, the following are the packages I have added to {seatunnel_home}/lib or {seatunnel_home}/plugins

aws-java-sdk-bundle-1.12.262.jar                hadoop-aws-2.10.2.jar                           hadoop-shaded-guava-1.1.1.jar
aws-java-sdk-core-1.11.1000.jar                 hadoop-client-2.10.2.jar                        
aws-java-sdk-s3-1.11.1000.jar                   hadoop-common-2.10.2.jar                        seatunnel-hadoop3-3.1.4-uber-2.3.0-optional.jar
TyrantLucifer commented 1 year ago

cc @EricJoy2048 @CalvinKirs

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

zhangzhaohuazai commented 1 year ago

Have you solved it yet?

YuriyGavrilov commented 9 months ago

Hi, has nearly the same error with s3 test.

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2023-11-29 23:06:48,734 WARN  org.apache.seatunnel.core.starter.seatunnel.args.ClientCommandArgs$MasterTypeValidator - 
******************************************************************************************
-e and --deploy-mode deprecated in 2.3.1, please use -m and --master instead of it
******************************************************************************************
нояб. 29, 2023 11:06:48 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
нояб. 29, 2023 11:06:48 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/seatunnel.yaml
нояб. 29, 2023 11:06:48 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
INFO: seatunnel.home is /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3
нояб. 29, 2023 11:06:48 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
нояб. 29, 2023 11:06:48 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/hazelcast.yaml
нояб. 29, 2023 11:06:49 PM com.hazelcast.instance.impl.HazelcastInstanceFactory
WARNING: Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
 --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
2023-11-29 23:06:49,150 WARN  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-611685] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2023-11-29 23:06:49,150 INFO  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-611685] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1, 0:0:0:0:0:0:0:1]
2023-11-29 23:06:49,151 INFO  com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-611685] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1]
2023-11-29 23:06:49,205 INFO  org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start...
2023-11-29 23:06:49,207 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel-611685] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2023-11-29 23:06:49,207 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel-611685] [5.1] Cluster name: seatunnel-611685
2023-11-29 23:06:49,207 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel-611685] [5.1] 

 _____               _____                             _ 
/  ___|             |_   _|                           | |
\ `--.   ___   __ _   | |   _   _  _ __   _ __    ___ | |
 `--. \ / _ \ / _` |  | |  | | | || '_ \ | '_ \  / _ \| |
/\__/ /|  __/| (_| |  | |  | |_| || | | || | | ||  __/| |
\____/  \___| \__,_|  \_/   \__,_||_| |_||_| |_| \___||_|

2023-11-29 23:06:49,207 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel-611685] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2023-11-29 23:06:49,207 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel-611685] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following: 
  - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-11-29 23:06:49,210 INFO  com.hazelcast.system - [localhost]:5801 [seatunnel-611685] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-11-29 23:06:49,613 INFO  com.hazelcast.system.security - [localhost]:5801 [seatunnel-611685] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2023-11-29 23:06:49,683 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-611685] [5.1] Using TCP/IP discovery
2023-11-29 23:06:49,684 WARN  com.hazelcast.cp.CPSubsystem - [localhost]:5801 [seatunnel-611685] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
нояб. 29, 2023 11:06:49 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
нояб. 29, 2023 11:06:49 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/seatunnel.yaml
нояб. 29, 2023 11:06:49 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
нояб. 29, 2023 11:06:49 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/hazelcast.yaml
2023-11-29 23:06:49,872 INFO  org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-611685] [5.1] Created new BusWork : 2008821270
2023-11-29 23:06:49,872 WARN  org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-611685] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2023-11-29 23:06:49,908 WARN  org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
2023-11-29 23:06:49,913 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-611685] [5.1] 
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   0
corePoolSize              :                   0
maximumPoolSize           :          2147483647
poolSize                  :                   0
completedTaskCount        :                   0
taskCount                 :                   0
***********************************************

2023-11-29 23:06:49,941 INFO  com.hazelcast.internal.diagnostics.Diagnostics - [localhost]:5801 [seatunnel-611685] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-11-29 23:06:49,946 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5801 is STARTING
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5820 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5825 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5822 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5815 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5804 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5819 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5810 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5817 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5808 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5813 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5802 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5821 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5823 is added to the blacklist.
2023-11-29 23:06:49,992 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5812 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5829 is added to the blacklist.
2023-11-29 23:06:49,992 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5811 is added to the blacklist.
2023-11-29 23:06:49,992 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5807 is added to the blacklist.
2023-11-29 23:06:49,992 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5809 is added to the blacklist.
2023-11-29 23:06:49,991 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5806 is added to the blacklist.
2023-11-29 23:06:49,992 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5803 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5828 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5824 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5826 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5814 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5816 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5805 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5818 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5827 is added to the blacklist.
2023-11-29 23:06:49,993 INFO  com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5830 is added to the blacklist.
2023-11-29 23:06:50,987 INFO  com.hazelcast.internal.cluster.ClusterService - [localhost]:5801 [seatunnel-611685] [5.1] 

Members {size:1, ver:1} [
    Member [localhost]:5801 - 17cac56e-7734-452f-a5e9-9a0a20dd0cc1 this
]

2023-11-29 23:06:50,997 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5801 is STARTED
нояб. 29, 2023 11:06:50 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
нояб. 29, 2023 11:06:50 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
2023-11-29 23:06:51,042 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-611685] [5.1] Running with 2 response threads, dynamic=true
2023-11-29 23:06:51,053 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-611685] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-11-29 23:06:51,053 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-611685] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-11-29 23:06:51,058 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-611685] [5.1] Trying to connect to cluster: seatunnel-611685
2023-11-29 23:06:51,059 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-611685] [5.1] Trying to connect to [localhost]:5801
2023-11-29 23:06:51,071 INFO  com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [localhost]:5801 [seatunnel-611685] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:59440, qualifier=null, endpoint=[127.0.0.1]:59440, remoteUuid=f5ef88f3-edc9-485b-8e36-4a4cdd9de4c1, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: f5ef88f3-edc9-485b-8e36-4a4cdd9de4c1, client name: hz.client_1, client version: 5.1
2023-11-29 23:06:51,073 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-611685] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-11-29 23:06:51,073 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-611685] [5.1] Authenticated with server [localhost]:5801:17cac56e-7734-452f-a5e9-9a0a20dd0cc1, server version: 5.1, local address: /127.0.0.1:59440
2023-11-29 23:06:51,075 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-611685] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-11-29 23:06:51,081 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-611685] [5.1] 

Members [1] {
    Member [localhost]:5801 - 17cac56e-7734-452f-a5e9-9a0a20dd0cc1
}

2023-11-29 23:06:51,100 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-11-29 23:06:51,250 INFO  org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
2023-11-29 23:06:51,264 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /Users/yuriygavrilov/git_project/SeaTunnel/data/seatunnel1.template
2023-11-29 23:06:51,311 INFO  org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-11-29 23:06:51,347 INFO  org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "bucket" : "s3a://test",
            "path" : "/seatunnel/",
            "secret_key" : "XXX",
            "file_format_type" : "parquet",
            "access_key" : "XXX",
            "fs.s3a.aws.credentials.provider" : "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
            "plugin_name" : "S3File",
            "fs.s3a.endpoint" : "XXX"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "plugin_name" : "Console"
        }
    ]
}

2023-11-29 23:06:51,371 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-11-29 23:06:51,372 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-11-29 23:06:51,376 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/connectors/seatunnel
2023-11-29 23:06:51,381 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
2023-11-29 23:06:51,385 INFO  org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
2023-11-29 23:06:51,386 INFO  org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-11-29 23:06:51,400 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /Users/yuriygavrilov/git_project/SeaTunnel/apache-seatunnel-2.3.3/connectors/seatunnel
2023-11-29 23:06:51,408 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-611685] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-11-29 23:06:51,411 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-611685] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:59440, qualifier=null, endpoint=[127.0.0.1]:59440, remoteUuid=f5ef88f3-edc9-485b-8e36-4a4cdd9de4c1, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-11-29 23:06:51,411 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-611685] [5.1] Removed connection to endpoint: [localhost]:5801:17cac56e-7734-452f-a5e9-9a0a20dd0cc1, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:59440->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-11-29 23:06:51.259, lastWriteTime=2023-11-29 23:06:51.258, closedTime=2023-11-29 23:06:51.409, connected server version=5.1}
2023-11-29 23:06:51,411 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-611685] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-11-29 23:06:51,413 INFO  com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-611685] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:59440, qualifier=null, endpoint=[127.0.0.1]:59440, remoteUuid=f5ef88f3-edc9-485b-8e36-4a4cdd9de4c1, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=f5ef88f3-edc9-485b-8e36-4a4cdd9de4c1, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1701288411069, latest clientAttributes=lastStatisticsCollectionTime=1701288411100,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1701288411060,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=0,os.freePhysicalMemorySize=210055168,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=0,os.openFileDescriptorCount=0,os.processCpuTime=0,os.systemLoadAverage=2.26611328125,os.totalPhysicalMemorySize=34359738368,os.totalSwapSpaceSize=0,runtime.availableProcessors=12,runtime.freeMemory=1006044032,runtime.maxMemory=1073741824,runtime.totalMemory=1073741824,runtime.uptime=3271,runtime.usedMemory=67697792, labels=[]}
2023-11-29 23:06:51,414 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-611685] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-11-29 23:06:51,414 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-11-29 23:06:51,414 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-11-29 23:06:51,416 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-611685] [5.1] Shutting down connection manager...
2023-11-29 23:06:51,418 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-611685] [5.1] Shutting down node engine...
2023-11-29 23:06:51,422 INFO  org.apache.seatunnel.engine.server.SeaTunnelServer - master node check interrupted
2023-11-29 23:06:53,638 INFO  com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-611685] [5.1] Destroying node NodeExtension.
2023-11-29 23:06:53,638 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-611685] [5.1] Hazelcast Shutdown is completed in 2223 ms.
2023-11-29 23:06:53,638 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-611685] [5.1] [localhost]:5801 is SHUTDOWN
2023-11-29 23:06:53,639 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2023-11-29 23:06:53,639 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 

===============================================================================

2023-11-29 23:06:53,639 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error, 

2023-11-29 23:06:53,639 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/seatunnel/issues

2023-11-29 23:06:53,639 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 

2023-11-29 23:06:53,640 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='S3File'} not found.
    at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:223)
    at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:61)
    at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:81)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
    at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
    at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
    ... 2 more

2023-11-29 23:06:53,641 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
===============================================================================

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='S3File'} not found.
    at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:223)
    at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:61)
    at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:81)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
    at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
    at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
    ... 2 more