apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.07k stars 1.83k forks source link

[Bug] [seatunnel] rest-api post task always fail #5686

Closed QiuZiXian closed 1 year ago

QiuZiXian commented 1 year ago

Search before asking

What happened

do as https://seatunnel.apache.org/docs/2.3.3/start-v2/locally/quick-start-seatunnel-engine. ./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local throw error: ` 2023-10-23 11:25:10,845 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map at java.util.ArrayList.forEach(ArrayList.java:1257) at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:150) at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119) at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104) at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53) at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67) at java.util.Optional.orElseGet(Optional.java:267) at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.(MultipleTableJobConfigParser.java:127) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getJobConfigParser(JobExecutionEnvironment.java:63) at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143) ... 2 more

2023-10-23 11:25:10,845 ERROR org.apache.seatunnel.core.starter.SeaTunnel -

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map at java.util.ArrayList.forEach(ArrayList.java:1257) at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:150) at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119) at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104) at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53) at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67) at java.util.Optional.orElseGet(Optional.java:267) at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.(MultipleTableJobConfigParser.java:127) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getJobConfigParser(JobExecutionEnvironment.java:63) at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143) ... 2 more`

SeaTunnel Version

2.3.3

SeaTunnel Config

plugin_config:

--connectors-v2--
connector-fake
connector-console
connector-tdengine
connector-tdengine
onnector-amazondynamodb
connector-assert
connector-cassandra
connector-cdc-mysql
connector-cdc-mongodb
connector-cdc-sqlserver
connector-clickhouse
connector-datahub
connector-dingtalk
connector-doris
connector-elasticsearch
connector-email
connector-file-ftp
connector-file-hadoop
connector-file-local
connector-file-oss
connector-file-oss-jindo
connector-file-s3
connector-file-sftp
connector-hive
connector-http-base
connector-http-feishu
connector-http-gitlab
connector-http-github
connector-http-jira
connector-http-wechat
connector-hudi
connector-iceberg
connector-influxdb
connector-jdbc
connector-kafka
connector-mongodb
connector-neo4j
connector-openmldb
connector-pulsar
connector-rabbitmq
connector-redis
connector-slack
connector-socket
connector-hbase
--end--

v2.batch.config.template
```bash
env {
  # You can set SeaTunnel environment configuration here
  execution.parallelism = 2
  job.mode = "BATCH"
  checkpoint.interval = 10000
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  FakeSource {
    parallelism = 2
    result_table_name = "fake"
    row.num = 16
    schema = {
      fields {
        name = "string"
        age = "int"
      }
    }
  }
transform {
FieldMapper {
source_table_name = "fake"
result_table_name = "fake1"
field_mapper{
age = age
name = new_name
}
}

}

  # If you would like to get more information about how to configure SeaTunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

sink {
  Console {
  source_talbe_name = "fake1"
  }

  # If you would like to get more information about how to configure SeaTunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

### Running Command

```shell
./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

Error Exception

`
2023-10-23 11:25:10,845 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:150)
        at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
        at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
        at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
        at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
        at java.util.Optional.orElseGet(Optional.java:267)
        at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.<init>(MultipleTableJobConfigParser.java:127)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getJobConfigParser(JobExecutionEnvironment.java:63)
        at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
        ... 2 more

2023-10-23 11:25:10,845 ERROR org.apache.seatunnel.core.starter.SeaTunnel - 
===============================================================================

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.processConfig(ConfigShadeUtils.java:150)
        at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:119)
        at org.apache.seatunnel.core.starter.utils.ConfigShadeUtils.decryptConfig(ConfigShadeUtils.java:104)
        at org.apache.seatunnel.core.starter.utils.ConfigBuilder.ofInner(ConfigBuilder.java:53)
        at org.apache.seatunnel.core.starter.utils.ConfigBuilder.lambda$of$1(ConfigBuilder.java:67)
        at java.util.Optional.orElseGet(Optional.java:267)
        at org.apache.seatunnel.core.starter.utils.ConfigBuilder.of(ConfigBuilder.java:67)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.<init>(MultipleTableJobConfigParser.java:127)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getJobConfigParser(JobExecutionEnvironment.java:63)
        at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
        ... 2 more`

Zeta or Flink or Spark Version

zeta

Java or Scala Version

java1.8

ava version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Screenshots

image image

Are you willing to submit PR?

Code of Conduct

CheneyYin commented 1 year ago
field_mapper{
age = age
name = new_name
}

may be modified to

field_mapper = {
age = age
name = new_name
}
QiuZiXian commented 1 year ago

thanks. but it not works.I rechecked my configuration. The correct configuration is as follows image But there are new problems:

2023-10-23 15:05:45,804 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-405331] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-10-23 15:05:45,809 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-405331] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:31027, qualifier=null, endpoint=[127.0.0.1]:31027, remoteUuid=91603d25-04c5-44ab-a0aa-dd99cefbe535, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-10-23 15:05:45,810 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-405331] [5.1] Removed connection to endpoint: [localhost]:5801:44a6f32d-2736-47eb-88d3-322b2f4caff9, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:31027->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-10-23 15:05:45.797, lastWriteTime=2023-10-23 15:05:45.759, closedTime=2023-10-23 15:05:45.805, connected server version=5.1}
2023-10-23 15:05:45,810 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-405331] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-10-23 15:05:45,812 INFO  com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-405331] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:31027, qualifier=null, endpoint=[127.0.0.1]:31027, remoteUuid=91603d25-04c5-44ab-a0aa-dd99cefbe535, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=91603d25-04c5-44ab-a0aa-dd99cefbe535, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1698044742786, latest clientAttributes=lastStatisticsCollectionTime=1698044742839,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1698044742715,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=17767665664,os.freePhysicalMemorySize=437083398144,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=409600,os.openFileDescriptorCount=57,os.processCpuTime=12840000000,os.systemLoadAverage=2.89,os.totalPhysicalMemorySize=540564987904,os.totalSwapSpaceSize=0,runtime.availableProcessors=48,runtime.freeMemory=976949352,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=3466,runtime.usedMemory=52227992, labels=[]}
2023-10-23 15:05:45,813 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-405331] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-10-23 15:05:45,814 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-10-23 15:05:45,814 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-405331] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-10-23 15:05:45,818 INFO  com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801 [seatunnel-405331] [5.1] Shutdown request of Member [localhost]:5801 - 44a6f32d-2736-47eb-88d3-322b2f4caff9 this is handled
2023-10-23 15:05:45,827 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-405331] [5.1] Shutting down connection manager...
2023-10-23 15:05:45,830 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-405331] [5.1] Shutting down node engine...
2023-10-23 15:05:48,347 INFO  com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-405331] [5.1] Destroying node NodeExtension.
2023-10-23 15:05:48,348 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-405331] [5.1] Hazelcast Shutdown is completed in 2531 ms.
2023-10-23 15:05:48,348 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-405331] [5.1] [localhost]:5801 is SHUTDOWN
2023-10-23 15:05:48,348 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2023-10-23 15:05:48,348 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-10-23 15:05:48,350 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal

image

run shutdown hook because get close signal Is this an unsuccessful run?

QiuZiXian commented 1 year ago

Will there be a visual console address if the operation is successful?

CheneyYin commented 1 year ago

thanks. but it not works.I rechecked my configuration. The correct configuration is as follows image But there are new problems:

2023-10-23 15:05:45,804 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-405331] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-10-23 15:05:45,809 INFO  com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-405331] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:31027, qualifier=null, endpoint=[127.0.0.1]:31027, remoteUuid=91603d25-04c5-44ab-a0aa-dd99cefbe535, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-10-23 15:05:45,810 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-405331] [5.1] Removed connection to endpoint: [localhost]:5801:44a6f32d-2736-47eb-88d3-322b2f4caff9, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:31027->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-10-23 15:05:45.797, lastWriteTime=2023-10-23 15:05:45.759, closedTime=2023-10-23 15:05:45.805, connected server version=5.1}
2023-10-23 15:05:45,810 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-405331] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-10-23 15:05:45,812 INFO  com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-405331] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:31027, qualifier=null, endpoint=[127.0.0.1]:31027, remoteUuid=91603d25-04c5-44ab-a0aa-dd99cefbe535, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=91603d25-04c5-44ab-a0aa-dd99cefbe535, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1698044742786, latest clientAttributes=lastStatisticsCollectionTime=1698044742839,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1698044742715,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=17767665664,os.freePhysicalMemorySize=437083398144,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=409600,os.openFileDescriptorCount=57,os.processCpuTime=12840000000,os.systemLoadAverage=2.89,os.totalPhysicalMemorySize=540564987904,os.totalSwapSpaceSize=0,runtime.availableProcessors=48,runtime.freeMemory=976949352,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=3466,runtime.usedMemory=52227992, labels=[]}
2023-10-23 15:05:45,813 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-405331] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-10-23 15:05:45,814 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-10-23 15:05:45,814 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-405331] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-10-23 15:05:45,818 INFO  com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801 [seatunnel-405331] [5.1] Shutdown request of Member [localhost]:5801 - 44a6f32d-2736-47eb-88d3-322b2f4caff9 this is handled
2023-10-23 15:05:45,827 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-405331] [5.1] Shutting down connection manager...
2023-10-23 15:05:45,830 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-405331] [5.1] Shutting down node engine...
2023-10-23 15:05:48,347 INFO  com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-405331] [5.1] Destroying node NodeExtension.
2023-10-23 15:05:48,348 INFO  com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-405331] [5.1] Hazelcast Shutdown is completed in 2531 ms.
2023-10-23 15:05:48,348 INFO  com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-405331] [5.1] [localhost]:5801 is SHUTDOWN
2023-10-23 15:05:48,348 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2023-10-23 15:05:48,348 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-10-23 15:05:48,350 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal

image

run shutdown hook because get close signal Is this an unsuccessful run?

Did you find any logs with 'ConsoleSinkWriter'?If you can, you will look some output records.

CheneyYin commented 1 year ago

Will there be a visual console address if the operation is successful?

example

QiuZiXian commented 1 year ago

thanks,but I mean this:https://seatunnel.apache.org/seatunnel_web/1.0.0/deploy where can I get help of or commit the issue of seatunnel_web ?

QiuZiXian commented 1 year ago

I deploy seatunnel by the method of quick-start-seatunnel-engin. and The following code runs successfully ` cd "apache-seatunnel-${version}" ./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

but with [rest-api](https://seatunnel.apache.org/docs/2.3.3/seatunnel-engine/rest-api) post task always fail.(Whether it is fake data or use mysql) ![image](https://github.com/apache/seatunnel/assets/28780269/71ee437f-55d8-4488-bab0-55f9320cecd5) ![image](https://github.com/apache/seatunnel/assets/28780269/9b275716-9d29-4786-9c9b-276fa7f97ac3) json

{ "env": { "job.mode": "batch" }, "source": [ { "plugin_name": "FakeSource", "result_table_name": "fake", "row.num": 100, "schema": { "fields": { "name": "string", "age": "int", "card": "int" } } } ], "transform": [ ], "sink": [ { "plugin_name": "Console", "source_table_name": ["fake"] } ] } `

liunaijie commented 1 year ago

seatunnel web repo: https://github.com/apache/seatunnel-web.