把influxdb里面的数据同步到doris

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

2024-07-23 16:32:32,247 WARN org.apache.seatunnel.core.starter.seatunnel.args.ClientCommandArgs$MasterTypeValidator -

-e and --deploy-mode deprecated in 2.3.1, please use -m and --master instead of it

Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config' Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml Jul 23, 2024 4:32:32 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig INFO: seatunnel.home is /home/data/apache-seatunnel-2.3.3 Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config' Jul 23, 2024 4:32:32 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml 2024-07-23 16:32:32,606 WARN com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct. 2024-07-23 16:32:32,606 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] Resolving domain name 'x.0.0x' to address(es): [x.0.0.x] 2024-07-23 16:32:32,607 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-735458] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [19.0.0x/19.0.0.x] 2024-07-23 16:32:32,628 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start... 2024-07-23 16:32:32,629 INFO com.hazelcast.system - [19.0.0x]:5801 [seatunnel-735458] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7) 2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.0x]:5801 [seatunnel-735458] [5.1] Cluster name: seatunnel-735458 2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.0x]:5801 [seatunnel-735458] [5.1]

/ | | | | | \ `--. | | | | --. \ / _ \ / _ | | | | | | || ' \ | ' \ / | | /_/ /| /| (| | | | | || || | | || | | || /| | _/ \| _,| _/ _,||| |||| || __|||

2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.0x]:5801 [seatunnel-735458] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation. 2024-07-23 16:32:32,630 INFO com.hazelcast.system - [19.0.0x]:5801 [seatunnel-735458] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed. To enable integrity checker do one of the following:

Change member config using Java API: config.setIntegrityCheckerEnabled(true);
Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load) 2024-07-23 16:32:32,632 INFO com.hazelcast.system - [19.0.0x]:5801 [seatunnel-735458] [5.1] The Jet engine is disabled. To enable the Jet engine on the members, do one of the following:
Change member config using Java API: config.getJetConfig().setEnabled(true)
Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load) 2024-07-23 16:32:32,949 INFO com.hazelcast.system.security - [19.0.0x]:5801 [seatunnel-735458] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config. 2024-07-23 16:32:33,002 INFO com.hazelcast.instance.impl.Node - [19.0.0x]:5801 [seatunnel-735458] [5.1] Using TCP/IP discovery 2024-07-23 16:32:33,003 WARN com.hazelcast.cp.CPSubsystem - [19.0.0x]:5801 [seatunnel-735458] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees. Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config' Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/seatunnel.yaml Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config' Jul 23, 2024 4:32:33 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast.yaml 2024-07-23 16:32:33,159 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [19.0.0x]:5801 [seatunnel-735458] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling 2024-07-23 16:32:33,159 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [19.0.0x]:5801 [seatunnel-735458] [5.1] Created new BusWork : 1083086553 2024-07-23 16:32:33,173 WARN org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [19.0.0x]:5801 2024-07-23 16:32:33,176 INFO org.apache.seatunnel.engine.server.CoordinatorService - [19.0.0x]:5801 [seatunnel-735458] [5.1]

CoordinatorService Thread Pool Status

activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0

2024-07-23 16:32:33,211 INFO com.hazelcast.internal.diagnostics.Diagnostics - [19.0.0x]:5801 [seatunnel-735458] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments. 2024-07-23 16:32:33,217 INFO com.hazelcast.core.LifecycleService - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is STARTING 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5816 is added to the blacklist. 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5814 is added to the blacklist. 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5812 is added to the blacklist. 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5810 is added to the blacklist. 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5822 is added to the blacklist. 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5820 is added to the blacklist. 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5824 is added to the blacklist. 2024-07-23 16:32:33,244 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5818 is added to the blacklist. 2024-07-23 16:32:33,245 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5830 is added to the blacklist. 2024-07-23 16:32:33,245 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5828 is added to the blacklist. 2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5826 is added to the blacklist. 2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5807 is added to the blacklist. 2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5805 is added to the blacklist. 2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5803 is added to the blacklist. 2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5815 is added to the blacklist. 2024-07-23 16:32:33,246 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5813 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5811 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5823 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5821 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5817 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5819 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5827 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5829 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5825 is added to the blacklist. 2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5806 is added to the blacklist. 2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5804 is added to the blacklist. 2024-07-23 16:32:33,247 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5809 is added to the blacklist. 2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5802 is added to the blacklist. 2024-07-23 16:32:33,248 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5808 is added to the blacklist. 2024-07-23 16:32:33,255 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.0x]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35139, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed 2024-07-23 16:32:33,257 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.0x]:5801 [seatunnel-735458] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:35139, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-07-23 16:32:33,626 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.0x]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=2, /127.0.0.1:5801->/127.0.0.1:60975, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed 2024-07-23 16:32:33,627 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.0x]:5801 [seatunnel-735458] [5.1] Connection[id=2, /127.0.0.1:5801->/127.0.0.1:60975, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-07-23 16:32:34,248 INFO com.hazelcast.internal.cluster.ClusterService - [19.0.0x]:5801 [seatunnel-735458] [5.1]

Members {size:1, ver:1} [ Member [19.0.0x]:5801 - feb828c8-a852-47eb-8c0b-68b497c2e6e1 this ]

2024-07-23 16:32:34,258 INFO com.hazelcast.core.LifecycleService - [x.0.0.x]:5801 [seatunnel-735458] [5.1] [19.0.0x]:5801 is STARTED Jul 23, 2024 4:32:34 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Loading configuration '/home/data/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config' Jul 23, 2024 4:32:34 PM com.hazelcast.internal.config.AbstractConfigLocator INFO: Using configuration file at /home/data/apache-seatunnel-2.3.3/config/hazelcast-client.yaml 2024-07-23 16:32:34,330 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-735458] [5.1] Running with 2 response threads, dynamic=true 2024-07-23 16:32:34,340 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING 2024-07-23 16:32:34,341 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED 2024-07-23 16:32:34,345 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Trying to connect to cluster: seatunnel-735458 2024-07-23 16:32:34,346 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Trying to connect to [x.0.0.x]:5801 2024-07-23 16:32:34,350 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [x.0.0.x]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=4, /127.0.0.1:5801->/127.0.0.1:44547, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed 2024-07-23 16:32:34,351 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.0x]:5801 [seatunnel-735458] [5.1] Connection[id=4, /127.0.0.1:5801->/127.0.0.1:44547, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-07-23 16:32:34,351 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.0x]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=3, /19.0.0.37:5801->/19.0.0.37:45643, qualifier=null, endpoint=[19.0.0.x]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: d01daf7d-5edf-4225-b98c-630dc83d9ba3, client name: hz.client_1, client version: 5.1 2024-07-23 16:32:34,353 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED 2024-07-23 16:32:34,353 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Authenticated with server [19.0.0x]:5801:feb828c8-a852-47eb-8c0b-68b497c2e6e1, server version: 5.1, local address: /19.0.0.37:45643 2024-07-23 16:32:34,354 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-735458] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments. 2024-07-23 16:32:34,359 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-735458] [5.1]

Members [1] { Member [19.0.0x]:5801 - feb828c8-a852-47eb-8c0b-68b497c2e6e1 }

2024-07-23 16:32:34,379 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds. 2024-07-23 16:32:34,483 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[] 2024-07-23 16:32:34,498 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ./config/influxdb_to_doris.conf 2024-07-23 16:32:34,534 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64] 2024-07-23 16:32:34,569 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: { "env" : { "parallelism" : 1, "job.mode" : "BATCH", "checkpoint.interval" : 10000 }, "source" : [ { "schema" : { "fields" : { "node_cpu_seconds_total" : "STRING", "pg_database_size_bytes" : "STRING", "node_memory_MemTotal_bytes" : "STRING" } }, "database" : "sbyw", "password" : "influx123", "lower_bound" : 1, "source_table_name" : [ "prometheus_remote_write" ], "result_table_name" : [ "sbyw_application_process_type" ], "upper_bound" : 100, "plugin_name" : "InfluxDB", "user" : "influx", "url" : "http://x.x.x.x:8086", "sql" : "SELECT * FROM prometheus_remote_write WHERE time > now() - 1h", "partition_num" : 10 } ], "sink" : [ { "password" : "sbyw@123456", "fenodes" : [ "x.x.x.x:8030" ], "sink.enable-2pc" : "true", "doris.config" : { "format" : "json", "read_json_by_line" : "true" }, "table.identifier" : "sbyw_data_acquisition.sbyw_application_process_type", "source_table_name" : [ "prometheus_remote_write" ], "sink.enable-delete" : "true", "plugin_name" : "Doris", "sink.label-prefix" : "test-cdc", "username" : "root" } ], "transform" : [ { "source_table_name" : [ "prometheus_remote_write" ], "result_table_name" : [ "sbyw_application_process_type" ], "fields" : { "system_cpu_usage" : "node_cpu_seconds_total", "process_occupy_physical_memory_size" : "pg_database_size_bytes", "process_read_written_file_system_total_bytes" : "node_memory_MemTotal_bytes" }, "plugin_name" : "Copy" } ] }

2024-07-23 16:32:34,588 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-07-23 16:32:34,589 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-07-23 16:32:34,592 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /home/data/apache-seatunnel-2.3.5/connectors/seatunnel 2024-07-23 16:32:34,596 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations. 2024-07-23 16:32:34,596 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-07-23 16:32:34,597 ERROR org.apache.seatunnel.engine.core.parse.ConfigParserUtil - The 'source_table_name' option configured in [Copy] is incorrect, and the source/transform[prometheus_remote_write] is not found. 2024-07-23 16:32:34,597 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations. 2024-07-23 16:32:34,597 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-07-23 16:32:34,597 ERROR org.apache.seatunnel.engine.core.parse.ConfigParserUtil - The 'source_table_name' option configured in [Doris] is incorrect, and the source/transform[prometheus_remote_write] is not found. 2024-07-23 16:32:34,597 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources. 2024-07-23 16:32:34,598 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory' 2024-07-23 16:32:34,626 WARN com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [19.0.0x]:5801 [seatunnel-735458] [5.1] Received auth from Connection[id=5, /127.0.0.1:5801->/127.0.0.1:39627, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=NONE, planeIndex=-1] with clientUuid 32640c62-52cb-4829-af51-3566ec9d70b3 and clientName hz.client_1, authentication failed 2024-07-23 16:32:34,627 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.0x]:5801 [seatunnel-735458] [5.1] Connection[id=5, /127.0.0.1:5801->/127.0.0.1:39627, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-07-23 16:32:34,635 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /home/data/apache-seatunnel-2.3.5/connectors/seatunnel 2024-07-23 16:32:34,675 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='InfluxDB'} from classpath 2024-07-23 16:32:34,921 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN 2024-07-23 16:32:34,923 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [19.0.0x]:5801 [seatunnel-735458] [5.1] Connection[id=3, /19.0.0.x:5801->/x.0.0.x:45643, qualifier=null, endpoint=[19.0.0.x]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side 2024-07-23 16:32:34,923 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-735458] [5.1] Removed connection to endpoint: [19.0.0x]:5801:feb828c8-a852-47eb-8c0b-68b497c2e6e1, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/19.0.0.x:45643->/19.0.0.x:5801}, remoteAddress=[19.0.037]:5801, lastReadTime=2024-07-23 16:32:34.493, lastWriteTime=2024-07-23 16:32:34.492, closedTime=2024-07-23 16:32:34.921, connected server version=5.1} 2024-07-23 16:32:34,923 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED 2024-07-23 16:32:34,924 INFO com.hazelcast.client.impl.ClientEndpointManager - [19.0.0x]:5801 [seatunnel-735458] [5.1] Destroying ClientEndpoint{connection=Connection[id=3, /19.0.0.x:5801->/19.0.0.x:45643, qualifier=null, endpoint=[19.0.0.x]:45643, remoteUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=d01daf7d-5edf-4225-b98c-630dc83d9ba3, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1721723554350, latest clientAttributes=lastStatisticsCollectionTime=1721723554379,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1721723554347,clientAddress=19.0.0.x,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=11341529088,os.freePhysicalMemorySize=45426728960,os.freeSwapSpaceSize=8535400448,os.maxFileDescriptorCount=65535,os.openFileDescriptorCount=122,os.processCpuTime=5410000000,os.systemLoadAverage=0.16,os.totalPhysicalMemorySize=68326711296,os.totalSwapSpaceSize=8535400448,runtime.availableProcessors=16,runtime.freeMemory=968396248,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=2607,runtime.usedMemory=60781096, labels=[]} 2024-07-23 16:32:34,925 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-735458] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN 2024-07-23 16:32:34,925 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client...... 2024-07-23 16:32:34,925 INFO com.hazelcast.core.LifecycleService - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTTING_DOWN 2024-07-23 16:32:34,928 INFO com.hazelcast.instance.impl.Node - [19.0.0x]:5801 [seatunnel-735458] [5.1] Shutting down connection manager... 2024-07-23 16:32:34,929 INFO com.hazelcast.instance.impl.Node - [19.0.0x]:5801 [seatunnel-735458] [5.1] Shutting down node engine... 2024-07-23 16:32:34,933 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - master node check interrupted 2024-07-23 16:32:37,944 INFO com.hazelcast.instance.impl.NodeExtension - [19.0.0x]:5801 [seatunnel-735458] [5.1] Destroying node NodeExtension.2024-07-23 16:32:37,945 INFO com.hazelcast.instance.impl.Node - [19.0.0x]:5801 [seatunnel-735458] [5.1] Hazelcast Shutdown is completed in 3017 ms. 2024-07-23 16:32:37,945 INFO com.hazelcast.core.LifecycleService - [19.0.0x]:5801 [seatunnel-735458] [5.1] [19.0.0x]:5801 is SHUTDOWN 2024-07-23 16:32:37,945 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ...... Exception in thread "main" java.lang.NoSuchFieldError: Companion at okhttp3.internal.Util.(Util.kt:71) at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1239) at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1634) at okhttp3.HttpUrl.get(HttpUrl.kt) at retrofit2.Retrofit$Builder.baseUrl(Retrofit.java:554) at org.influxdb.impl.InfluxDBImpl.(InfluxDBImpl.java:178) at org.influxdb.impl.InfluxDBImpl.(InfluxDBImpl.java:120) at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:72) at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:89) at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179) at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)

SeaTunnel Version

SeaTunnel 2.3.3

SeaTunnel Config

# influxdb_to_doris.conf

env {
  parallelism = 1
  job.mode = "BATCH"
  checkpoint.interval = 10000
}

source {

    InfluxDB {
        url = "http://x.x.x.x:8086"
        sql = "SELECT node_cpu_seconds_total,pg_database_size_bytes,node_memory_MemTotal_bytes FROM prometheus_remote_write WHERE time > now() - 1h"
        database = "sbyw"
    user = "influx"
        password = "influx123"
        upper_bound = 100
        lower_bound = 1
        partition_num = 10
        source_table_name = ["prometheus_remote_write"]
        result_table_name = ["sbyw_application_process_type"]
        #split_column = "value"
        schema {
            fields {
        node_cpu_seconds_total = STRING
                pg_database_size_bytes = STRING
                node_memory_MemTotal_bytes = STRING
            }
         }

    }

}

sink {
  Doris {
    fenodes = ["x.x.x.x:8030"]
    username = "root"
    password = "sbyw@123456"
    table.identifier = "sbyw_data_acquisition.sbyw_application_process_type"
    sink.label-prefix = "test-cdc"
    sink.enable-2pc = "true"
    sink.enable-delete = "true"
    source_table_name = ["prometheus_remote_write"]
    doris.config {
      format = "json"
      read_json_by_line = "true"
    }
  }
}

transform {
  Copy {
    source_table_name = ["prometheus_remote_write"]
    result_table_name = ["sbyw_application_process_type"]
    fields {
      system_cpu_usage = node_cpu_seconds_total
      process_occupy_physical_memory_size = pg_database_size_bytes
      process_read_written_file_system_total_bytes = node_memory_MemTotal_bytes
    }
  }
}

Running Command

./bin/seatunnel.sh -f --config ./config/influxdb_to_doris.conf -e local

Error Exception

2024-07-23 16:32:37,945 INFO  com.hazelcast.core.LifecycleService - [19.0.037]:5801 [seatunnel-735458] [5.1] [19.0.037]:5801 is SHUTDOWN
2024-07-23 16:32:37,945 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
Exception in thread "main" java.lang.NoSuchFieldError: Companion
        at okhttp3.internal.Util.<clinit>(Util.kt:71)
        at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1239)
        at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1634)
        at okhttp3.HttpUrl.get(HttpUrl.kt)
        at retrofit2.Retrofit$Builder.baseUrl(Retrofit.java:554)
        at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:178)
        at org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:120)
        at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:72)
        at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:89)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:85)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:317)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:179)
        at org.apache.seatunnel.engine.core.job.AbstractJobEnvironment.getLogicalDag(AbstractJobEnvironment.java:109)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:73)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:143)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

apache / seatunnel