[Bug] [HTTP SUBMIT JOB] Error reported when submitting tasks concurrently

W-dragan commented 7 months ago

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

When I submit more than 30 HTTP tasks instantaneously, it is a timed task that triggers the submission of more than 30 HTTP tasks to seatunel, causing errors. Do I need to use locking and other mechanisms to prevent concurrent errors Caused by: java.lang.ClassNotFoundException: org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSink

SeaTunnel Version

2.3.4

SeaTunnel Config

any

Running Command

http

Error Exception

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
Caused by: java.lang.ClassNotFoundException: org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSink
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_212]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_212]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_212]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_212]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_212]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_212]
at com.hazelcast.internal.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:297) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:261) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:867) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868) ~[?:1.8.0_212]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) ~[?:1.8.0_212]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) ~[?:1.8.0_212]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) ~[?:1.8.0_212]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) ~[?:1.8.0_212]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) ~[?:1.8.0_212]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) ~[?:1.8.0_212]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) ~[?:1.8.0_212]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) ~[?:1.8.0_212]
at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:92) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:85) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:349) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at org.apache.seatunnel.engine.core.dag.logical.LogicalVertex.readData(LogicalVertex.java:99) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:349) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at org.apache.seatunnel.engine.core.dag.logical.LogicalDag.readData(LogicalDag.java:154) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:268) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject.deserializeWithCustomClassLoader(CustomClassLoadedObject.java:66) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:210) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$3(CoordinatorService.java:419) ~[seatunnel-starter.jar:2.3.4-baidu1.0.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_212]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

W-dragan commented 7 months ago

@Hisoka-X @liugddx please help cc

hailin0 commented 7 months ago

@W-dragan

Please retry run #6509

W-dragan commented 7 months ago

After using code # 6509, preliminary verification has not reproduced the previous issue for now

I will continue to follow up and verify repeatedly in the future @Hisoka-X good job!

apache / seatunnel