search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.08k stars 1.83k forks source link

[Bug] [Connector-V2] cannot assign instance of com.clickhouse.client.ClickHouseNode #3999

Open datagic opened 1 year ago

datagic commented 1 year ago

Search before asking

What happened

Synchronizing the hdfs data to the clickhouse has been difficult and has never been successful so far.

SeaTunnel Version

seatunnel ver: 2.3.0 clickhouse ver: 21.8.4.51

SeaTunnel Config

env {
  execution.parallelism = 10
  job.mode = "BATCH"
}

source {

HdfsFile {
  path = "/user/hive/warehouse/data.db/xxxxx/"
  type = "text"
  fs.defaultFS = "hdfs://xxx:8020"
}

}

transform {

}

sink {
  Clickhouse {
    host = "1.1.1.1:8123"
    database = "dname"
    table = "tname"
    fields = ["a", "b", "c", "d", "e", "f"]
    username = "default"
    password = "xxxxx"
    }

}

Running Command

./seatunnel.sh --config ../config/xxx.conf -e local

Error Exception

2023-01-30 14:00:13,869 ERROR org.apache.seatunnel.core.starter.Seatunnel - Exception StackTrace:java.util.concurrent.CompletionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
        at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
        at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709)
        at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294)
        at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321)
        at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304)
        at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164)
        at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141)
        at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60)
        at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251)
        at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243)
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245)
        at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135)
        at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
        at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
        at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
        at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
        at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
        at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
        at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:111)
        at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89)
        at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136)
        at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
        at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
        at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
        at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
        at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:111)
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:355)
        at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)
        at org.apache.seatunnel.engine.core.dag.logical.LogicalVertex.readData(LogicalVertex.java:100)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51)
        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:349)
        at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)
        at org.apache.seatunnel.engine.core.dag.logical.LogicalDag.readData(LogicalDag.java:154)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51)
        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:268)
        at com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject.deserializeWithCustomClassLoader(CustomClassLoadedObject.java:66)
        at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:159)
        at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:338)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: cannot assign instance of com.clickhouse.client.ClickHouseNode to field org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard.node of type com.clickhouse.client.ClickHouseNode in instance of org.apache.seatunnel.connectors.seatunnel.clickhouse.shard.Shard
        at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
        at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
        at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:92)
        at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:85)
        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:349)
        ... 22 more

Flink or Spark Version

Not used

Java or Scala Version

jdk 1.8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

qianmoQ commented 1 year ago

Please provide the Click House environment?

datagic commented 1 year ago

Please provide the Click House environment?

clickhouse ver: 21.8.4.51

I can execute it through the earlier version of waterdrop-1.5.0，However, the above error reports were encountered in Seatunnel-2.3.0.

qianmoQ commented 1 year ago

Please provide the Click House environment?

clickhouse ver: 21.8.4.51

I can execute it through the earlier version of waterdrop-1.5.0，However, the above error reports were encountered in Seatunnel-2.3.0.

Unfortunately, I did not reproduce the problem described in multiple tests with hadoop HA and non-HA clusters

tianxingdao commented 1 year ago

I have the same problem

Ackyo commented 1 year ago

I have the same problem too

ditgittube commented 1 year ago

I have the same problem too

BigTailMonkey commented 10 months ago

Please provide the Click House environment?

clickhouse ver: 21.8.4.51 I can execute it through the earlier version of waterdrop-1.5.0，However, the above error reports were encountered in Seatunnel-2.3.0.

Unfortunately, I did not reproduce the problem described in multiple tests with hadoop HA and non-HA clusters

I have the same problem too.This is my config file: `env { execution.parallelism = 1 job.mode = "BATCH" checkpoint.interval = 10000 }

source { FakeSource { parallelism = 1 result_table_name = "fake" row.num = 1 schema = { fields { id = "int" a = "string" b = "string" } } } }

sink { Clickhouse { source_table_name = "fake" host = "192.168.1.110:8123" database = "test_seatunnel" table = "table_a" username = "default" password = "" } }`

But I succeeded twice and failed once with same ClickHouse Database and ClickHouse Driver. The one success was on my own macbook pro with seatunnel 2.3.3+jdk 1.8.0_321. And the other was also using seatunnel 2.3.3+jdk 11.0.21 on my own macbook pro The one failure was on centos 7 with seatunnel 2.3.3+jdk 11.0.21.

So I'm guessing it's the OS env? That's a bad guess.

BigTailMonkey commented 10 months ago

I solved the problem: typed a zip archive of seatunnel from my own computer, transferred it to the machine that was reporting the error, and ran it again and it worked. But I still don't know what's causing it. Both downloads of seatunnel are version 2.3.3 and the MD5 is the same when compared.

qianmoQ commented 10 months ago

I solved the problem: typed a zip archive of seatunnel from my own computer, transferred it to the machine that was reporting the error, and ran it again and it worked. But I still don't know what's causing it. Both downloads of seatunnel are version 2.3.3 and the MD5 is the same when compared.

Please provide complete diary？

WuJiY commented 9 months ago

解决了吗

taojun610-chris commented 8 months ago

I have the same problem

kuz-liu commented 4 months ago

I have the same problem

weipengfei-sj commented 3 weeks ago

I have the same problem

huda3016 commented 3 days ago

I have the same problem