search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.1k stars 834 forks source link

[Bug] RichCdcMultiplexRecord cant be deserialized by kryoSerializer. throw UnsupportedOperationException #3630

Closed LinMingQiang closed 3 days ago

LinMingQiang commented 4 days ago

Search before asking

Paimon version

master

Compute Engine

flink

Minimal reproduce step

  1. set pipeline.operator-chaining=false in flink-conf.yaml. RichCdcMultiplexRecord would be serialized an deserialized.

  2. run flink job : MySqlSyncTableAction.

  3. alter mysql table to add a new column.

What doesn't meet your expectations?

Paimon table schema should be changed.

Anything else?

com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException Serialization trace: fields (org.apache.paimon.flink.sink.cdc.RichCdcMultiplexRecord) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-2.24.0.jar:?] at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) ~[kryo-2.24.0.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) ~[kryo-2.24.0.jar:?] at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:403) ~[flink-core-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:188) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:43) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53) ~[flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.runtime.io.network.api.serialization.NonSpanningWrapper.readInto(NonSpanningWrapper.java:337) ~[flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNonSpanningRecord(SpillingAdaptiveSpanningRecordDeserializer.java:128) ~[flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:103) ~[flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:93) ~[flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:100) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) ~[flink-streaming-java-1.18.0.jar:1.18.0] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) ~[flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932) [flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) [flink-runtime-1.18.0.jar:1.18.0] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-runtime-1.18.0.jar:1.18.0] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_381] Caused by: java.lang.UnsupportedOperationException at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057) ~[?:1.8.0_381] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.24.0.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) ~[kryo-2.24.0.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679) ~[kryo-2.24.0.jar:?] at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) ~[kryo-2.24.0.jar:?] ... 21 more

Are you willing to submit a PR?