search

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.36k stars 2.42k forks source link

[SUPPORT] Table with delete block written by 0.13.x will not be readable by lower versions, such as 0.12.x #9195

Open Zouxxyy opened 1 year ago

Zouxxyy commented 1 year ago

Describe the problem you faced

https://github.com/apache/hudi/pull/7024 change the serialization of uft8, which cause table with delete block will not be readable by lower versions, such as 0.12.x

Caused by: org.apache.hudi.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 27 
   Serialization trace: orderingVal (org.apache.hudi.common.model.DeleteRecord) 
   at org.apache.hudi.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137) 
   at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693) 
   at org.apache.hudi.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118) 
   at org.apache.hudi.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) 
   at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731) 
   at org.apache.hudi.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:391) 
   at org.apache.hudi.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302) 
   at org.apache.hudi.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) 
   at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:100) 
   at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:74) 
   at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.deserialize(HoodieDeleteBlock.java:106) 
   at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.getRecordsToDelete(HoodieDeleteBlock.java:91) 
   at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:473) 
   at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:343) ...
KnightChess commented 1 year ago

yes, you are right, I met this question, and if upgrade hudi without compact log file to base file and write new log with delete block, the log file should be unable to use any hudi version compact

Zouxxyy commented 1 year ago

@KnightChess Looks like we need to complete https://issues.apache.org/jira/browse/HUDI-5760 Then https://github.com/apache/hudi/pull/7917 and https://github.com/apache/hudi/pull/7024 can be reverted. Although this is still a version incompatible change.