Open CodingJun opened 3 months ago
Is there any error log for equality delete?
Is there any error log for equality delete?
No error, If I read directly from snapshot id: 3, the result is correct.
I found the code to drop the equality delete files here.
base
is refreshed, base.lastSequenceNumber()
is 3.UNASSIGNED_SEQ
, only the newly append file at t3.minDataSequenceNumber
is also 3, the manifest for equality delete will be droped.I think the base.lastSequenceNumber()
should be 1 at the start snapshot instead of 3.
@CodingJun: Your analysis seems correct to me. We need to take the minDataSequenceNumber
and startingSequenceNumber
.
@RussellSpitzer and @aokolnychyi might know more.
your process is in use-starting-sequence-number = true
?
I test with use-starting-sequence-number = true
and compact failed(apache iceberg1.4.3):
Exception in thread "main" org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new delete for replaced data file: GenericDataFile ...
your process is in
use-starting-sequence-number = true
? I test withuse-starting-sequence-number = true
and compact failed(apache iceberg1.4.3):Exception in thread "main" org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new delete for replaced data file: GenericDataFile ...
Yes, The default setting is true. Can you debug it to see if the configuration is effective?
Do you know if this is a bug? @RussellSpitzer @aokolnychyi
@szehon-ho: This is a potential bug with compaction and equality delete files. Could you please help me understand if this is a valid scenarion?
The validation error looks correct. Since we shouldn't be able to do the rewrite if the equality delete has been written in the middle.
When we se using starting sequence number to false i'm not sure we are guaranteeing serializability
@RussellSpitzer: The compaction started at t1 and it will replace files created before t1. The equality delete is created later at t2. So if we use the same sequence number for the new files as it was for t1 then theoretically we should be able to change files and the result should be correct.
Do I missunderstand how use-starting-sequence-number
is handled?
yea I think @RussellSpitzer is right, we should rely on validation error to prevent this scenario here, ie T1 should not be able to commit successfully.
i need to understand one thing:
When I set use-starting-sequence-number = false for rewriteDataFiles, Thread 1 compact data files failed at t4. stacktrace:
Caused by: org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new delete for replaced data file: GenericDataFile{content=data, file_path=/var/folders/5z/dqrlv_ts0wqf36vd39bb384h0000gn/T/junit17491575750166086656/9f77fae8-d62a-426d-971f-a342b6775c44/test_schema/test_table/data/00000-2-52ae94aa-b796-4c42-bf9c-92d36c52e522-00001.parquet, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=1, file_size_in_bytes=407, column_sizes=null, value_counts=org.apache.iceberg.util.SerializableMap@0, null_value_counts=org.apache.iceberg.util.SerializableMap@1, nan_value_counts=org.apache.iceberg.util.SerializableMap@0, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, key_metadata=null, split_offsets=[4], equality_ids=null, sort_order_id=null} at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:50) at org.apache.iceberg.MergingSnapshotProducer.validateNoNewDeletesForDataFiles(MergingSnapshotProducer.java:418) at org.apache.iceberg.MergingSnapshotProducer.validateNoNewDeletesForDataFiles(MergingSnapshotProducer.java:367) at org.apache.iceberg.BaseRewriteFiles.validate(BaseRewriteFiles.java:108) at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:175) at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:296) at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:295) at org.apache.iceberg.actions.RewriteDataFilesCommitManager.commitFileGroups(RewriteDataFilesCommitManager.java:89) at org.apache.iceberg.actions.RewriteDataFilesCommitManager.commitOrClean(RewriteDataFilesCommitManager.java:110) at org.apache.iceberg.spark.actions.RewriteDataFilesSparkAction.doExecute(RewriteDataFilesSparkAction.java:291) ... 8 more
your process is in use-starting-sequence-number = true ? I test with use-starting-sequence-number = true and compact failed(apache iceberg1.4.3): Exception in thread "main" org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new delete for replaced data file: GenericDataFile ...
from above conversation it seem we get the validationException in both code-paths, isnt it?
Apache Iceberg version
1.5.1
Query engine
Spark
Please describe the bug 🐞
I have a program that continuously write streaming data to iceberg, and regularly use spark to compact data files. But I found that after compact the data files, some of the data was not deleted correctly. The following are the examples to reproduce:
Original table:
Writing process:
Result:
The correct result should be:
PS:
When I set
use-starting-sequence-number = false
for rewriteDataFiles, Thread 1 compact data files failed at t4. stacktrace:Question:
Why are the equality deleted files lost? Is this correct or a bug?