An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
There is a problem today that the iceberg conversion on replace/overwrite uniform table is sync, so if user do concurrent replace, many will fail because there are very likely conversion of higher versions succeeded/committed. Making the conversion async/one-at-a-time will be a good solution to stick with existing behavior for all other kinds of commits. If there is a conflict, thats because a higher version of conversion committed and we should not worry about the conflict.
Which Delta project/connector is this regarding?
Description
There is a problem today that the iceberg conversion on replace/overwrite uniform table is sync, so if user do concurrent replace, many will fail because there are very likely conversion of higher versions succeeded/committed. Making the conversion async/one-at-a-time will be a good solution to stick with existing behavior for all other kinds of commits. If there is a conflict, thats because a higher version of conversion committed and we should not worry about the conflict.
How was this patch tested?
manual test
Does this PR introduce any user-facing changes?