Closed mikeburkat closed 3 weeks ago
Looks like this is mainly due to a known issue in pyarrow: https://github.com/apache/arrow/issues/39030
However, I did find that a _delta_log transaction which has multiple add actions can have it's part files in an unsorted order in the transaction, which also contributes to this problem given that if the add action files are read in the "transaction order" then the data can also appear unsorted even though data in individual files is ordered.
This is something we cannot guarantee in the first place and the PyArrow engine will be deprecated as of v0.19, so I am closing this issue
Environment
Delta-rs version: 0.18.2
Binding: python
Environment:
Bug
What happened: I wrote a monotonically incrementing sequence into a deltalake table using the pyarrow engine. When reading this deltalake table, the data is no longer monotonically incrementing.
What you expected to happen: I expect the data to be monotonically incrementing. The
rust
engine appears to work as expected, however thepyarrow
engine appears to re-order the data.How to reproduce it: Minimal example which reproduces the bug consistently on my laptop.
Run using the command:
Following exception is raised:
More details: The file which appears to be out of order on my machine is part-5 (
$PWD/out_of_order/pyarrow/0-6e748f6b-f69e-47dd-8857-dc652c73cfef-5.parquet
). This is reproducible when running multiple times, implying it's somewhat deterministic, however it appears that a different "row group" is out of order each time, given that when inspecting the file a different increment number is non monotonic.