Open mrocklin opened 6 months ago
I'd also like to sort partitions by max value of the partition column, but couldn't find an easy way to get statistics out of the metadata
The min/max values are available through the deltalake.DeltaTable.get_add_actions API mentioned in this issue: https://github.com/delta-io/delta-rs/issues/2233#issuecomment-1971991963
I'm going to reopen this for now. I think that we could probably do better by looking at the data coming out of the get_add_actions
function.
I've stored a bunch of data partitioned by date, and written it to delta using the deltalake package like so:
(although actually this was done in parallel, and so things maybe got written out of order
When I go to read it I find that the data isn't sorted by partition
We should order things if we can I think. I propose the following:
"stats"
attribute of the deltalake metadataProbably both the effort and uncertainty increase as we go down that list. The first item seems pretty straightforward to me though.