When loading an Ensemble, you can currently set whether the data is already sorted and whether to sort it via the respective sorted and sort boolean flags.
However currently loading an unsorted dataset with sort=False and sorted=True will produce an unsorted index.
Note however that though each individual partition is unsorted, the divisions are still set (all values within a partition are within a given range that doesn't overlap with other partitions). However we should still make sure the data is sorted or change the description of the flag.
When we load with sort=True and sorted=False, we do get a sorted index but now there are warnings that the divisions are not set.
@dougbrn noted that this seems related to dask-expr issue https://github.com/dask/dask-expr/issues/975 where we lose divisions with only one partition after a reset_index call. This is mildly annoying to fix since our npartitions parameter only triggers the repartition call after we have reset the index and thus already lost the divisions.
When loading an
Ensemble
, you can currently set whether the data is already sorted and whether to sort it via the respectivesorted
andsort
boolean flags.However currently loading an unsorted dataset with
sort=False
andsorted=True
will produce an unsorted index.Note however that though each individual partition is unsorted, the divisions are still set (all values within a partition are within a given range that doesn't overlap with other partitions). However we should still make sure the data is sorted or change the description of the flag.
When we load with
sort=True
andsorted=False
, we do get a sorted index but now there are warnings that the divisions are not set.@dougbrn noted that this seems related to dask-expr issue https://github.com/dask/dask-expr/issues/975 where we lose divisions with only one partition after a
reset_index
call. This is mildly annoying to fix since ournpartitions
parameter only triggers therepartition
call after we have reset the index and thus already lost the divisions.