Open JamiePringle opened 1 year ago
A smaller repro:
import numpy as np
import xarray as xr
xr.DataArray(np.arange(5), dims="x", coords={"x": [2, 1, 0, 3, 5]})
da.sel(x=slice(2, 3))
Returns:
<xarray.DataArray (x: 4)>
array([0, 1, 2, 3])
Coordinates:
* x (x) int64 2 4 5 3
da.sel(x=[2, 3])
Returns
<xarray.DataArray (x: 2)>
array([0, 3])
Coordinates:
* x (x) int64 2 3
Yes, good point. What might be possible is to warn if selecting with a slice and the index is not monotonic increasing or decreasing.
So is this an expected behavior? I can work around it by explicitly creating the indices with arange() or the like. I do wonder if this is what is causing to_zarr() to fail even with compute=False? But I can work around that.
Xarray passes the label indexers to the underlying pandas index:
import pandas as pd
# "x" coordinate index
idx = pd.Index([2, 1, 0, 3, 5])
# da.sel(x=slice(2, 3)) does this:
idx.slice_indexer(2, 3)
# which returns slice(0, 4, None)
# da.sel(x=[2, 3]) does this:
idx.get_indexer([2, 3])
# which returns array([0, 3])
What might be possible is to warn if selecting with a slice and the index is not monotonic increasing or decreasing.
Is it always desirable? Asked differently, are there cases where one intentionally wants to select with a slice a non monotonic index? If yes, a warning might be annoying.
Maybe this could be clarified in the docs too?
Jup, that's always the tradeoff - #1613 discusses a similar case.
I am happy to close this; it would be lovely if the documentation was more explicit about this issue. I was certainly surprised even after a close reading of the docs.
Jamie
On Fri, Sep 2, 2022 at 10:07 AM Mathias Hauser @.***> wrote:
CAUTION: This email originated from outside of the University System. Do not click links or open attachments unless you recognize the sender and know the content is safe.
CAUTION: This email originated from outside of the University System. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Jup, that's always the tradeoff - #1613 https://github.com/pydata/xarray/issues/1613 discusses a similar case.
— Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/6976#issuecomment-1235549943, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADBZR25KUX6CEF5ESPUJLKLV4ICYRANCNFSM6AAAAAAQCNUNFM . You are receiving this because you authored the thread.Message ID: @.***>
What happened?
I am not sure if what I report is a bug; however, it is certainly not what I expect from a careful reading of the documentation, and I wonder if it is leading to some issues I describe below.
I am working with a large dataset produced by merging the output of runs made by different MPI processes. There are two coordinates, ("trajectory","obs"). All of the "obs" in the dataset are in order, but the "trajectory" coordinate is not in order. I made a smaller dataset that illustrates the issue below by reducing the number of "obs" from 250 to 2; this dataset can be found at http://oxbow.sr.unh.edu/data/smallExample.zarr.zip . This dataset looks like:
Note that the trajectory coordinate is not in order; this is due to how the problem is partitioned into MPI jobs.
If I want an ordered set of trajectories, say trajectories [1,2,3,4,5,6,7,8,9,10], and I do this with
subSet=dataIn.sel(trajectory=arange(1,11))
, I get what I would expect from the documentation: The first through 10th trajectories, in order:But if I use the slice operator to specify what I want, I get something very different:
dataIn.sel(trajectory=slice(1,11))
returns 2567339 trajectories, starting with the location of trajectory coordinate 1 in dataIn and extending to the location of trajectory coordinate 10 in dataIn:This is not what I expect -- as I understand the documentation, .sel should work in coordinate space, and I would expect
dataIn.sel(trajectory=slice(1,11))
andsubSet=dataIn.sel(trajectory=arange(1,11))
to return the same thing. If I am wrong in this interpretation, perhaps a documentation update would be helpful.I have had all sorts of issues with the full dataset, including .to_zarr(dataset, compute=False) failing to return a delayedObject because it used all the memory,
.sortby(['trajectory'])
failing by memory exhaustion, etc. I wonder if the issue reported here can be at the root of many of these issues? On a side note, the failure of.sortby(['trajectory'])
makes re-ordering the dataset difficult, and I would be happy to hear any suggestions on that front.What did you expect to happen?
See above for full descriptions
Minimal Complete Verifiable Example
MVCE confirmation
Relevant log output
No response
Anything else we need to know?
No response
Environment