search

intake / intake-esm

An intake plugin for parsing an Earth System Model (ESM) catalog and loading assets into xarray datasets.
https://intake-esm.readthedocs.io
Apache License 2.0
130 stars 42 forks source link

ENH: Searching within a time interval #623

Open aulemahal opened 11 months ago

aulemahal commented 11 months ago

(I first made this as a comment to PR #291, but I realized afterwards that the PR had been staled for three years, so I figured out that a new issue might be more appropriate.)

We had this issue and need when writing xscen. Our solution does not yet feel generic enough to be implemented here, but if you want to have a look at a first version is here : https://github.com/Ouranosinc/xscen/blob/b261a04ed73e398a60a0632bdb29be324dc3f5b6/xscen/catalog.py#L899-L998

The idea is similar to the staled PR. You have a "date_start" and a "date_end" column in the catalog. For a given "period", the code returns the row where the date_start - date_end interval overlaps with that period. Because of the limitations of pandas <2, we had to use pd.Period objects in our catalogs and the code suffers a bit from this workaround.

In addition to a simple "overlap", our function tries to guess the percentage of the period that is covered by the rows of the dataframe, so we only return the subset if a significant percentage is obtained. This has the restriction that it make sense if the rows of the dataset are not temporally overlapping, like for a single variable divided temporally in multiple files. A "full overlap" would often be too strict because of so many caveats (different calendars, imprecise date bounds).

I recently tried to use datetime64[ms] columns with pandas >= 2, which allows to simplify the function a bit and use more pd.Interval magic. It is here: https://github.com/Ouranosinc/xscen/blob/05054bfbf450c6b332e239e7866f766f51a47ed0/xscen/catalog.py#L892-L974.

There are still some caveats and questions to answer I think.

With input from the intake-esm devs and users, we (Ouranos) could consider investing some time into adapting and upstreaming our solution as we would be more than happy to make xscen thinner.

dcherian commented 11 months ago

cc @klindsay28 who has worked on a solution for this

An alternative suggestion is to kerchunk, and then just subset using .sel so zarr handles the subsetting to appropriate files.

aulemahal commented 11 months ago

@dcherian indeed! I guess I need to go back and see if kerchunk works well with local netCDFs ? Last time we looked (more than a year ago), it seemed to add more issues than it solved.

Further info : One way we use this functionality in xscen is to find which datasets in the catalog provide the given time period, but then we use the result to repeat the search without a time period and use the actuel full dataset. EX: I want to run my computation on the full simulations, but I need at least the 1990-2020 period. Could be nice if a final implementation would to this logic in a single search (return the dataset if is contains the period, without subsetting).