intake / intake-xarray

Intake plugin for xarray
https://intake-xarray.readthedocs.io/
BSD 2-Clause "Simplified" License
74 stars 36 forks source link

Add logic for remote or local files in NetCDFSource #93

Closed scottyhq closed 3 years ago

scottyhq commented 3 years ago

This PR follows up #82 to do a few things

Fixes:

cc @rabernat @martindurant

scottyhq commented 3 years ago

quick update - I added quite a few new tests under test_remote.py and also added test_network.py which includes simple examples from linked issues reading over the network from http, s3, gcs (these currently are not run via CI though).

added a py39 environment file and test, and now install xarray master for upstream CI test.

I realized in putting a few more tests together that open_rasterio() required chunks as a positional argument, so I set that to a default of chunks=None since it's sometimes nice to bypass dask. This is checked in test_remote::test_open_rasterio.

martindurant commented 3 years ago

Looks good!

For s3, you could use the same moto fixture that s3fs does, rather than hitting the real backend. I would not try to implement the gcsfs fixture here.

scottyhq commented 3 years ago

For s3, you could use the same moto fixture that s3fs does, rather than hitting the real backend.

Woohoo! That took some figuring out, but I've added basic tests for rasterio and netcdf with moto.

If it's okay with you @martindurant I'll go ahead and merge?