Closed rsignell-usgs closed 6 years ago
I'm very glad to see this.
Some things that would be interesting to try if anyone has time:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.
I was able to successfully run a demonstration notebook accessing data from HSDS, which, like
zarr
, stores HDF5 or NETCDF4 datasets as chunks, with each chunk in an S3 object.In the sample notebook here, I'm accessing data from an HSDS instance on XSEDE, yet the access times are comparable to running the same notebook on XSEDE. Google Cloud and XSEDE are connected via Internet 2, I assume.
To run this notebook on pangeo as I did, you would need to:
h5pyd
custom conda environment (see below)nb_conda_kernels
to theroot
environment so that the custom kernel list appears.Here's the procedure I used for creating the
h5pyd
environment:For more info on HSDS, check out John Readey's Scipy 2017 talk on HSDS