search

AlabamaWaterInstitute / data_access_examples

MIT License
7 stars 61 forks source link

Document performance tradeoffs for different s3/zarr access methods #4

Closed jameshalgren closed 2 years ago

jameshalgren commented 2 years ago

@karnesh If you have time to add a few cells to the notebook to show the different timing of the other methods you explored, that might be helpful both in documenting the optimal method we landed on as well as to provide some history of the other options, in case things respond differently in a cloud or HPC environment.

karnesh commented 2 years ago

@jameshalgren I explored using s3fs and fsspec to access NWM data from AWS S3 storage. fsspec provides functionality to perform operations on file objects and is used by various libraries. s3fs is a library that provides a file interface to S3 and uses fsspec. So, there is no difference in time for both these libraries on the local machine.

jameshalgren commented 2 years ago

Closing with this update.