Writing COG to S3? - Githubissues

pangeo-data / cog-best-practices

Best practices with cloud-optimized-geotiffs (COGs)

BSD 3-Clause "New" or "Revised" License

78 stars 9 forks source link

Writing COG to S3? #11

Open rsignell-usgs opened 3 years ago

rsignell-usgs commented 3 years ago

A colleague asked me how to write COG to S3 from xarray, and I got something working using fsspec simplecache: https://nbviewer.jupyter.org/gist/rsignell-usgs/3472ca07779ded81833a0cd3deb24cf9

but I'm wondering if:

there is a better way (a best practice!) to do this?
if yes, whether that best practice might be worth demonstrating in a notebook?
(of course it wouldn't work in general -- folks would need to supply their own bucket and credentials, but might still be useful)?

scottyhq commented 3 years ago

thanks @rsignell-usgs definitely would welcome a notebook on writing COGs here! Looking quickly at your notebook I see "we just basically writing to local disk then pushing to S3". It's definitely possible to upload from memory directly to S3... I don't have an example on hand, but should document the two approaches.