search

ropensci / bowerbird

Keep a collection of sparkly data resources
https://docs.ropensci.org/bowerbird
Other
48 stars 6 forks source link

sync azure blob storage to local? #39

Open zackarno opened 1 month ago

zackarno commented 1 month ago

I was wondering if it is possible to sync azure blob storage to local with bowerbird?

I was reading the defining data sources section of the getting started guide and see there are some arguments for aws s3. These actually seem intended to sync your object storage from other sources. But I was wondering if there was a way to sync your object storage to local.

I have some large (temporally) COG stacks that just tend to cause a time out when running standard processing on them directly from remote object storage so I thought that bowerbird could be very useful for keeping a local copy synced for select data sources like this.

raymondben commented 1 month ago

Sorry for a slow reply. Haven't done anything with Azure to date - do you have an endpoint that I can look at to see what might be required? (Is it a STAC catalog?) For what it's worth, the S3 support in bowerbird currently does remote bucket -> local file system, or remote provider (http/ftp/whatever) -> remote bucket. But I can't remember off the top of my head if sourcing from a remote bucket will work with one that requires authentication.