search

traitecoevo / austraits-api-nectar

0 stars 1 forks source link

Retrieve local copy of AusTraits data before launching API #5

Closed franksc closed 2 years ago

franksc commented 2 years ago

Stash AusTraits data file/s (austraits-x.y.z.rds) somewhere 'local' such as a Nectar Swift bucket and retrieve to local storage on an instance before austraits-api is launched, so its invocation of austrait's load_austraits() will use this local copy rather than heading off to Zenodo to download.

This will work around any ephemeral slowness or flakiness in downloading data files from Zenodo. Note load_austraits() first queries Zenodo API for all AusTraits releases, so to eliminate dependency on Zenodo would require also locally caching a copy of this info.

dfalster commented 2 years ago

Good idea!

The load_austraits function takes a path argument. It only downloads the data if the file is missing at the path destination. Could we add the bucket as a path?

load_austraits(path="/nectar_volume/path")
franksc commented 2 years ago

Openstack Swift is object storage you interact with over an HTTP-accessible API (or using a client wrapping around this), not something you mount like an NFS volume.

Approach here would be to retrieve from Swift bucket to /some/local/path before doing load_austraits(path="/some/local/path")