EPIC: Archive workflow support - Githubissues

cedardevs / onestop

OneStop is a data discovery system being built by CIRES researchers on a grant from the NOAA National Centers for Environmental Information. We welcome contributions from the community!

GNU General Public License v2.0

43 stars 21 forks source link

EPIC: Archive workflow support #1381

Open dneufeldcu opened 3 years ago

dneufeldcu commented 3 years ago

Summary

As a	a data archive
I want to be able to	preserve data for the long-term at a reasonable cost
So that I can	feel confident in cloud based archival processes

Description

Move files landing in s3 to glacier based on temporal and parentId rules. Be able to restore the data to s3. Understand any AWS limits.

Tasks

[x] Design session and documentation of discussion
[x] Know when a file goes into glacier
[x] Persist the identifier and location of the file
[x] Be able to retrieve the file from glacier

Intelligent Storage

https://docs.google.com/document/d/1ga10PGamia4Y7-Hl4qWFNJbzr-QlGuPilBNv14y6SDg/edit#heading=h.bh4kpw8i4ph6

caesterlein commented 3 years ago

relevant - https://docs.google.com/document/d/1nxYJD4tk6yfT3GdGNHCR8lH8lfYRO4C364ZrJ68NVVk/edit?usp=sharing

dneufeldcu commented 3 years ago

Currently we are using the 'glacier' resource type in boto3. This should be switched to s3 resource as glacier api will be deprecated.