scimma / blast

Django web app for the automatic characterization of supernova hosts
MIT License
1 stars 2 forks source link

Version control of initial data set using S3 bucket versioning #258

Closed manning-ncsa closed 1 month ago

manning-ncsa commented 1 month ago

Fixes #256

This update introduces a new system for defining and verifying the integrity of the required initial file set. A new Python script generates a file set manifest from the objects in our dedicated S3 bucket, and the same script is used to parse that file upon application startup to validate and, if necessary, download required files. As Blast evolves and files are added or modified in this set, the S3 bucket versioning provides the version-control needed to recover previous states of the file set.