Add ability to create 'archival bundles'

simonsobs / librarian

The HERA Librarian.

BSD 2-Clause "Simplified" License

1 stars 3 forks source link

Open JBorrow opened 1 week ago

JBorrow commented 1 week ago

For writing to backup, it would be helpful to have an archival bundler that:

Tracks incoming files and once a limit of un-archived data is reached raises an alert
That list can be used by an external tool to generate an archive bundle
That archive bundle can be sent to tape/external disk/etc
The archive bundle can be registered back in the database

JBorrow commented 1 week ago

Could even fully automate this process for globus- or mounted- accessible storage systems like: https://docs.nersc.gov/filesystems/archive/#globus