Open zaneselvans opened 1 year ago
I think having success/failure conditions like this is a really good idea for making automated archives useful and hopefully catch errors early.
In general we expect the set of data partitions to either remain constant or grow over time
I think we could probably fail or at least require human review any time we would delete a partition outright as that's almost always unexpected.
Another thing we should probably start considering is some procedures for handling failures. For example, we should plan some sort of human intervention mechanism if we deem an archive to actually be acceptable even if it does generate a failure.
Our goal is to have the archivers running on an automated schedule in the background, taking snapshots of the original data sources which can be accessed programmatically. This will minimize the overhead associated with keeping our raw inputs up to date, but we still need the system to alert us when something goes wrong so we can fix it.