Add some kind of protection for accidental deletion of KG-COVID-19 builds

[justaddcoffee]

Right now we could in theory nuke existing KG-COVID-19 builds with an errant s3 command, which isn't ideal.

Describe the desired behavior

It'd be desirable to have some options if KG-COVID-19 builds were nuked. Some options:

• turn on versioning of the S3 bucket where builds are stored (possibly as easy as this command)
• backups/redundant storage:
  • S3 glaciar
  • push to Zenodo as GO does

Additional context

[justaddcoffee]

@kltm I was thinking for now to address this ticket we could just turn on versioning of that S3 bucket - cost is minimal IIUC

[kltm]

@justaddcoffee Does S3 versioning actually prevent deletion? If the object is gone, isn't it gone? Also, if one is thinking "backup", more S3 doesn't necessarily cover redundancy. If one is thinking permissions, more buckets with different access.

[justaddcoffee]

Does S3 versioning actually prevent deletion? If the object is gone, isn't it gone?

@kltm nope, if I understand the docs correctly, it does actually prevent deletion:

If you delete an object, instead of removing it permanently, Amazon S3 inserts a delete marker, which becomes the current object version. You can always restore the previous version

[justaddcoffee]

Also, if one is thinking "backup", more S3 doesn't necessarily cover redundancy. If one is thinking permissions, more buckets with different access.

This is true - I'm avoiding the word backups when referring to S3 versioning, as this isn't really a backup

[kltm]

@justaddcoffee Fair enough on both points. It might be worth actually looking at how a "recovery" would work in practice and have basic tooling for it first (from painful memories of "theoretical" safety systems in the past). I guess I think that this is sort of a neither-this-nor-that solution--not a backup and not a permissions system. It does have the advantage of being trivial to implement (although the costs may accumulate unless you have additional operations in the autiomated lifecycle/), but I might check out exactly how it would work in practice first.

[justaddcoffee]

Okay, I've enabled versioning for the kg-hub-public-data bucket, and tested that I can actually delete something, then undeleted it (or more accurately, delete the 'DeleteMarker' in AWS parlance).

This doc is really helpful: https://aws.amazon.com/premiumsupport/knowledge-center/s3-undelete-configuration/