Add MANIFEST files to release.geneontology.org

[kltm]

From discussion around geneontology/helpdesk#307, we'd like to start adding general MANIFEST files to the release directories and the release.geneontology.org top level. Talking to @cmungall (owner), we are looking at:

• MANIFEST in the top bucket (root of release.go.org) of all "directories"
• MANIFEST in the "root" of each release (e.g. /2020-06-01/MANIFEST) that lists /every/ file in that sub-bucket/directory (but not "directories", just the files)
• Exact format to be TBD, but the awsclient listing (or other unix-like plaintext tsv-ish listing) may be fine; perhaps consider JSON?

Notes:

• We're talking about a literal file named "MANIFEST"
• This is applied after archiving to zenodo the deployment to S3, so is not showing up in the Zenodo archive, which has its own manifests with bdbags.
• This will also be retrospectively applied to previous releases
  • (and will be filtered when putting the previous releases into Zenodo)

For consistency, and since there's only relative paths anyways, these should also be added to snapshot, current, etc.--just the normal release.

To accomplish this, we'll have the following tool uses:

• something to list all of the directories at the top level of release.go.org (probably something like awsclient) and produce a manifest file
  • something to push that to S3 (or accomplished with established tools in the pipeline)
• something to list all files in a release and produce a manifest file
  • something to push that to S3 (or accomplished with established tools in the pipeline)
  • manually or automatically do this for previous releases

[lpalbou]

That script I created to push the index.html to every S3 folders to get the archive page, could easily be transformed to create the manifest: https://github.com/lpalbou/aws-js-s3-explorer/blob/4c248661dff339deb574fa42bc7e78a2faa89090/s3_add_file.py#L17

Essentially, remove the filter on "virtual folders": if prefix_name.endswith('/'):

[dustine32]

Oh very cool. Thanks @lpalbou!