Allow versioned resource access for core datasets

[jameshadfield]

See commit messages for details. Some URLs as an example:

• https://nextstrain-s-james-reso-cterli.herokuapp.com/enterovirus/d68/vp1 -- the current dataset with 2395 genomes. Uploaded 2022-09-06.
• https://nextstrain-s-james-reso-cterli.herokuapp.com/enterovirus/d68/vp1@2022-09-06 -- The same as the above URL, but specifying the version (so the dataset won't change over time)
• https://nextstrain-s-james-reso-cterli.herokuapp.com/enterovirus/d68/vp1@2022-05-31 -- 2345 genomes. Includes a v2 main JSON, tip-frequencies & root-sequence sidecars. (If you are watching the server logs you can see the S3 version IDs for these files as they are fetched, but these are not exposed to the client.)
• https://nextstrain-s-james-reso-cterli.herokuapp.com/enterovirus/d68/vp1@2022-05-30 -- doesn't exist. Will redirect to 2022-05-31 (above)
• https://nextstrain-s-james-reso-cterli.herokuapp.com/enterovirus/d68/vp1@2019-09-17 -- 1653 genomes. a v1 (meta + tree) dataset, with no sidecars

Ready for review, but there are a few things before this can be merged:

• [x] More testing needed / tests added?
• [x] Run the manifest generation script somewhere (GitHub actions of a private repo on a schedule?) and upload the results to S3. This can be done following merge but we need a plan on how to implement it. See #747
• [x] Update the manifest on the server. The easiest is probably on a 1-hour interval and check for ETag changes.
• [x] Remove console.log statements
• [x] Ensure the IAM terraform configs have been applied to AWS Update: these were added in 1019326ebc378648e51ee54cf1c5024ab7460b85, and as of 2024-01-02 AWS has the correct permissions
• [x] As of 2023-11-07 (NZ), versioning has been enabled on nextstrain-inventories. Rerun the indexer in a day or two, once past versions of the inventory files have been deleted and thus exist as versions. (I don't expect any changes.)

[tsibley]

Talked with James in our 1:1. This is ready for me to review. He chose not to implement a new ResourceVersion class (as in Source → Resource → ResourceVersion → SubResource) as we'd talked about two weeks ago, but instead make the existing Resource class version-aware.

[tsibley]

I rebased this onto current master and made the appropriate changes to the server's IAM policies managed by Terraform: 4a0367f. I also dropped 5a33a9e, which was marked for dropping and which we don't want to keep.

[tsibley]

@jameshadfield Thinking about my review and suggestions some more, I wanted to offer to implement some of the larger changes I've suggested and then walk you thru them if it'd be more clear for me to do that rather than just discuss them. Not a problem if not. Totally your preference here.

[jameshadfield]

@tsibley could you re-review this at your convenience? All comments should have been addressed, except for (i) log statements which I'll remove later and (ii) the definition of "closest" / redirect to closest code which I'll defer until the wider group makes a decision on desired behavior here. The S3 resources index has been updated to the new format.

I wanted to offer to implement some of the larger changes I've suggested and then walk you thru them if it'd be more clear for me to do that rather than just discuss them

Thanks, but I'd prefer to discuss the concepts rather than read through an implementation. For what it's worth, none of the changes were large in terms of code changes, but conceptually I guess there are a few changes.

[tsibley]

Some other todos:

• [x] Indexer deployment plan See #747
• [x] nextstrain-inventories bucket in Terraform Moved to #748
• [x] Lifecycle for expiring inventories in nextstrain-inventories in Terraform Moved to #748
• [x] Inventory config for nextstrain-data in Terraform Moved to #748
• [x] Inventory config for nextstrain-staging in Terraform Moved to #748
• [ ] Adopt AWS config support added to master since this was branched. Probably swap away from v2 SDK and add necessary functions to src/s3.js?
• [x] Interaction with PUT/DELETE/OPTIONS methods is likely wrong. Not an immediate problem because while we have endpoints for those methods on core/staging, we don't use them. But as-written the interaction/behaviour if they did run is incorrect, and it will be an issue for Groups or if we start using those methods for core/staging (which I think we should). This has been moved to #744

Some of these are deferrable but they all need doing sooner than later.

[jameshadfield]

@tsibley could you work through your comments when you have a chance? I believe I've addressed them all. (And please resolve the ones you feel are addressed, else with this many comments the PR becomes hard to read through.)

Adopt AWS config support added to master since this was branched. Probably swap away from v2 SDK and add necessary functions to src/s3.js?

Could you add an in-line comment regarding this please. I presume you're talking about the AWS calls within updateResourceVersions?

[tsibley]

@jameshadfield Yep. Was updating myself a bit on this today. Will continue to dive in more.

Could you add an in-line comment regarding this please. I presume you're talking about the AWS calls within updateResourceVersions?

Yes, those. Comment added.

[tsibley]

I've done another full review here, including the new code and revisiting past comments and resolving them as appropriate.

[jameshadfield]

As well as addressing all the open conversations above, I've added a large number of tests to give confidence to the behaviour across sources and resource types. I'm going to merge this now, but feel free to make suggestions which we can address in future PRs. There's more work I'd like to do now that the server has access to the resource index. Many thanks to the reviewers - it's been a big PR, almost certainly with the most comments of any in Nextstrain's history.

P.S. I couldn't convince Heroku to rebuild the review app (unrelated to the contents of this PR), but I did a final round of testing via dev.nextstrain.org, which has the benefit of allowing logins.