Evidence
I checked for the presence of `✅ This pipeline has successfully finished 🎉` in the AWS Batch logs.
I also spot checked some lines in the generated metadata file:
```sh
# Download and decompress metadata files from latest rebuild and PR rebuild.
aws s3api get-object --bucket nextstrain-data --key files/ncov/open/metadata.tsv.zst metadata-current.tsv.zst --version-id 3wF7ccfqiJ73cw7dVjq.mlrPf5A6Vah6
unzstd metadata-current.tsv.zst
aws s3 cp s3://nextstrain-staging/files/ncov/open/branch/victorlin/centralized-ingest-git-subrepo/metadata.tsv.zst metadata-branch.tsv.zst
unzstd metadata-branch.tsv.zst
# There is no difference in the first and last 10k lines.
# Other differences can be assumed to be due data availability differences based on time of run.
diff <(head -n 10000 metadata-current.tsv) <(head -n 10000 metadata-branch.tsv)
diff <(tail -n 10000 metadata-current.tsv) <(tail -n 10000 metadata-branch.tsv)
```
Evidence
Since I already checked an output file in the GenBank run, here I just checked for the presence of `✅ This pipeline has successfully finished 🎉` in the AWS Batch logs.
[x] Post-merge: Slack notifications still work as intended.
Description of proposed changes
Begin usage of centralized ingest scripts and add details on how to pull new updates from the central repo.
Related issue(s)
Testing
[x] Checks pass (Update-image failure is unrelated and can be ignored. It was triggered by a non-functional change in update-image.yml).
[x] Run GenBank fetch and ingest on PR branch, verify successful run.
Evidence
I checked for the presence of `✅ This pipeline has successfully finished 🎉` in the AWS Batch logs. I also spot checked some lines in the generated metadata file: ```sh # Download and decompress metadata files from latest rebuild and PR rebuild. aws s3api get-object --bucket nextstrain-data --key files/ncov/open/metadata.tsv.zst metadata-current.tsv.zst --version-id 3wF7ccfqiJ73cw7dVjq.mlrPf5A6Vah6 unzstd metadata-current.tsv.zst aws s3 cp s3://nextstrain-staging/files/ncov/open/branch/victorlin/centralized-ingest-git-subrepo/metadata.tsv.zst metadata-branch.tsv.zst unzstd metadata-branch.tsv.zst # There is no difference in the first and last 10k lines. # Other differences can be assumed to be due data availability differences based on time of run. diff <(head -n 10000 metadata-current.tsv) <(head -n 10000 metadata-branch.tsv) diff <(tail -n 10000 metadata-current.tsv) <(tail -n 10000 metadata-branch.tsv) ```[x] Run GISAID fetch and ingest on PR branch, verify successful run.
Evidence
Since I already checked an output file in the GenBank run, here I just checked for the presence of `✅ This pipeline has successfully finished 🎉` in the AWS Batch logs.[x] Post-merge: Slack notifications still work as intended.