Move temporary migration AWS services to storage account

[tomcrane]

There is a mixture of different objects in bagger buckets created by different accounts. We should start again with migration related-data written and read by just one account, as it makes it difficult to report on progress.

• [x] Create new bagger buckets in storage acct, to be used instead of the platform buckets (but leave the platform buckets where they are for now)

~platform/temp-bagit-drop-test~ => storage/bagger-drop ~platform/temp-bagit-drop-test-mets-only~ => storage/bagger-drop-mets-only ~platform/temp-bagit-drop-test-errors~ => storage/bagger-errors

done in: https://github.com/wellcometrust/platform/pull/3236 https://github.com/wellcometrust/storage-service/pull/8

There's a storage account called dds that is used for ad hoc Python migration testing, batch enqueuing of bagging instructions, collecting data about current state of migration per bnumber and writing it to a DynamoDB table. The DDS application also uses this account.

• [x] leave the METS sync bucket (platform/wellcomecollection-assets-workingstorage) where it is, as it's too big to move
• [x] storage/dds account needs to read and list platform/wellcomecollection-assets-workingstorage
• [x] create a new DynamoDB table storage/storage-migration-status in the storage account to replace platform/storage-migration-status table

Applications that use the storage account

bagger

The bagger ECS task, and the storage/dds user:

• [x] Need to read and list platform/wellcomecollection-assets-workingstorage
• [x] Need to read, write and list storage/bagger-drop
• [x] Need to read, write and list storage/bagger-drop-mets-only
• [x] Need to read, write and list storage/bagger-errors
• [x] Need to read from the DLCS bucket in the dlcs account
• [x] Need to read from the Preservica bucket in the systemsstrategy account
• [x] (for local testing only) dds/storage needs to read from queue storage/storageNNNNNN_bagger

Even after this change the bagger works across 2 wellcomecollection estates, the dlcs estate and the systemstrategy estate, but there's no way round that for now.

Local enqueue is used to manually start bagging batches:

• [x] dds/storage needs to enqueue queue storage/storageNNNNNN_bagger

Error reporter runs as storage/dds so gets the permissions above, it uses a subset of what's required above.

Migration tools runs as storage/dds so gets the permissions above, but also means:

• [x] storage/dds needs to read from and write to new dynamoDB table storage/storage-migration-status

DDS (.NET)

• [x] make sure it's using the storage account where it needs to. It shouldn't need to use both platform and storage accounts after these changes.
• [ ] storage/dds needs to read from the access location bucket reported in the storage manifest.

Plan

• Start with clean buckets
• Don't run bulk bagging, ingest and sync ops until we see everything working nicely for just our 15 test examples again
• Make sure all the boxes above are ticked

This shouldn't take very long, it's just testing that permissions are all working in the right places.

Then... Let rip, and revert to plan in https://github.com/wellcometrust/platform/issues/2789

[kenoir]

The bagger seems to be working as expected in the new account.

[jtweed]

@tomcrane I'm going to close this, as we might need to revisit exactly where access copies are vs archive copies. Will keep you updated. I'm hoping we can work out a way to do it without having to change DLCS.