Content Ingest Workspace Design - Githubissues

CDLUC3 / mrt-doc

Documentation and Information regarding the Merritt repository

8 stars 4 forks source link

Content Ingest Workspace Design #1767

Closed terrywbrady closed 1 week ago

terrywbrady commented 8 months ago

https://github.com/CDLUC3/mrt-doc/tree/main/design/content-ingest-2024

See also https://github.com/CDLUC3/mrt-doc-private/issues/69

Eventually, this is how we will handle the ingest workspaces:

Create workspace project (S3, EC2, Lustre, Lambda) via Sceptre
- Project Name
- Size: 1.2T or a multiple of 2.4T
- Note that this will create a baseline cost for the life of the project. Eventually, each workspace should be torn down and destroyed.
- Projects can be re-sized within these constraints
- Note: 2 different projects can share the same workspace. Projects should only share the same workspace if they will be on-going for the same timeframe.
Content should be purged from S3 as it is verified in Merritt
Tear down project upon completion

terrywbrady commented 4 months ago

Thought from Ashley and Marisa: create this as a resource in the UC3 account.

terrywbrady commented 1 month ago

8/15 @ashleygould recommended XFS as an option to consider in addition to EBS.

8/22 Terry and @elopatin-uc3 discussed the copy process. We should performance test the following

scp from Mac to EFS from office on wifi
scp from Mac to EFS from office on ethernet
s3 upload from Mac on wifi
s3 upload from Mac on ethernet

https://github.com/CDLUC3/mrt-doc-private/issues/101