wellcomecollection / goobi-infrastructure

Wellcome Collection digital workflow infrastructure
MIT License
0 stars 3 forks source link

check and possibly reduce storage for (fargate) worker nodes #432

Open mgeerdsen opened 2 years ago

mgeerdsen commented 2 years ago

After switching bagit creation to EC2/EBS we should check the storage usage of the fargate based worker nodes:

Also we should check if we can simplify the workernode scripts if we don't need to decide which storage area to use anymore.

mgeerdsen commented 2 years ago

The following jobs use the WORKING_STORAGE_FAST exclusively if available:

only use WORKING_STORAGE:

All these use exactly one storage area and make no decision based on size (as bagit does), so we only need one kind of storage. As WORKING_STORAGE_FAST is set to ephemeral storage, this means that nearly all jobs already use that and nearly none use the EFS.

If no jobs have had any trouble with the ephemeral storage, we should check the remaining job types and could get rid of one storage type (probably EFS).