search

unity-sds / unity-sps-prototype

Apache License 2.0
2 stars 7 forks source link

[New Feature]: Add an optional PVC to an SPS deployment #192

Closed LucaCinquini closed 1 year ago

LucaCinquini commented 1 year ago

When deployment SPS, there should be an option to provision an optional PVC of desired size. This is needed to support large data processing like in the case of the CHIRP workflow, which requires processing of 16 days of data at one time.

The PVC should be backed up by EBS, and be mounted in all the containers of the Verdi pod: the Verdi container and the Docker-In-Docker container. Maybe use a mounting point as /stage or /cache.

LucaCinquini commented 1 year ago

Apparently mounting a PVC on every single node is not supported for a Kubernetes DaemonSet deployment, so we ended up mounting a common EFS partition on all worker nodes for the time being.