Open SCA-ZMT opened 5 months ago
@mguidon could you please edit the issue description with more info / tasks?
First brainstorming took place with @sanderegg, @mguidon, @matusdrobuliak66. We would like to experiment with AWS Elastic File System in three stages:
We may potentially investigate the AWS DataSync option, but for now, we prefer to stick with RClone.
Why?
@matusdrobuliak66 @sanderegg @mguidon why was rclone discarded? Has anybody thought about rclown mount
which uses fuse and will stream data on the go on access and use a cache for writing back to S3.
This will allow us to start services without waiting. Also saving will go to S3 directly. So once the user is done interacting with the FS, it's as if the file was already on S3.
@GitHK rclone is not discarded. we go stepwise. and until fixed, RClone presents some bad habit of blowing up without anyone knowing why.
@GitHK Indeed, we did not discarcd it, we go step by step. For now rclone sync
should do exactly what we want without the need for fuse
.
After some issues with rclone, it will be phased out in favour of aws s3 sync
. If we require fuse, we could use s3fs-fuse which is also managed by AWS.
We want to have a quick startup time and get rid of buffer machines.
docker image load
/var/lib/docker
data on the distributed file system as docker stores there also information about containers and it might be problematic if used by more people. Therefore I was looking for a way how to move only .../image
and .../overlay2
directories via symbolic link. In any case, it seems it is not possible to do it on EFS (nfs4 type) because I have encountered an issue with moving "backingFsBlockDev" file https://forums.docker.com/t/moving-var-lib-docker-to-amazon-efs/102070s3fs
aws s3 sync
or rclone
or data sync
to sync between S3 and EFS. As mention by @sanderegg if we setup correctly S3 Endpoint + Region where EC2 <-> S3 are located we can have zero S3 costs!
One of them links that I already did before
docker load --input /mnt/efs/fs1/docker_images/android.tar
from ECR --> A 6.5GB image saved as a tar file in ECR took 4 minutes.Two working examples (mounting EFS to docker container):
docker volume create --driver local --opt type=none --opt device=/mnt/efs/fs1/matus_docker_volume --opt o=bind my_limited_volume
docker run -d --name my_container -v my_limited_volume:/data redis
version: "3.7"
services:
wp_gary_gitton:
image: wordpress:6.3
volumes:
- wp_gary_gitton:/var/www/html/wp-content
volumes:
wp_gary_gitton:
driver_opts:
type: nfs
o: addr=fs-
It seems there is not straight forward option how to limit docker volume size when mounting. EFS doesn't support quotas. In any case I think it is not necessary. We can use CloudWatch and for example Lambda to monitor and manage EFS usage programmatically.
- Script to Monitor Directory Sizes: A script runs periodically to check the size of each project's directory and sends this data to CloudWatch.
- CloudWatch Alarms: Alarms are set up to monitor the directory sizes and trigger actions when limits are exceeded.
- Lambda Function: A Lambda function handles the alarms and takes appropriate actions, such as enforcing policies.
- For example: Using EFS Access Points with IAM Policies -> we can restrict access based on some policy.
@matusdrobuliak66 about differences between shutdown and termination, please look here: AWS reference, this link was also already referenced from before. Now regarding what you say with the EBS volume as "active", since on start/stop mechanism the actual "machine" changes are you sure that the access times when you restart the machine are on par with a hot machine? There are simple tests (like the pre-warm up test of AWS) to check that. if parsing the blocks takes a long time then that means the volume is there but 10-100 times slower. Please check it. Keeping volumes around is costly.
Another question, how do you setup the EBS volumes without EC2s?
Thanks @matusdrobuliak66. This looks like a plan. Lets keep rclone on the radar for mounting other filesystems (e.g. sftp
and 3rd party providers).
I suggest to immediately go on with 1, 2 and 4 and continue investigating 3.
testing:
EFS (Bursting Mode)
EFS (Elastic Mode)
Note:
nfs
type is not optimized for that.Mounting EFS to the Simcore node: https://git.speag.com/oSparc/osparc-infra/-/merge_requests/230#7af1093b9e6a1427b21a22270b4df48cc5311648
This is how you can enable it for testing purposes: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/639
NIH milestone
This is also a milestone for NIH, due by Y8Q2 (Feb. 2025)
Large Files
Improve handling of large files and reduce file-transfer operations as much as possible. This requires some research. The solution should ideally also work on-premise. Storage might need to be compeletey overhauled.
Data Accessibility [This is a Milestone for NIH Year 8 Q2]
Provide infrastructure to users to inspect/download/zip/delete files. This is a Milestone in NIH Year 8 Q2 (see https://github.com/ITISFoundation/osparc-issues/issues/1635)
Shared Folder(s) - Advanced
Provide users the possibility to mount arbitrary data into their services: