ITISFoundation / osparc-simcore

🐼 osparc-simcore simulation framework
https://osparc.io
MIT License
43 stars 27 forks source link

Replace RClone sync by AWS CLI sync #5833

Closed sanderegg closed 1 month ago

sanderegg commented 4 months ago

After extensive testing in https://github.com/ITISFoundation/osparc-simcore/discussions/5828 it showed that the AWS CLI is approximatively 1.5-2x faster at transferring data between local path and S3 than the current RClone sync used in dynamic-sidecar and agent.

It also is backed by AWS directly so should hopefuly be more stable (RClone 1.66.0 upgrade was a disaster as uploading would create kernel panics).

Since boto3 does not implement these tools, it would be needed to wrap the AWS CLI, checking whether there are any available tool already doing this might be worth the time.

### Tasks
- [x] Check for wrappers around AWS s3 sync call
- [x] implement wrapper of AWS s3 sync call with tests
- [x] ensure progress is parsed while transfering
- [x] replace rclone sync in dynamic-sidecar
- [ ] replace rclone sync in agent
- [ ] remove rclone ones aws cli is tested and used
matusdrobuliak66 commented 1 month ago