[Storage] Investigate using rclone sync to replace gsutil rsync

[landscapepainter]

According to some user report, rclone sync is an order of magnitude faster than gsutil rsync, which SkyPilot uses to upload from local node to cloud storage and to download from cloud storage to local node.

The user reported a case of fetching millions of images and json files worth of ~500GB from the storage to the node:

Yeah, in my case, it was a dataset with millions of images and json files. The full dataset was ~500GB. Not sure the average file size.

With some loose benchmark on 4 categories below, it turns out that using rclone sync is much faster than gsutil rsync in a case there's huge number of small files being fetched from cloud storage(GS) to node(GCP).

1. uploading 6 of 10GB files from local node(GCP) to cloud storage(GS)
2. uploading 1000 of 1MB files from local node(GCP) to cloud storage(GS)
3. downloading 6 of 10GB files from cloud storage(GS) to local node(GCP)
4. downloading 1000 of 1MB files from cloud storage(GS) to local node(GCP)

We should investigate if rclone sync has all the features we need to run smoothly with SkyPilot, and replace gsutil rsync with rclone sync if feasible.

Update: Added a new benchmark of gsutil -m copy -r, and used a larger amount of data using 10000 of 1MB files as opposed to the previous benchmark above which has 1000 of 1MB files.

[dtran24]

Hey, happy to pick this up! Some thoughts so far:

• Seems like the upload method will be a good place to start looking at making changes https://github.com/skypilot-org/skypilot/blob/76c5af9cc24663528d5e5f57253f252a9aa4d8bf/sky/data/storage.py#L1531-L1558
• While I haven't done any manual tests yet, from a cursory reading docs and community posts of gsutil rsync and rclone sync, seems like rclone can replace gsutil rsync. Some community posts:
  • https://github.com/rclone/rclone/issues/1275
  • https://www.reddit.com/r/rclone/comments/vzto6g/can_rclone_replace_rsync/

Please let me know if I'm on the right track, or if there's anything else I should know before diving more into the implementation.

[landscapepainter]

Welcome to Skypilot @dtran24!

upload method is a great place to start. After, you can take a look into _execute_file_mounts with cloud_stores.py/make_sync_dir_command and make_sync_file_command for fetching files/dirs from GCS to remote VM.

One thing to note is that IBM COS uses rclone as well, so we have an abstraction for rclone, data_utils.Rclone, that can be utilized.

[sqr00t]

Heya! I wonder if it's worth to add exploring s5cmd. For AWS, there's some benchmarking results to take with a grain of salt. Maybe an adaptor over this would allow a more unified API across clouds with better performance?

[landscapepainter]

Hey @sqr00t, thanks for sharing the benchmark results and suggestion. We actually do have a PR for adding s5cmd with crt client! Are you an active user of s5cmd outside of skypilot? Was wondering if you encountered any edge cases while using it compared to aws cli :)

[aseriesof-tubes]

Hi, I'm interested on working on this! If nobody's working on it, could I give it a shot?

[landscapepainter]

Hey @aseriesof-tubes thanks for taking on this issue. It's a very important issue to enhance user experience. I just assigned you to the problem.

[github-actions[bot]]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.