[Jobs] Allowing to specify intermediate bucket for file upload

[zpoint]

Feature for #3978

• Add a config jobs: bucket in ~/.sky/config.yaml.
  • Support s3, gcs, azure, r2, ibm
  • If this bucket is specified, the bucket name and storage type will be used. Otherwise, one will be generated named: skypilot-filemounts-{username}-{run_id}.
  • Add an attribute for storage YAML config: _bucket_sub_path(Support both COPY and MOUNT based on the _bucket_sub_path, The user will only see the subpath instead of the whole bucket if a subpath is specified)
    • _ means internal use only, remove _ to expose in the future
  • smoke test
    • Modify smoke test case: pytest -s tests/test_smoke.py::test_managed_jobs_storage
      • Verify that the new config file is loaded and intermediate bucket names in the config are used.
    • Add a test case: pytest -s tests/test_smoke.py::TestStorageWithCredentials::test_bucket_sub_path --gcp
      • Verify that the bucket subdirectory file sync and deletion have the expected behavior. Mainly, ensure the _bucket_sub_path works as expected.

For managed jobs:

• All these file sync operations will now use the intermediate buckets. Previously, each file sync would create a new temporary bucket:
  • Previous:
    • workdir specified in the task.yaml workdir -> {tmp-bucket-A}/workdir
    • Local file mount specified in the task.yaml file_mount -> {tmp-bucket-B}{i}
    • Temporary files tmp_files -> {tmp-bucket-C}/tmp-files
  • Now:
    • workdir specified in the task.yaml workdir -> {intermediate-bucket}/job-{run_id}/workdir
    • Local file mount specified in the task.yaml file_mount -> {intermediate-bucket}/job-{run_id}/local-file-mounts/{i}
    • Temporary files tmp_files -> {intermediate-bucket}/job-{run_id}/tmp-files
• Clean up
  • After the managed job finishes, when cleanup is triggered
    • If user doesn't specific bucket config, sky generated {intermediate-bucket}/job-{run_id} bucket will be deleted automatically as before.
    • If user specific bucket name in config, the directory created above will be deleted, the bucket still exists after that, and the user needs to manually delete the bucket as usual.(Due to potential race problem)
  • Currently aws, gcs, r2 use the command line support to remove the directory, while cos and https(azure) use the SDK to list all files under directory and then delete

Even using the same bucket, each job creates its own subdirectory based on the '{run-id}', and subdirectories of auto-created files will be cleaned up after the job is finished.

This allows users who don't have permission to create a bucket to manually specify a bucket name under ~/.sky/config.yaml.

Test plan

smoke test

pytest -s tests/test_smoke.py::test_managed_jobs_storage
pytest -s tests/test_smoke.py::TestStorageWithCredentials::test_bucket_sub_path --aws

custom test

(sky) ➜  cat ~/.sky/config.yaml
jobs:
  bucket:
    s3: "bucket-jobs-s3"
    gcs: "bucket-jobs-gcs"
    default: "bucket-jobs-default"

(sky) ➜ cat ~/Desktop/hello-sky/work_dir_1.yaml
name: test_workdir_bucket_name_2

workdir: .

resources:
  cloud: aws
  instance_type: t3.small

file_mounts:
  # this will use the user config
  /checkpoint:
    name: zpoint-filemounts-bucket
    source: ~/Desktop/dir1
    mode: MOUNT
    store: azure

  # these will all use same bucket configured in ~/.sky/config.yaml jobs->bucket now for bucket storage
  /dir1: ~/Desktop/dir1
  /dir2: ~/Desktop/dir2
  /dir3/dir3.py: ~/Desktop/dir1/dir1.py

run: |
  for i in {1..5}; do
    echo "Hello, SkyPilot World! $(date)"
    sleep 2
  done%   

Launch

(sky) ➜ sky jobs launch ~/Desktop/hello-sky/work_dir_1.yaml
⠋ Syncing ~/Desktop/dir1 -> https://xxx/zpoint-filemounts-bucket/
⠹ Syncing . -> s3://zpoint-bucket-s3/
⠧ Syncing ~/Desktop/dir1 -> s3://zpoint-bucket-s3/
⠦ Syncing ~/Desktop/dir2 -> s3://zpoint-bucket-s3/
⠴ Syncing /var/folders/83/zxqx914s57x310rfnhq8kk9r0000gn/T/skypilot-filemounts-files-aca97801 -> s3://zpoint-bucket-s3/

Looks good

Tested (run the relevant ones):

• [x] Code formatting: bash format.sh
• [x] Any manual or new tests for this PR (please specify below)
• [ ] All smoke tests: pytest tests/test_smoke.py
• [ ] Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• [ ] Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[romilbhardwaj]

Thanks @zpoint! I'm trying to run this yaml:

resources:
  cloud: aws

file_mounts:
  ~/aws: ~/aws

workdir: ~/tmp-workdir

num_nodes: 1

run: |
  echo "Hello, world!"
  ls ~/aws
  ls .

The task output should show me contents of my workdir and file_mount. Instead, I get:

├── Waiting for task resources on 1 node.
└── Job started. Streaming logs... (Ctrl-C to exit log streaming; job will not be killed)
(sky-3ba1-romilb, pid=2100) Hello, world!
(sky-3ba1-romilb, pid=2100) job-89ab894f
(sky-3ba1-romilb, pid=2100) job-89ab894f
✓ Managed job finished: 1 (status: SUCCEEDED).

Related to this: https://github.com/skypilot-org/skypilot/pull/4257#discussion_r1837141465?

[romilbhardwaj]

I think https://github.com/skypilot-org/skypilot/pull/4257#issuecomment-2469050952 is still not resolved.

I ran sky jobs launch with this YAML:

resources:
  cloud: aws

file_mounts:
  ~/aws: ~/aws

workdir: ~/tmp-workdir

num_nodes: 1

run: |
  echo "Hello, world!"
  ls ~/aws
  ls .

And I'm still getting

├── Waiting for task resources on 1 node.
└── Job started. Streaming logs... (Ctrl-C to exit log streaming; job will not be killed)
(sky-e0d7-romilb, pid=2076) Hello, world!
(sky-e0d7-romilb, pid=2076) job-24ad1167
(sky-e0d7-romilb, pid=2076) job-24ad1167

Instead of the actual contents of my workdir and mounted dir.

[zpoint]

@romilbhardwaj Thanks for the reply

4257 is now resolved, and I also updated the description of the PR to describe the mainly changes