Implement Automatic Bucket Creation and Data Transfer in `with_data` API

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

https://skypilot.readthedocs.io

Apache License 2.0

6.82k stars 513 forks source link

Implement Automatic Bucket Creation and Data Transfer in `with_data` API #4255

Open andylizf opened 2 weeks ago

andylizf commented 2 weeks ago

To implement the with_data API in #4254:

Automatic Bucket Creation
Add logic to create a storage bucket automatically when with_data is called. The API should link the upstream task’s output path to this bucket, streamlining data storage setup.
Data Transfer and Downstream Access
Enable automatic transfer of the upstream task’s output to the bucket, and configure the downstream task to retrieve data from it directly, ensuring seamless data flow without manual setup.

cblmemo commented 2 weeks ago

Assining to @euclidgame . Also we should implement the optimizer logic we've discussed in our meeting.