[k8s] Move setup and ray start to pod args to make them async - Githubissues

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

https://skypilot.readthedocs.io

Apache License 2.0

6.82k stars 513 forks source link

[k8s] Move setup and ray start to pod args to make them async #4389

Closed Michaelvll closed 3 hours ago

Michaelvll commented 1 day ago

Moved setup/ray start to the kubernetes pod args to make them async

TODO:

[x] Better observability: the current logs are in kubectl logs instead of the provision.log as before which makes it harder to debug for users. (We now have a best effort for printing out the failed logs if we detect it during setup).

Tested (run the relevant ones):

[ ] Code formatting: bash format.sh
[ ] Any manual or new tests for this PR (please specify below)
[ ] All smoke tests: pytest tests/test_smoke.py
[ ] Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
[ ] Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

Michaelvll commented 3 hours ago

Tested:

[x] sky launch --num-nodes 200
[x] sky launch --num-nodes 200 --image-id docker:ubuntu:24.04