[k8s] Improve multi-node provisioning time (nimbus)

[romilbhardwaj]

This PR introduces a bunch of optimizations for large scale k8s provisioning, including:

• Move setup to pod args (from #4389)
• Move to uv from pip (from #4394)
• Parallelizing pod creation, labelling
• Skip waiting for SSH
• Fix pod name collision at scale
• Used batch calls instead of per-pod read requests to API server
• Docs on how to build optimized images

Delivers > 4x speedup when provisioning 100s of nodes.

SKYPILOT_TIMELINE_FILE_PATH='timeline_100.prof' sky launch -y -c test --cloud kubernetes --num-nodes 100 --cpus 1 -- echo hi

• On master: 4m 10s
• This branch: 56s

Similar times with a NeMo derived image optimized with instructions in this PR.

Tested (run the relevant ones):

• [ ] Code formatting: bash format.sh
• [ ] Any manual or new tests for this PR (please specify below)
• [ ] All smoke tests: pytest tests/test_smoke.py
• [ ] Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• [ ] Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[romilbhardwaj]

Starting to run some final tests. @cg505 @Michaelvll if you find some time please do a quick round of reviews. Thanks!

[romilbhardwaj]

Should add uv to our base images. Takes ~2s to install it otherwise.

[cg505]

Should add uv to our base images. Takes ~2s to install it otherwise.

working on this now

[romilbhardwaj]

Running smoke tests:

• [x] AWS smoke tests - some failures, unrelated to this PR
• [x] Kubernetes smoke tests - some failures, unrelated to this PR

[cg505]

Ran backwards compatibility tests, no issues.

[romilbhardwaj]

Smoke tests for aws and k8s pass (barring a few unrelated failures). Merging now. Thanks for the great work @cg505 and @Michaelvll!