Open Conless opened 1 week ago
Hi @cblmemo ! Would you like to have a look at this?
Can we also update the PR description?
Also, it will be great if we can test on some real world usage (e.g. deploy an LLM service and expose metric port in vLLM or dashboard in ray
Current implementation of SkyServe only allows the replicas to expose one port. In some cases, services may need to expose multiple ports for custom controller, GUI interface, etc. This PR adds support for it by allowing multiple ports, but use the first one as the main port. For example, when we start a service with resource requirements:
The output of
sky serve status
will look like:while the other ports (8081, 10000) are still accessible.
Tested (run the relevant ones):
bash format.sh
[x] Any manual or new tests for this PR (please specify below)
sky serve up
,sky serve status
and accessing the service with config:resources: ports:
run: | python -m http.server 8080 & python -m http.server 10000 & python -m http.server 10001 &
pytest tests/test_smoke.py
pytest tests/test_smoke.py::test_fill_in_the_name
conda deactivate; bash -i tests/backward_compatibility_tests.sh