search

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.61k stars 479 forks source link

[SkyServe] Open extra ports on service replica #2859

Open cblmemo opened 9 months ago

cblmemo commented 9 months ago

One user required to see the ray dashboard on each service replica.

concretevitamin commented 6 months ago

Another user bumped this. On PyTriton, different ports are used for HTTP, gRPC, and metrics: https://triton-inference-server.github.io/pytriton/0.5.2/guides/deploying_in_clusters/#exposing-ports

To discuss: Can we allow a syntax where each replica can open multiple ports, and we assume the first primary port to route traffic is the "first" port?