search

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.85k stars 518 forks source link

[SkyServe] Open extra ports on service replica #2859

Open cblmemo opened 11 months ago

cblmemo commented 11 months ago

One user required to see the ray dashboard on each service replica.

concretevitamin commented 8 months ago

Another user bumped this. On PyTriton, different ports are used for HTTP, gRPC, and metrics: https://triton-inference-server.github.io/pytriton/0.5.2/guides/deploying_in_clusters/#exposing-ports

To discuss: Can we allow a syntax where each replica can open multiple ports, and we assume the first primary port to route traffic is the "first" port?

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

ashutoshsaboo commented 1 month ago

Not stale, any plans for this? @concretevitamin

concretevitamin commented 1 month ago

Not stale, any plans for this? @concretevitamin

We should get to it. What ports and for what purposes do you need on each replica? How do you work around it?