[SkyServe] Custom Setup Command on Controller

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

https://skypilot.readthedocs.io

Apache License 2.0

6.76k stars 503 forks source link

[SkyServe] Custom Setup Command on Controller #2994

Open cblmemo opened 9 months ago

cblmemo commented 9 months ago

We should have a customized setup command in case the user wants to run some command in the controller, e.g. a sidecar proxy. To avoid conflict in different services' setup commands, we could consider using a separate controller cluster for each service w/ a custom setup command.

Several usecases from users:

Spin up a proxy server to replace our load balancer;
Spin up a cloudflare/ngrok daemon for tunneling, on top of our load balancer.

Michaelvll commented 9 months ago

Can we have a specific use case for this, for example, why we would like to have the proxy on the controller? Also, it would be good to clearify whether it is need to be along with autoscaler or the load-balancer.

cblmemo commented 9 months ago

Can we have a specific use case for this, for example, why we would like to have the proxy on the controller? Also, it would be good to clearify whether it is need to be along with autoscaler or the load-balancer.

Good point! Just added.

github-actions[bot] commented 5 months ago

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

alecswan commented 3 months ago

Our use case requires us to deploy the controller in an environment without internet access. So, ~/.config/pip.conf needs to be configured to point to a private python repo before the controller and its python dependencies can be installed on the instance. These custom actions need to be run right after the host, e.g. ec2 instance, is provisioned, but before controller artifacts are installed. Is this possible today?