skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.82k stars 513 forks source link

[WIP][Serve] Enable launching multiple external LB on controller. #4362

Open cblmemo opened 1 week ago

cblmemo commented 1 week ago
$ sky serve status
Clusters
NAME                           LAUNCHED    RESOURCES                                                    STATUS  AUTOSTOP  COMMAND                       
ap-northeast-1-client          14 hrs ago  1x AWS(m6i.large)                                            UP      -         sky launch --region ap-no...  
sky-serve-controller-4a0782e9  1 min ago   1x GCP(n2-standard-4, disk_size=200, ports=['20001-20020'])  UP      10m       sky serve up examples/ser...  

Managed jobs
No in-progress managed jobs. (See: sky jobs -h)

Services
NAME      VERSION  UPTIME  STATUS      REPLICAS  ENDPOINT                      
test      1        9m 3s   READY       2/2       test.aws.cblmemo.net:8000       
test-dns  1        3m 52s  READY       2/2       test-dns.aws.cblmemo.net:8000   
service   -        -       NO_REPLICA  0/2       service.aws.cblmemo.net:8000  

Service Replicas
SERVICE_NAME  ID  VERSION  ENDPOINT                    LAUNCHED        RESOURCES       STATUS        REGION     
test          1   1        http://54.211.41.3:8080     9 mins ago      1x AWS(vCPU=2)  READY         us-east-1  
test          2   1        http://3.84.9.202:8080      9 mins ago      1x AWS(vCPU=2)  READY         us-east-1  
test-dns      1   1        http://18.234.191.41:8080   4 mins ago      1x AWS(vCPU=2)  READY         us-east-1  
test-dns      2   1        http://54.234.129.50:8080   4 mins ago      1x AWS(vCPU=2)  READY         us-east-1  
service       1   1        http://3.84.141.45:8080     a few secs ago  1x AWS(vCPU=2)  PROVISIONING  us-east-1  
service       2   1        http://44.201.191.235:8080  16 secs ago     1x AWS(vCPU=2)  STARTING      us-east-1  

External Load Balancers
SERVICE_NAME  ID  ENDPOINT            
test          0   3.137.187.47:8000   
test          1   54.238.10.138:8000  
test-dns      0   18.188.68.130:8000  
test-dns      1   13.230.9.98:8000    
service       0   18.227.81.22:8000   
service       1   13.115.58.105:8000  

* To see detailed service status: sky serve status -a
* 1 cluster has auto{stop,down} scheduled. Refresh statuses with: sky status --refresh

Tested (run the relevant ones):