romilbhardwaj
commented
1 month ago Hi @shengkaixuan - what commit and sky version are you on? sky -v and sky -c. Can you try on the latest master branch?
Any details on the Kubernetes cluster (e.g., GKE/EKS/K3s/Kind...) to help me reproduce would be helpful.
To debug:
- Is the pod running?
kubectl get pods - Does
kubectl port-forward pod/<pod_name> :22work? - Manually try running the ssh command:
ssh -T -i '~/.ssh/sky-key' sky@10.42.0.233 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o 'ProxyCommand=ssh -tt -i /root/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -W %h:%p sky@127.0.0.1 -o ProxyCommand='"'"'/root/.sky/kubernetes-port-forward-proxy-command.sh sheng-4756-head'"'"' ' uptime
shengkaixuan
the first time i use skypilot it went well . use k8s as the cloud, but now it successfully created the pod and stuck in the "⠦ Launching - Waiting for SSH access" state.
Task from YAML spec: test.yaml I 07-15 09:10:17 optimizer.py:695] == Optimizer == I 07-15 09:10:17 optimizer.py:718] Estimated cost: $0.0 / hour I 07-15 09:10:17 optimizer.py:718] I 07-15 09:10:17 optimizer.py:843] Considered resources (1 node): I 07-15 09:10:17 optimizer.py:913] --------------------------------------------------------------------------------------------- I 07-15 09:10:17 optimizer.py:913] CLOUD INSTANCE vCPUs Mem(GB) ACCELERATORS REGION/ZONE COST ($) CHOSEN I 07-15 09:10:17 optimizer.py:913] --------------------------------------------------------------------------------------------- I 07-15 09:10:17 optimizer.py:913] Kubernetes 2CPU--2GB 2 2 - kubernetes 0.00 ✔ I 07-15 09:10:17 optimizer.py:913] --------------------------------------------------------------------------------------------- I 07-15 09:10:17 optimizer.py:913] Launching a new cluster 'sheng'. Proceed? [Y/n]: Y I 07-15 09:10:19 cloud_vm_ray_backend.py:4411] Creating a new cluster: 'sheng' [1x Kubernetes(2CPU--2GB)]. I 07-15 09:10:19 cloud_vm_ray_backend.py:4411] Tip: to reuse an existing cluster, specify --cluster (-c). Runsky statusto see existing clusters. I 07-15 09:10:19 cloud_vm_ray_backend.py:1406] To view detailed progress: tail -n100 -f /root/sky_logs/sky-2024-07-15-09-10-17-195307/provision.log I 07-15 09:10:21 provisioner.py:73] Launching on Kubernetes 'sheng'. ⠦ Launching - Waiting for SSH accessHere are the logs:
D 07-15 09:11:28 provisioner.py:409] Retrying in 1 second... D 07-15 09:11:29 provisioner.py:354] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' sky@10.42.0.233 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o 'ProxyCommand=ssh -tt -i /root/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -W %h:%p sky@127.0.0.1 -o ProxyCommand='"'"'/root/.sky/kubernetes-port-forward-proxy-command.sh sheng-4756-head'"'"' ' uptime D 07-15 09:11:39 provisioner.py:369] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' sky@10.42.0.233 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o 'ProxyCommand=ssh -tt -i /root/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -W %h:%p sky@127.0.0.1 -o ProxyCommand='"'"'/root/.sky/kubernetes-port-forward-proxy-command.sh sheng-4756-head'"'"' ' uptimeError: Command '['ssh', '-T', '-i', '~/.ssh/sky-key', 'sky@10.42.0.233', '-p', '22', '-o', 'StrictHostKeyChecking=no', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10s', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'IdentitiesOnly=yes', '-o', 'ExitOnForwardFailure=yes', '-o', 'ServerAliveInterval=5', '-o', 'ServerAliveCountMax=3', '-o', "ProxyCommand=ssh -tt -i /root/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -W %h:%p sky@127.0.0.1 -o ProxyCommand='/root/.sky/kubernetes-port-forward-proxy-command.sh sheng-4756-head' ", 'uptime']' timed out after 10 seconds D 07-15 09:11:39 provisioner.py:409] Retrying in 1 second... D 07-15 09:11:40 provisioner.py:354] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' sky@10.42.0.233 -p 22 -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o 'ProxyCommand=ssh -tt -i /root/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -W %h:%p sky@127.0.0.1 -o ProxyCommand='"'"'/root/.sky/kubernetes-port-forward-proxy-command.sh sheng-4756-head'"'"' ' uptimehow to solve the problem??