Closed hongbo-miao closed 1 year ago
Michaelvll
commented
1 year ago Hi @Hongbo-Miao, thanks for the question! According to the log, it seems the SkyPilot is trying to install ray[default]==1.10.0 on the remote VM, which indicates that you may be using a very old version of SkyPilot. Could you please let us know, what SkyPilot version are you currently using?
hongbo-miao
commented
1 year ago I see, I met an issue installing it through Poetry on macOS (M1 chip). So I am using Docker way based on https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#quick-alternative-trying-in-docker
docker run \
--interactive \
--tty \
--rm \
--volume="$(pwd):/workspace" \
--volume="$HOME/.sky:/root/.sky:rw" \
--volume="$HOME/.aws:/root/.aws:rw" \
--volume="$HOME/.config/gcloud:/root/.config/gcloud:rw" \
--volume="$HOME/.azure:/root/.azure:rw" \
public.ecr.aws/a9w6z7w5/sky:latest \
/bin/bashSo I guess this public.ecr.aws/a9w6z7w5/sky:latest is out-of-date? Where can we find latest image? Thanks!
hongbo-miao
commented
1 year ago It would be also great to publish the Docker image at https://github.com/orgs/skypilot-org/packages?repo_name=skypilot or Docker Hub 😃
concretevitamin
commented
1 year ago Hey @Hongbo-Miao: yes, the image shown in https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#quick-alternative-trying-in-docker is very old and out of date. We'll see how to handle that part of the docs.
Would love to have CI/CD publish images to the repo's packages. Let us know if you're interested in contributing!
romilbhardwaj
commented
1 year ago @Hongbo-Miao - can you try berkeleyskypilot/skypilot:latest? This runs 0.3.2 with GCP and AWS dependencies
docker run -td --name sky --rm -v "$HOME/.sky:/root/.sky:rw" -v "$HOME/.aws:/root/.aws:rw" -v "$HOME/.config/gcloud:/root/.config/gcloud:rw" berkeleyskypilot/skypilot:latestThanks @Michaelvll @concretevitamin @romilbhardwaj !
The new berkeleyskypilot/skypilot:latest works well now! 😃
I am trying SkyPilot with a very simple YAML file:
But I failed to create this cluster when running
Click to expand to see full log!
```shell Task from YAML spec: skypilot.yaml Launching a new cluster 'hm-classify-fashion-mnist-cluster'. Proceed? [Y/n]: Running task on cluster hm-classify-fashion-mnist-cluster... I 07-07 06:09:46 optimizer.py:608] == Optimizer == I 07-07 06:09:46 optimizer.py:631] Estimated cost: $0.5 / hour I 07-07 06:09:46 optimizer.py:631] I 07-07 06:09:46 optimizer.py:677] Considered resources (1 node): I 07-07 06:09:46 optimizer.py:706] ---------------------------------------------------------- I 07-07 06:09:46 optimizer.py:706] CLOUD INSTANCE ACCELERATORS COST ($) CHOSEN I 07-07 06:09:46 optimizer.py:706] ---------------------------------------------------------- I 07-07 06:09:46 optimizer.py:706] AWS g4dn.xlarge T4:1 0.53 ✔ I 07-07 06:09:46 optimizer.py:706] ---------------------------------------------------------- I 07-07 06:09:46 optimizer.py:706] I 07-07 06:09:46 optimizer.py:721] Multiple AWS instances satisfy T4:1. The cheapest AWS(g4dn.xlarge, {'T4': 1}) is considered among: I 07-07 06:09:46 optimizer.py:721] ['g4dn.xlarge', 'g4dn.2xlarge', 'g4dn.4xlarge', 'g4dn.8xlarge', 'g4dn.16xlarge']. I 07-07 06:09:46 optimizer.py:721] I 07-07 06:09:46 optimizer.py:727] To list more details, run 'sky show-gpus T4'. I 07-07 06:09:46 cloud_vm_ray_backend.py:1246] Creating a new cluster: "hm-classify-fashion-mnist-cluster" [1x AWS(g4dn.xlarge, {'T4': 1})]. I 07-07 06:09:46 cloud_vm_ray_backend.py:1246] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters. I 07-07 06:09:46 cloud_vm_ray_backend.py:775] To view detailed progress: tail -n100 -f /root/sky_logs/sky-2023-07-07-06-09-46-262145/provision.log I 07-07 06:09:47 cloud_vm_ray_backend.py:955] Launching on AWS us-west-2 (us-west-2a,us-west-2b,us-west-2c,us-west-2d) I 07-07 06:11:26 log_utils.py:45] Head node is up. I 07-07 06:13:04 cloud_vm_ray_backend.py:505] ====== stdout ====== 2023-07-07 06:09:47,811 INFO commands.py:270 -- Cluster: hm-classify-fashion-mnist-cluster 2023-07-07 06:09:47,815 VWARN commands.py:316 -- Loaded cached provider configuration from /tmp/ray-config-76c95606fbc59f8ed058665050df9d119911fdf5 2023-07-07 06:09:47,815 WARN commands.py:322 -- If you experience issues with the cloud provider, try re-running the command with --no-config-cache. 2023-07-07 06:09:48,322 VINFO utils.py:145 -- Creating AWS resource `ec2` in `us-west-2` 2023-07-07 06:09:48,454 VINFO utils.py:145 -- Creating AWS resource `ec2` in `us-west-2` 2023-07-07 06:09:48,844 INFO commands.py:648 -- No head node found. Launching a new cluster. Confirm [y/N]: y [automatic, due to --yes] 2023-07-07 06:09:48,844 INFO commands.py:699 -- Acquiring an up-to-date head node 2023-07-07 06:09:50,694 INFO node_provider.py:428 -- Launched 1 nodes [subnet_id=subnet-0c4861e4debc15363] 2023-07-07 06:09:50,694 INFO node_provider.py:442 -- Launched instance i-0d0acadef7ec1481d [state=pending, info=pending] 2023-07-07 06:09:50,694 INFO commands.py:715 -- Launched a new head node 2023-07-07 06:09:50,694 INFO commands.py:719 -- Fetching the new head node 2023-07-07 06:09:50,888 INFO commands.py:734 -- <1/1> Setting up head node 2023-07-07 06:09:50,900 INFO commands.py:755 -- Prepared bootstrap config 2023-07-07 06:09:52,099 INFO updater.py:323 -- New status: waiting-for-ssh 2023-07-07 06:09:52,100 INFO updater.py:260 -- [1/7] Waiting for SSH to become available 2023-07-07 06:09:52,100 INFO updater.py:265 -- Running `uptime` as a test. 2023-07-07 06:09:52,279 INFO command_runner.py:394 -- Fetched IP: 34.222.43.144 2023-07-07 06:09:52,280 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:09:52,280 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:09:57,344 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:10:02,357 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:10:02,363 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:10:07,420 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:10:12,431 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:10:12,431 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:10:17,509 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:10:22,517 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:10:22,517 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:10:27,585 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:10:32,589 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:10:32,590 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:10:37,719 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:10:42,729 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:10:42,729 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:10:47,824 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:10:52,834 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:10:52,835 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:10:57,953 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:11:02,965 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:11:02,966 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:11:08,048 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:11:13,052 VINFO command_runner.py:552 -- Running `uptime` 06:11:26 up 1 min, 1 user, load average: 2.07, 0.56, 0.19 sending incremental file list hm-classify-fashion-mnist-cluster.yml sent 1,395 bytes received 35 bytes 953.33 bytes/sec total size is 2,574 speedup is 1.80 sending incremental file list ./ sky-0.1.dev0-py3-none-any.whl sent 464,044 bytes received 38 bytes 309,388.00 bytes/sec total size is 495,506 speedup is 1.07 sending incremental file list ./ .DS_Store config credentials sso/ sso/cache/ sso/cache/1319eff725ffd225b2b10d5c19301f50f9782a8b.json sso/cache/e6ba2466873cb4faa1c6c46b9218f9c13f4a0024.json sts_cache/ sts_cache/sts.9sr18trf sent 4,541 bytes received 149 bytes 3,126.67 bytes/sec total size is 12,648 speedup is 2.70 sending incremental file list ray-bootstrap-02qbacz0 sent 1,559 bytes received 35 bytes 1,062.67 bytes/sec total size is 4,618 speedup is 2.90 2023-07-07 06:11:13,053 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:11:13,301 INFO updater.py:311 -- SSH still not available (SSH command failed.), retrying in 5 seconds. 2023-07-07 06:11:18,309 VINFO command_runner.py:552 -- Running `uptime` 2023-07-07 06:11:18,310 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=5s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'` 2023-07-07 06:11:26,511 SUCC updater.py:279 -- Success. 2023-07-07 06:11:26,513 INFO updater.py:373 -- Updating cluster configuration. [hash=05010f82a792e1ec13911236714973ad3a7bed32] 2023-07-07 06:11:28,745 INFO updater.py:380 -- New status: syncing-files 2023-07-07 06:11:28,746 INFO updater.py:237 -- [2/7] Processing file mounts 2023-07-07 06:11:28,754 VINFO command_runner.py:552 -- Running `mkdir -p ~/.sky` 2023-07-07 06:11:28,754 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p ~/.sky)'` 2023-07-07 06:11:29,395 VINFO command_runner.py:595 -- Running `rsync --rsh ssh -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s -avz /root/.sky/generated/hm-classify-fashion-mnist-cluster.yml ubuntu@34.222.43.144:~/.sky/sky_ray.yml` 2023-07-07 06:11:30,460 VINFO updater.py:528 -- `rsync`ed /root/.sky/generated/hm-classify-fashion-mnist-cluster.yml (local) to ~/.sky/sky_ray.yml (remote) 2023-07-07 06:11:30,460 INFO updater.py:232 -- ~/.sky/sky_ray.yml from /root/.sky/generated/hm-classify-fashion-mnist-cluster.yml 2023-07-07 06:11:30,461 VINFO command_runner.py:552 -- Running `mkdir -p ~/.sky/sky_wheels` 2023-07-07 06:11:30,461 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p ~/.sky/sky_wheels)'` 2023-07-07 06:11:30,885 VINFO command_runner.py:595 -- Running `rsync --rsh ssh -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s -avz /tmp/8c17a11b31b1ef9226f51b48ed124aac/ ubuntu@34.222.43.144:~/.sky/sky_wheels/` 2023-07-07 06:11:31,969 VINFO updater.py:528 -- `rsync`ed /tmp/8c17a11b31b1ef9226f51b48ed124aac/ (local) to ~/.sky/sky_wheels/ (remote) 2023-07-07 06:11:31,970 INFO updater.py:232 -- ~/.sky/sky_wheels/ from /tmp/8c17a11b31b1ef9226f51b48ed124aac/ 2023-07-07 06:11:31,972 VINFO command_runner.py:552 -- Running `mkdir -p ~/.aws` 2023-07-07 06:11:31,973 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p ~/.aws)'` 2023-07-07 06:11:32,246 VINFO command_runner.py:595 -- Running `rsync --rsh ssh -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s -avz /root/.aws/ ubuntu@34.222.43.144:~/.aws/` 2023-07-07 06:11:33,421 VINFO updater.py:528 -- `rsync`ed /root/.aws/ (local) to ~/.aws/ (remote) 2023-07-07 06:11:33,421 INFO updater.py:232 -- ~/.aws/ from /root/.aws/ 2023-07-07 06:11:33,422 VINFO command_runner.py:552 -- Running `mkdir -p ~` 2023-07-07 06:11:33,422 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p ~)'` 2023-07-07 06:11:33,848 VINFO command_runner.py:595 -- Running `rsync --rsh ssh -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s -avz /tmp/ray-bootstrap-02qbacz0 ubuntu@34.222.43.144:~/ray_bootstrap_config.yaml` 2023-07-07 06:11:34,611 VINFO updater.py:528 -- `rsync`ed /tmp/ray-bootstrap-02qbacz0 (local) to ~/ray_bootstrap_config.yaml (remote) 2023-07-07 06:11:34,612 INFO updater.py:232 -- ~/ray_bootstrap_config.yaml from /tmp/ray-bootstrap-02qbacz0 2023-07-07 06:11:34,613 VINFO command_runner.py:552 -- Running `mkdir -p ~` 2023-07-07 06:11:34,613 VVINFO command_runner.py:554 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_63a9f0ea7b/8187ae1600/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@34.222.43.144 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p ~)'` sending incremental file list sky-key sent 1,383 bytes received 35 bytes 945.33 bytes/sec total size is 1,675 speedup is 1.18 Defaulting to user installation because normal site-packages is not writeable Collecting ray[default]==1.10.0 Downloading ray-1.10.0-cp39-cp39-manylinux2014_x86_64.whl (59.3 MB) |████████████████████████████████| 59.3 MB 2.6 kB/s Collecting filelock Downloading filelock-3.12.2-py3-none-any.whl (10 kB) Collecting numpy>=1.19.3 Downloading numpy-1.25.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.7 MB) |████████████████████████████████| 17.7 MB 106.1 MB/s Collecting protobuf>=3.15.3 Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl (304 kB) |████████████████████████████████| 304 kB 101.7 MB/s Collecting grpcio>=1.28.1 Downloading grpcio-1.56.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.2 MB) |████████████████████████████████| 5.2 MB 66.3 MB/s Collecting jsonschema Downloading jsonschema-4.18.0-py3-none-any.whl (81 kB) |████████████████████████████████| 81 kB 12.6 MB/s Collecting pyyaml Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB) |████████████████████████████████| 661 kB 79.8 MB/s Collecting attrs Downloading attrs-23.1.0-py3-none-any.whl (61 kB) |████████████████████████████████| 61 kB 15.9 MB/s Collecting msgpack<2.0.0,>=1.0.0 Downloading msgpack-1.0.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB) |████████████████████████████████| 322 kB 97.5 MB/s Collecting click>=7.0 Downloading click-8.1.4-py3-none-any.whl (98 kB) |████████████████████████████████| 98 kB 13.7 MB/s Collecting redis>=3.5.0 Downloading redis-4.6.0-py3-none-any.whl (241 kB) |████████████████████████████████| 241 kB 95.0 MB/s Collecting aiosignal Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB) Collecting frozenlist Downloading frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB) |████████████████████████████████| 158 kB 113.3 MB/s Collecting aiohttp-cors Downloading aiohttp_cors-0.7.0-py3-none-any.whl (27 kB) Collecting smart-open Downloading smart_open-6.3.0-py3-none-any.whl (56 kB) |████████████████████████████████| 56 kB 626 kB/s Collecting gpustat>=1.0.0b1 Downloading gpustat-1.1.tar.gz (97 kB) |████████████████████████████████| 97 kB 14.9 MB/s Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done Requirement already satisfied: requests in /opt/conda/lib/python3.9/site-packages (from ray[default]==1.10.0) (2.25.1) Collecting aiohttp>=3.7 Downloading aiohttp-3.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB) |████████████████████████████████| 1.0 MB 71.5 MB/s Collecting aioredis<2 Downloading aioredis-1.3.1-py3-none-any.whl (65 kB) |████████████████████████████████| 65 kB 459 kB/s Collecting colorful Downloading colorful-0.5.5-py2.py3-none-any.whl (201 kB) |████████████████████████████████| 201 kB 89.5 MB/s Collecting opencensus Downloading opencensus-0.11.2-py2.py3-none-any.whl (128 kB) |████████████████████████████████| 128 kB 83.7 MB/s Collecting prometheus-client>=0.7.1 Downloading prometheus_client-0.17.0-py3-none-any.whl (60 kB) |████████████████████████████████| 60 kB 389 kB/s Collecting py-spy>=0.2.0 Downloading py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB) |████████████████████████████████| 3.0 MB 101.4 MB/s Collecting charset-normalizer<4.0,>=2.0 Downloading charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB) |████████████████████████████████| 199 kB 72.4 MB/s Collecting multidict<7.0,>=4.5 Downloading multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB) |████████████████████████████████| 114 kB 85.5 MB/s Collecting yarl<2.0,>=1.0 Downloading yarl-1.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (269 kB) |████████████████████████████████| 269 kB 101.2 MB/s Collecting async-timeout<5.0,>=4.0.0a3 Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting hiredis Downloading hiredis-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (165 kB) |████████████████████████████████| 165 kB 95.6 MB/s Collecting blessed>=1.17.1 Downloading blessed-1.20.0-py2.py3-none-any.whl (58 kB) |████████████████████████████████| 58 kB 11.4 MB/s Collecting psutil>=5.6.0 Downloading psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (282 kB) |████████████████████████████████| 282 kB 96.3 MB/s Collecting nvidia-ml-py>=11.450.129 Downloading nvidia_ml_py-12.535.77-py3-none-any.whl (36 kB) Collecting wcwidth>=0.1.4 Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB) Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.9/site-packages (from blessed>=1.17.1->gpustat>=1.0.0b1->ray[default]==1.10.0) (1.16.0) Requirement already satisfied: idna>=2.0 in /opt/conda/lib/python3.9/site-packages (from yarl<2.0,>=1.0->aiohttp>=3.7->ray[default]==1.10.0) (2.10) Collecting referencing>=0.28.4 Downloading referencing-0.29.1-py3-none-any.whl (25 kB) Collecting jsonschema-specifications>=2023.03.6 Downloading jsonschema_specifications-2023.6.1-py3-none-any.whl (17 kB) Collecting rpds-py>=0.7.1 Downloading rpds_py-0.8.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) |████████████████████████████████| 1.2 MB 93.6 MB/s Collecting opencensus-context>=0.1.3 Downloading opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB) Collecting google-api-core<3.0.0,>=1.0.0 Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB) |████████████████████████████████| 120 kB 118.9 MB/s Collecting google-auth<3.0.dev0,>=2.14.1 Downloading google_auth-2.21.0-py2.py3-none-any.whl (182 kB) |████████████████████████████████| 182 kB 89.4 MB/s Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 Downloading googleapis_common_protos-1.59.1-py2.py3-none-any.whl (224 kB) |████████████████████████████████| 224 kB 70.2 MB/s Collecting rsa<5,>=3.1.4 Downloading rsa-4.9-py3-none-any.whl (34 kB) Collecting pyasn1-modules>=0.2.1 Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB) |████████████████████████████████| 181 kB 87.4 MB/s Collecting cachetools<6.0,>=2.0.0 Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB) Requirement already satisfied: urllib3<2.0 in /opt/conda/lib/python3.9/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.10.0) (1.26.4) Collecting pyasn1<0.6.0,>=0.4.6 Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB) |████████████████████████████████| 83 kB 4.3 MB/s Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.9/site-packages (from requests->ray[default]==1.10.0) (4.0.0) Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.9/site-packages (from requests->ray[default]==1.10.0) (2020.12.5) Building wheels for collected packages: gpustat Building wheel for gpustat (PEP 517) ... done Created wheel for gpustat: filename=gpustat-1.1-py3-none-any.whl size=26280 sha256=b278a57c6602e33d65647686f601eeadf18e20528ac49271e5ebe5d338a292f2 Stored in directory: /home/ubuntu/.cache/pip/wheels/91/f0/b3/8566d6821307110981a5db015cbf8fd88697446f81e5f40a27 Successfully built gpustat Installing collected packages: rpds-py, pyasn1, attrs, rsa, referencing, pyasn1-modules, protobuf, multidict, frozenlist, cachetools, yarl, wcwidth, jsonschema-specifications, googleapis-common-protos, google-auth, charset-normalizer, async-timeout, aiosignal, redis, pyyaml, psutil, opencensus-context, nvidia-ml-py, numpy, msgpack, jsonschema, hiredis, grpcio, google-api-core, filelock, click, blessed, aiohttp, smart-open, ray, py-spy, prometheus-client, opencensus, gpustat, colorful, aioredis, aiohttp-cors WARNING: The scripts pyrsa-decrypt, pyrsa-encrypt, pyrsa-keygen, pyrsa-priv2pub, pyrsa-sign and pyrsa-verify are installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script normalizer is installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts f2py, f2py3 and f2py3.9 are installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script jsonschema is installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts ray, ray-operator, rllib, serve and tune are installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script gpustat is installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed aiohttp-3.8.4 aiohttp-cors-0.7.0 aioredis-1.3.1 aiosignal-1.3.1 async-timeout-4.0.2 attrs-23.1.0 blessed-1.20.0 cachetools-5.3.1 charset-normalizer-3.1.0 click-8.1.4 colorful-0.5.5 filelock-3.12.2 frozenlist-1.3.3 google-api-core-2.11.1 google-auth-2.21.0 googleapis-common-protos-1.59.1 gpustat-1.1 grpcio-1.56.0 hiredis-2.2.3 jsonschema-4.18.0 jsonschema-specifications-2023.6.1 msgpack-1.0.5 multidict-6.0.4 numpy-1.25.0 nvidia-ml-py-12.535.77 opencensus-0.11.2 opencensus-context-0.1.3 prometheus-client-0.17.0 protobuf-4.23.4 psutil-5.9.5 py-spy-0.3.14 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyyaml-6.0 ray-1.10.0 redis-4.6.0 referencing-0.29.1 rpds-py-0.8.8 rsa-4.9 smart-open-6.3.0 wcwidth-0.2.6 yarl-1.9.2 Usage: kill [options]I saw this error in the log
but which seems having been fixed by https://github.com/skypilot-org/skypilot/pull/885
Any ideas? Thanks!