Closed romilbhardwaj closed 2 years ago
I am unable to use sky launch. Git credential forwarding fails in _get_ssh_key.
Seems related to #211, reverting it (to 790e8dce2fdd0996906cd13e3130fb9933e9de3e) seems to work fine.
790e8dce2fdd0996906cd13e3130fb9933e9de3e
I am using gh auth to setup my git credentials - could that be related?
gh auth
Run ./examples/run_smoke_tests.sh
./examples/run_smoke_tests.sh
(base) romilb@romilbx1yoga:/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype$ ./examples/run_smoke_tests.sh +++ dirname -- ./examples/run_smoke_tests.sh ++ cd -- ./examples ++ pwd + DIR=/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/examples + python /mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/examples/example_app.py I 01-21 11:38:51 resources.py:52] Missing tf_version in accelerator_args, using default (2.5.0) I 01-21 11:38:51 resources.py:56] Missing tpu_name in accelerator_args, using default (sky_tpu) I 01-21 11:38:51 optimizer.py:46] AWS -> GCP egress cost: $53.91 for 600.0 GB I 01-21 11:38:51 optimizer.py:46] GCP -> AWS egress cost: $0.012 for 0.1 GB I 01-21 11:38:51 optimizer.py:46] GCP -> AWS egress cost: $0.012 for 0.1 GB I 01-21 11:38:51 optimizer.py:308] Optimizer - plan minimizing cost (~$57.1): I 01-21 11:38:51 optimizer.py:321] I 01-21 11:38:51 optimizer.py:321] TASK BEST_RESOURCE I 01-21 11:38:51 optimizer.py:321] train_op AWS(p3.2xlarge) I 01-21 11:38:51 optimizer.py:321] infer_op AWS(inf1.2xlarge) I 01-21 11:38:51 optimizer.py:321] I 01-21 11:38:51 optimizer.py:276] Details: task -> {resources -> cost} I 01-21 11:38:51 optimizer.py:277] {train_op: {AWS(p3.2xlarge): 55.859051519999994, I 01-21 11:38:51 optimizer.py:277] AWS(p3.8xlarge): 113.27995151999997, I 01-21 11:38:51 optimizer.py:277] GCP(n1-standard-8, {'tpu-v3-8': 1}, accelerator_args={'tf_version': '2.5.0', 'tpu_name': 'sky_tpu'}): 97.61669841451885}, I 01-21 11:38:51 optimizer.py:277] infer_op: {AWS(p3.2xlarge): 67.16405152, I 01-21 11:38:51 optimizer.py:277] GCP(n1-standard-4, {'T4': 1}): 59.542121622564096, I 01-21 11:38:51 optimizer.py:277] AWS(inf1.2xlarge): 57.11285360333333, I 01-21 11:38:51 optimizer.py:277] GCP(n1-standard-8, {'T4': 1}): 60.83801223794871}} I 01-21 11:38:51 optimizer.py:277] + sky launch -c min /mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/examples/minimal.yaml Task from YAML spec: /mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/examples/minimal.yaml Running task on cluster min... Running task on cluster min ... I 01-21 11:38:55 cloud_vm_ray_backend.py:640] To view detailed progress: tail -n100 -f sky_logs/sky-2022-01-21-11-38-55-253876/provision.log I 01-21 11:38:55 cloud_vm_ray_backend.py:650] I 01-21 11:38:55 cloud_vm_ray_backend.py:650] Launching on AWS us-east-1 (us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f) WARNING: You are using pip version 20.3.3; however, version 21.3.1 is available. You should consider upgrading via the '/mnt/d/wsl/anaconda3/bin/python -m pip install --upgrade pip' command. Shared connection to 3.90.70.125 closed. Shared connection to 3.90.70.125 closed. Shared connection to 3.90.70.125 closed. Shared connection to 3.90.70.125 closed. Shared connection to 3.90.70.125 closed. Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com Requirement already satisfied: ray[default]==1.9.2 in ./.local/lib/python3.6/site-packages (1.9.2) Requirement already satisfied: protobuf>=3.15.3 in /usr/local/lib/python3.6/dist-packages (from ray[default]==1.9.2) (3.18.1) Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from ray[default]==1.9.2) (5.4.1) Requirement already satisfied: numpy>=1.16 in /usr/local/lib/python3.6/dist-packages (from ray[default]==1.9.2) (1.19.5) Requirement already satisfied: grpcio>=1.28.1 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (1.43.0) Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (1.0.3) Requirement already satisfied: dataclasses in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (0.8) Requirement already satisfied: jsonschema in /usr/local/lib/python3.6/dist-packages (from ray[default]==1.9.2) (3.2.0) Requirement already satisfied: filelock in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (3.4.1) Requirement already satisfied: redis>=3.5.0 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (4.1.1) Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from ray[default]==1.9.2) (21.2.0) Requirement already satisfied: click>=7.0 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (8.0.3) Requirement already satisfied: aiohttp-cors in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (0.7.0) Requirement already satisfied: aioredis<2 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (1.3.1) Requirement already satisfied: smart-open in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (5.2.1) Requirement already satisfied: frozenlist in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (1.2.0) Requirement already satisfied: prometheus-client>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from ray[default]==1.9.2) (0.11.0) Requirement already satisfied: opencensus in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (0.8.0) Requirement already satisfied: py-spy>=0.2.0 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (0.3.11) Requirement already satisfied: aiohttp>=3.7 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (3.8.1) Requirement already satisfied: gpustat>=1.0.0b1 in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (1.0.0b1) Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from ray[default]==1.9.2) (2.26.0) Requirement already satisfied: colorful in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (0.5.4) Requirement already satisfied: aiosignal in ./.local/lib/python3.6/site-packages (from ray[default]==1.9.2) (1.2.0) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.6/dist-packages (from aiohttp>=3.7->ray[default]==1.9.2) (2.0.6) Requirement already satisfied: asynctest==0.13.0 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]==1.9.2) (0.13.0) Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.6/dist-packages (from aiohttp>=3.7->ray[default]==1.9.2) (3.10.0.2) Requirement already satisfied: idna-ssl>=1.0 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]==1.9.2) (1.1.0) Requirement already satisfied: multidict<7.0,>=4.5 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]==1.9.2) (5.2.0) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]==1.9.2) (4.0.2) Requirement already satisfied: yarl<2.0,>=1.0 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]==1.9.2) (1.7.2) Requirement already satisfied: hiredis in ./.local/lib/python3.6/site-packages (from aioredis<2->ray[default]==1.9.2) (2.0.0) Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.6/dist-packages (from click>=7.0->ray[default]==1.9.2) (4.8.1) Requirement already satisfied: six>=1.7 in /usr/local/lib/python3.6/dist-packages (from gpustat>=1.0.0b1->ray[default]==1.9.2) (1.16.0) Requirement already satisfied: psutil in ./.local/lib/python3.6/site-packages (from gpustat>=1.0.0b1->ray[default]==1.9.2) (5.9.0) Requirement already satisfied: nvidia-ml-py3>=7.352.0 in ./.local/lib/python3.6/site-packages (from gpustat>=1.0.0b1->ray[default]==1.9.2) (7.352.0) Requirement already satisfied: blessed>=1.17.1 in ./.local/lib/python3.6/site-packages (from gpustat>=1.0.0b1->ray[default]==1.9.2) (1.19.1) Requirement already satisfied: wcwidth>=0.1.4 in /usr/local/lib/python3.6/dist-packages (from blessed>=1.17.1->gpustat>=1.0.0b1->ray[default]==1.9.2) (0.2.5) Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.6/dist-packages (from idna-ssl>=1.0->aiohttp>=3.7->ray[default]==1.9.2) (3.2) Requirement already satisfied: deprecated>=1.2.3 in ./.local/lib/python3.6/site-packages (from redis>=3.5.0->ray[default]==1.9.2) (1.2.13) Requirement already satisfied: packaging>=20.4 in /usr/local/lib/python3.6/dist-packages (from redis>=3.5.0->ray[default]==1.9.2) (21.0) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.6/dist-packages (from deprecated>=1.2.3->redis>=3.5.0->ray[default]==1.9.2) (1.12.1) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata->click>=7.0->ray[default]==1.9.2) (3.6.0) Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging>=20.4->redis>=3.5.0->ray[default]==1.9.2) (2.4.7) Requirement already satisfied: pyrsistent>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from jsonschema->ray[default]==1.9.2) (0.18.0) Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from jsonschema->ray[default]==1.9.2) (58.2.0) Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in ./.local/lib/python3.6/site-packages (from opencensus->ray[default]==1.9.2) (2.4.0) Requirement already satisfied: opencensus-context==0.1.2 in ./.local/lib/python3.6/site-packages (from opencensus->ray[default]==1.9.2) (0.1.2) Requirement already satisfied: contextvars in ./.local/lib/python3.6/site-packages (from opencensus-context==0.1.2->opencensus->ray[default]==1.9.2) (2.4) Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in ./.local/lib/python3.6/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.9.2) (1.54.0) Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in ./.local/lib/python3.6/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.9.2) (2.3.3) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.9.2) (0.2.1) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.9.2) (4.7.2) Requirement already satisfied: cachetools<5.0,>=2.0.0 in ./.local/lib/python3.6/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.9.2) (4.2.4) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->ray[default]==1.9.2) (2021.5.30) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->ray[default]==1.9.2) (1.26.7) Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<5,>=3.1.4->google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.9.2) (0.4.8) Requirement already satisfied: immutables>=0.9 in ./.local/lib/python3.6/site-packages (from contextvars->opencensus-context==0.1.2->opencensus->ray[default]==1.9.2) (0.16) WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available. Shared connection to 3.90.70.125 closed. You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command. Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com Processing ./.sky/sky_wheels/sky-0.1.dev0-py3-none-any.whl Requirement already satisfied: ray[default] in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (1.9.2) Requirement already satisfied: pendulum in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (2.1.2) Requirement already satisfied: PrettyTable in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (2.5.0) Requirement already satisfied: wheel in /usr/local/lib/python3.6/dist-packages (from sky==0.1.dev0) (0.37.0) Requirement already satisfied: jinja2 in /usr/local/lib/python3.6/dist-packages (from sky==0.1.dev0) (3.0.2) Requirement already satisfied: Click in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (8.0.3) Requirement already satisfied: pytest in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (6.2.5) Requirement already satisfied: docker in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (5.0.3) Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from sky==0.1.dev0) (2.5.1) Requirement already satisfied: oauth2client in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (4.1.3) Requirement already satisfied: tabulate in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (0.8.9) Requirement already satisfied: pycryptodome==3.12.0 in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (3.12.0) Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from sky==0.1.dev0) (1.1.5) Requirement already satisfied: colorama in /usr/local/lib/python3.6/dist-packages (from sky==0.1.dev0) (0.4.3) Requirement already satisfied: absl-py in ./.local/lib/python3.6/site-packages (from sky==0.1.dev0) (1.0.0) Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from absl-py->sky==0.1.dev0) (1.16.0) Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.6/dist-packages (from Click->sky==0.1.dev0) (4.8.1) Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/local/lib/python3.6/dist-packages (from docker->sky==0.1.dev0) (2.26.0) Requirement already satisfied: websocket-client>=0.32.0 in ./.local/lib/python3.6/site-packages (from docker->sky==0.1.dev0) (1.2.3) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests!=2.18.0,>=2.14.2->docker->sky==0.1.dev0) (3.2) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests!=2.18.0,>=2.14.2->docker->sky==0.1.dev0) (1.26.7) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.6/dist-packages (from requests!=2.18.0,>=2.14.2->docker->sky==0.1.dev0) (2.0.6) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests!=2.18.0,>=2.14.2->docker->sky==0.1.dev0) (2021.5.30) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata->Click->sky==0.1.dev0) (3.6.0) Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata->Click->sky==0.1.dev0) (3.10.0.2) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.6/dist-packages (from jinja2->sky==0.1.dev0) (2.0.1) Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.6/dist-packages (from networkx->sky==0.1.dev0) (4.4.2) Requirement already satisfied: httplib2>=0.9.1 in /usr/local/lib/python3.6/dist-packages (from oauth2client->sky==0.1.dev0) (0.19.1) Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.6/dist-packages (from oauth2client->sky==0.1.dev0) (0.4.8) Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/lib/python3/dist-packages (from oauth2client->sky==0.1.dev0) (0.2.1) Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from oauth2client->sky==0.1.dev0) (4.7.2) Requirement already satisfied: pyparsing<3,>=2.4.2 in /usr/local/lib/python3.6/dist-packages (from httplib2>=0.9.1->oauth2client->sky==0.1.dev0) (2.4.7) Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas->sky==0.1.dev0) (2.8.2) Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->sky==0.1.dev0) (2021.3) Requirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.6/dist-packages (from pandas->sky==0.1.dev0) (1.19.5) Requirement already satisfied: pytzdata>=2020.1 in ./.local/lib/python3.6/site-packages (from pendulum->sky==0.1.dev0) (2020.1) Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from PrettyTable->sky==0.1.dev0) (0.2.5) Requirement already satisfied: py>=1.8.2 in ./.local/lib/python3.6/site-packages (from pytest->sky==0.1.dev0) (1.11.0) Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.6/dist-packages (from pytest->sky==0.1.dev0) (21.2.0) Requirement already satisfied: iniconfig in ./.local/lib/python3.6/site-packages (from pytest->sky==0.1.dev0) (1.1.1) Requirement already satisfied: toml in /usr/local/lib/python3.6/dist-packages (from pytest->sky==0.1.dev0) (0.10.2) Requirement already satisfied: pluggy<2.0,>=0.12 in ./.local/lib/python3.6/site-packages (from pytest->sky==0.1.dev0) (1.0.0) Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from pytest->sky==0.1.dev0) (21.0) Requirement already satisfied: filelock in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (3.4.1) Requirement already satisfied: jsonschema in /usr/local/lib/python3.6/dist-packages (from ray[default]->sky==0.1.dev0) (3.2.0) Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (1.0.3) Requirement already satisfied: redis>=3.5.0 in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (4.1.1) Requirement already satisfied: protobuf>=3.15.3 in /usr/local/lib/python3.6/dist-packages (from ray[default]->sky==0.1.dev0) (3.18.1) Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from ray[default]->sky==0.1.dev0) (5.4.1) Requirement already satisfied: dataclasses in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (0.8) Requirement already satisfied: grpcio>=1.28.1 in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (1.43.0) Requirement already satisfied: aiohttp>=3.7 in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (3.8.1) Requirement already satisfied: colorful in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (0.5.4) Requirement already satisfied: gpustat>=1.0.0b1 in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (1.0.0b1) Requirement already satisfied: aiosignal in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (1.2.0) Requirement already satisfied: frozenlist in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (1.2.0) Requirement already satisfied: opencensus in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (0.8.0) Requirement already satisfied: aiohttp-cors in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (0.7.0) Requirement already satisfied: prometheus-client>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from ray[default]->sky==0.1.dev0) (0.11.0) Requirement already satisfied: aioredis<2 in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (1.3.1) Requirement already satisfied: smart-open in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (5.2.1) Requirement already satisfied: py-spy>=0.2.0 in ./.local/lib/python3.6/site-packages (from ray[default]->sky==0.1.dev0) (0.3.11) Requirement already satisfied: asynctest==0.13.0 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (0.13.0) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (4.0.2) Requirement already satisfied: multidict<7.0,>=4.5 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (5.2.0) Requirement already satisfied: yarl<2.0,>=1.0 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (1.7.2) Requirement already satisfied: idna-ssl>=1.0 in ./.local/lib/python3.6/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (1.1.0) Requirement already satisfied: hiredis in ./.local/lib/python3.6/site-packages (from aioredis<2->ray[default]->sky==0.1.dev0) (2.0.0) Requirement already satisfied: nvidia-ml-py3>=7.352.0 in ./.local/lib/python3.6/site-packages (from gpustat>=1.0.0b1->ray[default]->sky==0.1.dev0) (7.352.0) Requirement already satisfied: blessed>=1.17.1 in ./.local/lib/python3.6/site-packages (from gpustat>=1.0.0b1->ray[default]->sky==0.1.dev0) (1.19.1) Requirement already satisfied: psutil in ./.local/lib/python3.6/site-packages (from gpustat>=1.0.0b1->ray[default]->sky==0.1.dev0) (5.9.0) Requirement already satisfied: deprecated>=1.2.3 in ./.local/lib/python3.6/site-packages (from redis>=3.5.0->ray[default]->sky==0.1.dev0) (1.2.13) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.6/dist-packages (from deprecated>=1.2.3->redis>=3.5.0->ray[default]->sky==0.1.dev0) (1.12.1) Requirement already satisfied: pyrsistent>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from jsonschema->ray[default]->sky==0.1.dev0) (0.18.0) Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from jsonschema->ray[default]->sky==0.1.dev0) (58.2.0) Requirement already satisfied: opencensus-context==0.1.2 in ./.local/lib/python3.6/site-packages (from opencensus->ray[default]->sky==0.1.dev0) (0.1.2) Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in ./.local/lib/python3.6/site-packages (from opencensus->ray[default]->sky==0.1.dev0) (2.4.0) Requirement already satisfied: contextvars in ./.local/lib/python3.6/site-packages (from opencensus-context==0.1.2->opencensus->ray[default]->sky==0.1.dev0) (2.4) Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in ./.local/lib/python3.6/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->sky==0.1.dev0) (2.3.3) Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in ./.local/lib/python3.6/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->sky==0.1.dev0) (1.54.0) Requirement already satisfied: cachetools<5.0,>=2.0.0 in ./.local/lib/python3.6/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->sky==0.1.dev0) (4.2.4) Requirement already satisfied: immutables>=0.9 in ./.local/lib/python3.6/site-packages (from contextvars->opencensus-context==0.1.2->opencensus->ray[default]->sky==0.1.dev0) (0.16) sky is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel. WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available. You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command. running setup Shared connection to 3.90.70.125 closed. Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com Requirement already satisfied: boto3>=1.4.8 in /usr/local/lib/python2.7/dist-packages (1.17.112) Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python2.7/dist-packages (from boto3>=1.4.8) (0.10.0) Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /usr/local/lib/python2.7/dist-packages (from boto3>=1.4.8) (0.4.2) Requirement already satisfied: botocore<1.21.0,>=1.20.112 in /usr/local/lib/python2.7/dist-packages (from boto3>=1.4.8) (1.20.112) Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.7" in /usr/local/lib/python2.7/dist-packages (from s3transfer<0.5.0,>=0.4.0->boto3>=1.4.8) (3.3.0) Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python2.7/dist-packages (from botocore<1.21.0,>=1.20.112->boto3>=1.4.8) (2.8.2) Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python2.7/dist-packages (from botocore<1.21.0,>=1.20.112->boto3>=1.4.8) (1.26.7) Requirement already satisfied: six>=1.5 in /usr/local/lib/python2.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.21.0,>=1.20.112->boto3>=1.4.8) (1.16.0) WARNING: You are using pip version 20.3.4; however, version 21.3.1 is available. Shared connection to 3.90.70.125 closed. Shared connection to 3.90.70.125 closed. You should consider upgrading via the '/usr/bin/python2 -m pip install --upgrade pip' command. Cluster: min File Mount: (/tmp/setup.sh:/tmp/sky_setup_2521886668.sh) refers to a file. To ensure this mount updates properly, please use a directory. Loaded cached provider configuration If you experience issues with the cloud provider, try re-running the command with --no-config-cache. AWS config IAM Profile: ray-autoscaler-v1 [default] EC2 Key pair (all available node types): sky-key-1bbf6e VPC Subnets (all available node types): subnet-a1fea8ae, subnet-bcfe5d82, subnet-ce694684, subnet-b536ad9b, subnet-ae138fc9, subnet-3f39a063 [default] EC2 Security groups (all available node types): sg-0a45c15ae63192f37 [default] EC2 AMI (all available node types): ami-0e3c68b57d50caf64 Cluster Ray runtime will not be restarted due to `--no-restart`. Updating cluster configuration and running setup commands. Confirm [y/N]: y [automatic, due to --yes] <1/1> Setting up head node Prepared bootstrap config New status: waiting-for-ssh [1/7] Waiting for SSH to become available Running `uptime` as a test. Fetched IP: 3.90.70.125 Success. Updating cluster configuration. [hash=dcde2fa12c6551bf2334d6451bcc67f190a2b086] New status: syncing-files [2/7] Processing file mounts ~/.sky/sky_wheels/ from /tmp/sky_wheels/ /tmp/setup.sh from /tmp/sky_setup_2521886668.sh [3/7] No worker file mounts to sync New status: setting-up [4/7] No initialization commands to run. [5/7] Initalizing command runner [6/7] Running setup commands (0/4) pip3 install -U ray[default]==... (1/4) pip3 install ~/.sky/sky_wheels... (2/4) cd ~/sky_workdir && bash /tmp/... (3/4) pip install 'boto3>=1.4.8' [7/7] Starting the Ray runtime New status: up-to-date Useful commands Monitor autoscaling with ray exec /mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/config/user/min.yml 'tail -n 100 -f /tmp/ray/session_latest/logs/monitor*' Connect to a terminal on the cluster head: ray attach /mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/config/user/min.yml Get a remote shell to the cluster manually: ssh -o IdentitiesOnly=yes -i ~/.ssh/sky-key ubuntu@3.90.70.125 I 01-21 11:39:38 cloud_vm_ray_backend.py:730] Successfully provisioned or found existing VM. Setup completed. I 01-21 11:39:38 cloud_vm_ray_backend.py:732] I 01-21 11:39:38 cloud_vm_ray_backend.py:732] To log into the head VM: ssh min I 01-21 11:39:38 cloud_vm_ray_backend.py:732] Traceback (most recent call last): File "/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/sky/execution.py", line 119, in _execute cluster_name=cluster_name) File "/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/sky/backends/cloud_vm_ray_backend.py", line 1120, in provision _add_cluster_to_ssh_config(cluster_name, handle.head_ip, auth_config) File "/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/sky/backends/cloud_vm_ray_backend.py", line 143, in _add_cluster_to_ssh_config auth_config) File "/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/sky/backends/backend_utils.py", line 259, in add_cluster GitCredentialsHelper.start_agent() File "/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/sky/backends/backend_utils.py", line 354, in start_agent key_path = cls._get_ssh_key() File "/mnt/d/Romil/Berkeley/Research/sky-experiments/prototype/sky/backends/backend_utils.py", line 340, in _get_ssh_key key_path = [e for e in line.split() if '.ssh' in e][0] IndexError: list index out of range NAME LAUNCHED RESOURCES COMMAND STATUS min a few secs ago 1x AWS(m4.2xlarge) sky launch -c min ... UP
investigating now
gmittal
commented
2 years ago
I am unable to use sky launch. Git credential forwarding fails in _get_ssh_key.
Seems related to #211, reverting it (to
790e8dce2fdd0996906cd13e3130fb9933e9de3e) seems to work fine.I am using
gh authto setup my git credentials - could that be related?Steps to reproduce
Run
./examples/run_smoke_tests.shLogs