spotty-cloud / spotty

Training deep learning models on AWS and GCP instances
https://spotty.cloud
MIT License
493 stars 43 forks source link

Mention in the docs how to access an instance from another machine #48

Closed vadimkantorov closed 5 years ago

vadimkantorov commented 5 years ago

I have some instances running some tasks (managed by spotty). I'm trying to check the execution via ssh from another machine and getting Permission denied (publickey). from ssh.

The new machine also has awscli configured to the same access and secret key.

apls777 commented 5 years ago

When you start an instance, Spotty creates an EC2 Key Pair and downloads a private key to the ~/.spotty/keys/aws directory. If you want to have access to the instance from a different machine, copy the key there to the same directory.

vadimkantorov commented 5 years ago

Now I can't ssh even from the machine that started the instance:

vadimkantorov@5CD8223LBY:~/convasr/scripts$ spotty start
Syncing the project with S3 bucket...
Preparing CloudFormation template...
  - volume "convasr-instance-train1-docker" (vol-0395c41421eab9040) will be attached
  - volume "convasr-instance-train1-workspace" (vol-0aacfb50fabe22a0e) will be attached
  - volume "dataset_open_stt" (vol-09085bee257b3b52d) will be attached
  - availability zone: us-east-1d
  - maximum Spot Instance price: 4.0000
  - AMI: "Deep Learning Base AMI (Ubuntu) Version 19.0" (ami-0b5b391ed8ccaa538)
  - Docker data will be stored on the "docker" volume

Volumes:
+------------------+-------------------+------------+-----------------+
| Name             | Container Dir     | Type       | Deletion Policy |
+==================+===================+============+=================+
| dataset_open_stt | /dataset_open_stt | EBS volume | Retain Volume   |
+------------------+-------------------+------------+-----------------+
| workspace        | /workspace        | EBS volume | Retain Volume   |
+------------------+-------------------+------------+-----------------+
| docker           | -                 | EBS volume | Retain Volume   |
+------------------+-------------------+------------+-----------------+

Waiting for the stack to be created...
  - launching the instance...
  - waiting for the Docker container to be ready...
Error:
------
Stack "spotty-instance-convasr-instance-train1" was not created.
Please, see CloudFormation logs for the details.
vadimkantorov@5CD8223LBY:~/convasr/scripts$ spotty ssh
['ssh', '-ti', '/home/vadimkantorov/.spotty/keys/spotty-key-convasr-us-east-1', '-o', 'StrictHostKeyChecking no', 'ubuntu@34.228.73.237', 'tmux new -s spotty-ssh-container -A "sudo /tmp/spotty/instance/scripts/container_bash.sh || tmux set remain-on-exit on"']
Warning: Permanently added '34.228.73.237' (ECDSA) to the list of known hosts.
ubuntu@34.228.73.237: Permission denied (publickey).

The keyfile indeed exists, but somehow isn't accepted by the instance.

vadimkantorov commented 5 years ago

Hmm, I removed the keys and restarted, ssh half-works. It logins, but the "Pane is dead" immediately.

apls777 commented 5 years ago

@vadimkantorov If you scroll the pane one line to the top, you'll see the error message (the "Pane is dead" message itself is being printed at the end of the pane and hides the first line).

Looking at the output you sent, it seems that your container is failed to start, so I assume the error says that the container not found. So you need to figure out what's wrong with it from the logs. You should be able to connect to the host using the spotty ssh -H command.

Have you already managed to connect to the same instance from different machines?

vadimkantorov commented 5 years ago

Finally it was not so pressing since I carried the laptop with me

Looking at CloudFormation logs was helpful at many occasions. It would be great if these got downloaded automatically if requested (or if an error occurred).

apls777 commented 5 years ago

Added it to FAQ in the docs: How to ssh to a Spotty instance from a different machine?