search

dask / dask-ec2

Start a cluster in EC2 for dask.distributed
106 stars 37 forks source link

check_ssh failed #37

Closed raymondchua closed 7 years ago

raymondchua commented 7 years ago

Hi, I tried to do the following: dask-ec2 up --keyname raymond --keypair raymond.pem --name raymond_dask --region-name eu-central-1 --ami ami-f9619996 --username ubuntu --type t2.micro

But I keep getting ssh connection errors:

Checking SSH connection to nodes
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 1/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 2/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 3/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 4/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 5/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 6/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 7/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 8/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 9/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 35.156.103.53
DEBUG: Attempt 10/10 of function 'check_ssh' failed
ERROR: Retries limit exceeded
raymondchua commented 7 years ago

Is there a way to increase the number of retries? I think it will work if I can have more retries. When it is trying to connect, my instance is still initialising.

raymondchua commented 7 years ago

I manage to edit the source code and give it a 1000 retries. However, even when my instance is ready and running, I should have the ssh connection failed error.

quasiben commented 7 years ago

@raymondchua can you manually ssh to the machine ?

ssh ubuntu@EC2_IP -i raymond.pem

It could be a permission issue with the pem file. Also, t2.micro is probably too small an instance for dask to be setup/run

raymondchua commented 7 years ago

@quasiben thanks for the reply. I can manually ssh into the machine using: ssh -i "raymond.pem" ubuntu@.. So it is a bit strange for me. I am using t2.micro, ubuntu image.

quasiben commented 7 years ago

Can you try with a larger instance? Perhaps m3.2xlarge ? And be mindful of the cost: https://github.com/dask/dask-ec2/issues/36

raymondchua commented 7 years ago

@quasiben I have tried with m3.2xlarge and it failed. I am suspecting that it could be due to my proxy settings. Is there a way I can set the proxy settings?

raymondchua commented 7 years ago

I manage to solve it by adding the following to the SSHClient class. conf = paramiko.SSHConfig() conf.parse(open(os.path.expanduser('~/.ssh/config'))) self.host_conf = conf.lookup(self.host)

mynameisvinn commented 7 years ago

for me, it was an issue with my security group ("dask-ec2-default"), which didnt have the proper inbound rules in place. without explicit inbound permission, your local machine wont be able to ssh into your ec2 instances.

to change this, go security groups (under "Network & Security"). then select the security group, which, as of writing, is called "dask-ec2-default". highlight it and add an inbound rule: 

type: ssh
port: 22
source: my_ip

try dask-ec2 up --keyname raymond --keypair raymond.pem again.

hope this helps!