search

dask / dask-ec2

Start a cluster in EC2 for dask.distributed
106 stars 37 forks source link

Cluster is not setup correctly #107

Open drorata opened 6 years ago

drorata commented 6 years ago

I'm trying to start a cluster using dask-ec2 up. Here's the command I'm using:

dask-ec2 up --keypair ~/.aws/dask_poc.pem \
 --name dask_poc \
--region-name eu-west-1 \ # Note I'm using different region from the default
--ami ami-47a23a30 \ # I also change the image to match the region I use
--type t2.nano \ # Very small machine; just for testing
--security-group dask-ec2-default \
--volume-size 100 \
--keyname dask_poc \
--tags dummy:tag

The process finished without errors but seems not to provide a cluster.

From the cluster: dask-ec2 ssh works, but ipython for instance is not installed. Furthermore, in the python shell from dask.distributed import Client, progress suggests that dask is not installed on the node.

From local: In Ipython, 

In [1]: from dask.distributed import Client, progress
In [2]: e = Client('34.252.95.207:8786')

Yields a timeout/connection refused error OSError: Timed out trying to connect to 'tcp://34.252.95.207:8786' after 5 s: in <distributed.comm.tcp.TCPConnector object at 0x10727e9b0>: ConnectionRefusedError: [Errno 61] Connection refused

In addition, I cannot access http://34.252.95.207:8888/ but I can ping the IP.

While the script was running, I noticed:

Installing scheduler
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-0  | 0                    | 0               |
+---------+----------------------+-----------------+
Installing workers
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-1  | 0                    | 0               |
+---------+----------------------+-----------------+
Dask.Distributed Installation succeeded

Which might already suggest that there's a problem.

DEBUG: Searching for default VPC
DEBUG: Default VPC found - Using VPC ID: vpc-22bab345
DEBUG: Searching for default subnet in VPC vpc-22bab345
DEBUG: Default subnet found - Using Subnet ID: subnet-73dfa328
Launching nodes
DEBUG: Checking that keyname 'dask_poc' exists on EC2
DEBUG: Checking that security group 'dask-ec2-default' exists on EC2
DEBUG: Getting all security groups and filtering by VPC ID vpc-22bab345 and name dask-ec2-default
DEBUG: Found Security groups: [ec2.SecurityGroup(id='sg-f21c7689')]
DEBUG: Getting all security groups and filtering by VPC ID vpc-22bab345 and name dask-ec2-default
DEBUG: Found Security groups: [ec2.SecurityGroup(id='sg-f21c7689')]
DEBUG: Creating 4 instances on EC2
DEBUG: Tagging instance 'i-0ce610fe1209ba0e6'
DEBUG: Tagging instance 'i-067e08dd680c4a9c6'
DEBUG: Tagging instance 'i-03de3bcf81d4c40e9'
DEBUG: Tagging instance 'i-0fb82743348b270db'
Checking SSH connection to nodes
DEBUG: Checking ssh connection for 34.252.95.207
DEBUG: Attempt 1/10 of function 'check_ssh' failed
DEBUG: Checking ssh connection for 34.252.95.207
DEBUG: Running command bash -c 'ls' on '34.252.95.207'
DEBUG: Checking ssh connection for 52.16.173.252
DEBUG: Running command bash -c 'ls' on '52.16.173.252'
DEBUG: Checking ssh connection for 52.212.24.235
DEBUG: Running command bash -c 'ls' on '52.212.24.235'
DEBUG: Checking ssh connection for 52.51.184.247
DEBUG: Running command bash -c 'ls' on '52.51.184.247'
+------------------+-----------+
| Node IP          | SSH check |
+==================+===========+
| 34.252.95.207:22 | True      |
| 52.16.173.252:22 | True      |
| 52.212.24.235:22 | True      |
| 52.51.184.247:22 | True      |
+------------------+-----------+
Bootstrapping salt master
DEBUG: Running command sudo -S bash -c 'curl -sS -L https://bootstrap.saltstack.com | sh -s -- -d -X -M -N stable' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -M -N -P -L -p salt-api stable' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/templates/auto_accept.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/auto_accept.conf' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'apt-get install -y python-pip libssl-dev libffi-dev python-dev' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'pip install --upgrade pip packaging appdirs six' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'pip install cherrypy' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'pip install PyOpenSSL==16.2.0' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'salt-call --local tls.create_self_signed_cert' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/templates/rest_cherrypy.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/rest_cherrypy.conf' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/templates/external_auth.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/external_auth.conf' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'id -u saltdev &>/dev/null || useradd -p $(openssl passwd -1 saltdev) saltdev' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'service salt-master restart' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'service salt-api restart' on '34.252.95.207'
Bootstrapping salt minions
DEBUG: Installing salt-minion on all the nodes
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.252.95.207 -i node-2 stable' on '52.212.24.235'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.252.95.207 -i node-1 stable' on '52.16.173.252'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.252.95.207 -i node-0 stable' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.252.95.207 -i node-3 stable' on '52.51.184.247'
DEBUG: Configuring salt-mine on the salt minions
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '52.51.184.247'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '52.212.24.235'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '52.16.173.252'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '52.51.184.247'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '52.212.24.235'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '52.16.173.252'
DEBUG: Restarting the salt-minion service
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '52.212.24.235'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '52.51.184.247'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '52.16.173.252'
Uploading salt formulas
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt to /srv/salt
DEBUG: Running command sudo -S bash -c 'mkdir -p /srv/salt' on '34.252.95.207'
DEBUG: Creating directory /tmp/.__tmp_copy mode=511
DEBUG: Creating directory /tmp/.__tmp_copy/dask mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask to /tmp/.__tmp_copy/dask
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask' on '34.252.95.207'
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed to /tmp/.__tmp_copy/dask/distributed
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed' on '34.252.95.207'
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/scheduler mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/scheduler to /tmp/.__tmp_copy/dask/distributed/scheduler
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/scheduler' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/scheduler/init.sls to /tmp/.__tmp_copy/dask/distributed/scheduler/init.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/settings.sls to /tmp/.__tmp_copy/dask/distributed/settings.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/init.sls to /tmp/.__tmp_copy/dask/distributed/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/templates mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates to /tmp/.__tmp_copy/dask/distributed/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/templates' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates/dask-worker.conf to /tmp/.__tmp_copy/dask/distributed/templates/dask-worker.conf
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates/dask-scheduler.conf to /tmp/.__tmp_copy/dask/distributed/templates/dask-scheduler.conf
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/debug.sls to /tmp/.__tmp_copy/dask/distributed/debug.sls
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/worker mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/worker to /tmp/.__tmp_copy/dask/distributed/worker
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/worker' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/worker/init.sls to /tmp/.__tmp_copy/dask/distributed/worker/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/conda mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda to /tmp/.__tmp_copy/conda
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/conda' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/settings.sls to /tmp/.__tmp_copy/conda/settings.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/init.sls to /tmp/.__tmp_copy/conda/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/conda/templates mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/templates to /tmp/.__tmp_copy/conda/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/conda/templates' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/templates/conda.sh to /tmp/.__tmp_copy/conda/templates/conda.sh
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/debug.sls to /tmp/.__tmp_copy/conda/debug.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/top.sls to /tmp/.__tmp_copy/top.sls
DEBUG: Creating directory /tmp/.__tmp_copy/java mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java to /tmp/.__tmp_copy/java
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java' on '34.252.95.207'
DEBUG: Creating directory /tmp/.__tmp_copy/java/openjdk mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk to /tmp/.__tmp_copy/java/openjdk
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java/openjdk' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/settings.sls to /tmp/.__tmp_copy/java/openjdk/settings.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/init.sls to /tmp/.__tmp_copy/java/openjdk/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/java/openjdk/templates mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/templates to /tmp/.__tmp_copy/java/openjdk/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java/openjdk/templates' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/templates/java.sh to /tmp/.__tmp_copy/java/openjdk/templates/java.sh
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/env.sls to /tmp/.__tmp_copy/java/openjdk/env.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/settings.sls to /tmp/.__tmp_copy/java/settings.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/init.sls to /tmp/.__tmp_copy/java/init.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/debug.sls to /tmp/.__tmp_copy/java/debug.sls
DEBUG: Creating directory /tmp/.__tmp_copy/system mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/system to /tmp/.__tmp_copy/system
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/system' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/base.sls to /tmp/.__tmp_copy/system/base.sls
DEBUG: Creating directory /tmp/.__tmp_copy/system/templates mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/templates to /tmp/.__tmp_copy/system/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/system/templates' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/templates/limits.conf to /tmp/.__tmp_copy/system/templates/limits.conf
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/macros.sls to /tmp/.__tmp_copy/system/macros.sls
DEBUG: Creating directory /tmp/.__tmp_copy/jupyter mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter to /tmp/.__tmp_copy/jupyter
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/settings.sls to /tmp/.__tmp_copy/jupyter/settings.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/init.sls to /tmp/.__tmp_copy/jupyter/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/jupyter/templates mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates to /tmp/.__tmp_copy/jupyter/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter/templates' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates/jupyter-notebook.conf to /tmp/.__tmp_copy/jupyter/templates/jupyter-notebook.conf
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates/jupyter_notebook_config_py to /tmp/.__tmp_copy/jupyter/templates/jupyter_notebook_config_py
DEBUG: Creating directory /tmp/.__tmp_copy/jupyter/notebook mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook to /tmp/.__tmp_copy/jupyter/notebook
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter/notebook' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook/settings.sls to /tmp/.__tmp_copy/jupyter/notebook/settings.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook/init.sls to /tmp/.__tmp_copy/jupyter/notebook/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/supervisor mode=511
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor to /tmp/.__tmp_copy/supervisor
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/supervisor' on '34.252.95.207'
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor/settings.sls to /tmp/.__tmp_copy/supervisor/settings.sls
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor/init.sls to /tmp/.__tmp_copy/supervisor/init.sls
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy/* /srv/salt' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
DEBUG: Uploading directory /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/pillar to /srv/pillar
DEBUG: Running command sudo -S bash -c 'mkdir -p /srv/pillar' on '34.252.95.207'
DEBUG: Creating directory /tmp/.__tmp_copy mode=511
DEBUG: Uploading file /Users/username/anaconda3/envs/my_conda_env/lib/python3.6/site-packages/dask_ec2/formulas/pillar/top.sls to /tmp/.__tmp_copy/top.sls
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy/* /srv/pillar' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
Uploading conda and cluster settings
DEBUG: Uploading file /var/folders/6z/h5jqtwss0m13gg40vv06gs_40000gn/T/tmp2mwafn3l to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/conda.sls' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
DEBUG: Uploading file /var/folders/6z/h5jqtwss0m13gg40vv06gs_40000gn/T/tmp42r8vosu to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/cluster.sls' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-1  | 6                    | 0               |
+---------+----------------------+-----------------+
DEBUG: Uploading file /var/folders/6z/h5jqtwss0m13gg40vv06gs_40000gn/T/tmp11kv24gq to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/dask.sls' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
Installing scheduler
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-0  | 0                    | 0               |
+---------+----------------------+-----------------+
Installing workers
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-1  | 0                    | 0               |
+---------+----------------------+-----------------+
Dask.Distributed Installation succeeded

Addresses
---------
Web Interface:    http://34.252.95.207:8787/status
TCP Interface:           34.252.95.207:8786

To connect from the cluster
---------------------------

dask-ec2 ssh  # ssh into head node
ipython  # start ipython shell

from dask.distributed import Client, progress
c = Client('127.0.0.1:8786')  # Connect to scheduler running on the head node

To connect locally
------------------

Note: this requires you to have identical environments on your local machine and cluster.

ipython  # start ipython shell

from dask.distributed import Client, progress
e = Client('34.252.95.207:8786')  # Connect to scheduler running on the head node

To destroy
----------

dask-ec2 destroy
Installing Jupyter notebook on the head node
DEBUG: Uploading file /var/folders/6z/h5jqtwss0m13gg40vv06gs_40000gn/T/tmp01xcpfl4 to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/jupyter.sls' on '34.252.95.207'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.252.95.207'
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-0  | 0                    | 0               |
+---------+----------------------+-----------------+
Jupyter notebook available at http://34.252.95.207:8888/
Login with password: jupyter