search

dask / dask-ec2

Start a cluster in EC2 for dask.distributed
106 stars 37 forks source link

dask_ec2.libpepper.PepperException: Server error. #109

Open lionfish0 opened 6 years ago

lionfish0 commented 6 years ago

I get this exception when "installing workers". It doesn't happen all the time. I wonder if there's a race condition or similar that's causing it? Maybe something on the servers isn't initialised in time? Command: dask-ec2 up --keyname research --keypair .ssh/research.pem --region-name eu-west-1 --ami ami-c8b51fb1 --tags research:dp --count 1 --spot-count 16 --spot-price 0.796 --volume-size 10 --type c4.large --nprocs 2 I should say that I'm using my forked version of this repo.

The error messages:

Installing scheduler
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-0  | 22                   | 0               |
+---------+----------------------+-----------------+
Installing workers
Traceback (most recent call last):
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 130, in req
    f = urlopen(req, context=con)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 23, in start
    cli(obj={})
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 201, in up
    dask=dask, notebook=notebook, nprocs=nprocs, source=source)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 347, in provision
    ctx.invoke(dask_install, filepath=filepath, nprocs=nprocs, source=source)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/daskd.py", line 80, in dask_install
    output = cluster.salt_call("node-[1-9]*", "state.sls", ["dask.distributed.worker"])
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cluster.py", line 65, in salt_call
    return self.pepper.local(target, module, args)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 225, in local
    return self.low([low], path='/')
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 200, in low
    return self.req(path, lowstate)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 142, in req
    raise PepperException('Server error.')
dask_ec2.libpepper.PepperException: Server error.

The whole output:

DEBUG: Searching for default VPC
DEBUG: Default VPC found - Using VPC ID: vpc-7f49ce1a
DEBUG: Searching for default subnet in VPC vpc-7f49ce1a
DEBUG: Default subnet found - Using Subnet ID: subnet-8a0199fd
Launching nodes
DEBUG: Checking that keyname 'research' exists on EC2
DEBUG: Checking that security group 'dask-ec2-default' exists on EC2
DEBUG: Getting all security groups and filtering by VPC ID vpc-7f49ce1a and name dask-ec2-default
DEBUG: Found Security groups: [ec2.SecurityGroup(id='sg-9146afe9')]
DEBUG: Getting all security groups and filtering by VPC ID vpc-7f49ce1a and name dask-ec2-default
DEBUG: Found Security groups: [ec2.SecurityGroup(id='sg-9146afe9')]
DEBUG: Creating 16 spot instances on EC2
DEBUG: Creating 1 instances on EC2
DEBUG: spot request `sir-4xj866en` fulfilled!
DEBUG: spot request `sir-idh86iwm` fulfilled!
DEBUG: spot request `sir-aeti6fsq` fulfilled!
DEBUG: spot request `sir-i9c85n5p` fulfilled!
DEBUG: spot request `sir-wpvr5jnn` fulfilled!
DEBUG: spot request `sir-65t87h7q` fulfilled!
DEBUG: spot request `sir-e2mr463m` fulfilled!
DEBUG: spot request `sir-5f2i5hpq` fulfilled!
DEBUG: spot request `sir-txzg6kjm` fulfilled!
DEBUG: spot request `sir-32pr4app` fulfilled!
DEBUG: spot request `sir-p8e87jip` fulfilled!
DEBUG: spot request `sir-ppwi4fqm` fulfilled!
DEBUG: spot request `sir-eg986wwn` fulfilled!
DEBUG: spot request `sir-ctz84wtq` fulfilled!
DEBUG: spot request `sir-mntr5gam` fulfilled!
DEBUG: spot request `sir-t79g65ap` fulfilled!
DEBUG: all spots fulfilled!
DEBUG: Tagging instance 'i-0e1bddcde99715e6c'
DEBUG: Tagging instance 'i-0c96a28a6933f073c'
DEBUG: Tagging instance 'i-0c980a3ed3edb5078'
DEBUG: Tagging instance 'i-0618b6ae501690189'
DEBUG: Tagging instance 'i-0221d206dcfb9a982'
DEBUG: Tagging instance 'i-064379cd368ff4288'
DEBUG: Tagging instance 'i-0333b5e5f67e7728b'
DEBUG: Tagging instance 'i-0b3265f862d4b662b'
DEBUG: Tagging instance 'i-0f6edef96ac796c49'
DEBUG: Tagging instance 'i-0106523d86c02f84f'
DEBUG: Tagging instance 'i-0a1217026dd6f511e'
DEBUG: Tagging instance 'i-0a9280704174aa0dc'
DEBUG: Tagging instance 'i-06ec5d5d579372d53'
DEBUG: Tagging instance 'i-0aaab406064d9d47e'
DEBUG: Tagging instance 'i-04ae4cee6255f70fd'
DEBUG: Tagging instance 'i-0e4e84e330da6c5f9'
DEBUG: Tagging instance 'i-05692d9ccabeba9a1'
Checking SSH connection to nodes
DEBUG: Checking ssh connection for 34.253.212.200
DEBUG: Running command bash -c 'ls' on '34.253.212.200'
DEBUG: Checking ssh connection for 54.154.40.56
DEBUG: Running command bash -c 'ls' on '54.154.40.56'
DEBUG: Checking ssh connection for 34.242.214.91
DEBUG: Running command bash -c 'ls' on '34.242.214.91'
DEBUG: Checking ssh connection for 54.171.53.67
DEBUG: Running command bash -c 'ls' on '54.171.53.67'
DEBUG: Checking ssh connection for 54.171.131.4
DEBUG: Running command bash -c 'ls' on '54.171.131.4'
DEBUG: Checking ssh connection for 54.229.249.113
DEBUG: Running command bash -c 'ls' on '54.229.249.113'
DEBUG: Checking ssh connection for 34.253.208.68
DEBUG: Running command bash -c 'ls' on '34.253.208.68'
DEBUG: Checking ssh connection for 54.171.211.136
DEBUG: Running command bash -c 'ls' on '54.171.211.136'
DEBUG: Checking ssh connection for 54.229.24.59
DEBUG: Running command bash -c 'ls' on '54.229.24.59'
DEBUG: Checking ssh connection for 54.229.55.142
DEBUG: Running command bash -c 'ls' on '54.229.55.142'
DEBUG: Checking ssh connection for 54.171.43.34
DEBUG: Running command bash -c 'ls' on '54.171.43.34'
DEBUG: Checking ssh connection for 54.194.52.177
DEBUG: Running command bash -c 'ls' on '54.194.52.177'
DEBUG: Checking ssh connection for 54.246.136.108
DEBUG: Running command bash -c 'ls' on '54.246.136.108'
DEBUG: Checking ssh connection for 54.171.209.246
DEBUG: Running command bash -c 'ls' on '54.171.209.246'
DEBUG: Checking ssh connection for 54.246.229.87
DEBUG: Running command bash -c 'ls' on '54.246.229.87'
DEBUG: Checking ssh connection for 34.242.164.173
DEBUG: Running command bash -c 'ls' on '34.242.164.173'
DEBUG: Checking ssh connection for 176.34.137.27
DEBUG: Running command bash -c 'ls' on '176.34.137.27'
+-------------------+-----------+
| Node IP           | SSH check |
+===================+===========+
| 34.253.212.200:22 | True      |
| 54.154.40.56:22   | True      |
| 34.242.214.91:22  | True      |
| 54.171.53.67:22   | True      |
| 54.171.131.4:22   | True      |
| 54.229.249.113:22 | True      |
| 34.253.208.68:22  | True      |
| 54.171.211.136:22 | True      |
| 54.229.24.59:22   | True      |
| 54.229.55.142:22  | True      |
| 54.171.43.34:22   | True      |
| 54.194.52.177:22  | True      |
| 54.246.136.108:22 | True      |
| 54.171.209.246:22 | True      |
| 54.246.229.87:22  | True      |
| 34.242.164.173:22 | True      |
| 176.34.137.27:22  | True      |
+-------------------+-----------+
Bootstrapping salt master
DEBUG: Running command sudo -S bash -c 'curl -sS -L https://bootstrap.saltstack.com | sh -s -- -d -X -M -N stable' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -M -N -P -L -p salt-api stable' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/auto_accept.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/auto_accept.conf' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'apt-get install -y python-pip libssl-dev libffi-dev python-dev' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'pip install --upgrade pip packaging appdirs six' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'pip install cherrypy==3.2.3' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'pip install PyOpenSSL==16.2.0' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'salt-call --local tls.create_self_signed_cert' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/rest_cherrypy.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/rest_cherrypy.conf' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/external_auth.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/master.d/external_auth.conf' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'id -u saltdev &>/dev/null || useradd -p $(openssl passwd -1 saltdev) saltdev' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'service salt-master restart' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'service salt-api restart' on '34.253.212.200'
Bootstrapping salt minions
DEBUG: Installing salt-minion on all the nodes
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-6 stable' on '34.253.208.68'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-12 stable' on '54.246.136.108'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-3 stable' on '54.171.53.67'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-2 stable' on '34.242.214.91'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-1 stable' on '54.154.40.56'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-5 stable' on '54.229.249.113'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-7 stable' on '54.171.211.136'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-13 stable' on '54.171.209.246'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-9 stable' on '54.229.55.142'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-8 stable' on '54.229.24.59'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-4 stable' on '54.171.131.4'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-10 stable' on '54.171.43.34'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-14 stable' on '54.246.229.87'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-15 stable' on '34.242.164.173'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-11 stable' on '54.194.52.177'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-16 stable' on '176.34.137.27'
DEBUG: Running command sudo -S bash -c 'curl -L https://bootstrap.saltstack.com | sh -s -- -d -X -P -L -A 34.253.212.200 -i node-0 stable' on '34.253.212.200'
DEBUG: Configuring salt-mine on the salt minions
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/templates/mine_functions.conf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.229.24.59'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.229.55.142'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.53.67'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.242.214.91'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.229.249.113'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.154.40.56'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.246.229.87'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.246.136.108'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '176.34.137.27'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.242.164.173'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.209.246'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.131.4'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '34.253.208.68'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.43.34'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.171.211.136'
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /etc/salt/minion.d/mine.conf' on '54.194.52.177'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.229.24.59'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.229.55.142'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.53.67'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.242.214.91'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.154.40.56'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.246.229.87'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.229.249.113'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.242.164.173'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '176.34.137.27'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.131.4'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.208.68'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.209.246'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.246.136.108'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.43.34'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.171.211.136'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '54.194.52.177'
DEBUG: Restarting the salt-minion service
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.131.4'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.211.136'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.229.55.142'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.53.67'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.229.249.113'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.43.34'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.253.208.68'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.171.209.246'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.229.24.59'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.154.40.56'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.194.52.177'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.242.214.91'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '34.242.164.173'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '176.34.137.27'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.246.229.87'
DEBUG: Running command sudo -S bash -c 'service salt-minion restart' on '54.246.136.108'
Uploading salt formulas
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt to /srv/salt
DEBUG: Running command sudo -S bash -c 'mkdir -p /srv/salt' on '34.253.212.200'
DEBUG: Creating directory /tmp/.__tmp_copy mode=511
DEBUG: Creating directory /tmp/.__tmp_copy/supervisor mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor to /tmp/.__tmp_copy/supervisor
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/supervisor' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor/init.sls to /tmp/.__tmp_copy/supervisor/init.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/supervisor/settings.sls to /tmp/.__tmp_copy/supervisor/settings.sls
DEBUG: Creating directory /tmp/.__tmp_copy/dask mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask to /tmp/.__tmp_copy/dask
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask' on '34.253.212.200'
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed to /tmp/.__tmp_copy/dask/distributed
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed' on '34.253.212.200'
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/worker mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/worker to /tmp/.__tmp_copy/dask/distributed/worker
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/worker' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/worker/init.sls to /tmp/.__tmp_copy/dask/distributed/worker/init.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/debug.sls to /tmp/.__tmp_copy/dask/distributed/debug.sls
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/scheduler mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/scheduler to /tmp/.__tmp_copy/dask/distributed/scheduler
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/scheduler' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/scheduler/init.sls to /tmp/.__tmp_copy/dask/distributed/scheduler/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/dask/distributed/templates mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates to /tmp/.__tmp_copy/dask/distributed/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/dask/distributed/templates' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates/dask-worker.conf to /tmp/.__tmp_copy/dask/distributed/templates/dask-worker.conf
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/templates/dask-scheduler.conf to /tmp/.__tmp_copy/dask/distributed/templates/dask-scheduler.conf
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/init.sls to /tmp/.__tmp_copy/dask/distributed/init.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/dask/distributed/settings.sls to /tmp/.__tmp_copy/dask/distributed/settings.sls
DEBUG: Creating directory /tmp/.__tmp_copy/conda mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda to /tmp/.__tmp_copy/conda
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/conda' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/debug.sls to /tmp/.__tmp_copy/conda/debug.sls
DEBUG: Creating directory /tmp/.__tmp_copy/conda/templates mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/templates to /tmp/.__tmp_copy/conda/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/conda/templates' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/templates/conda.sh to /tmp/.__tmp_copy/conda/templates/conda.sh
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/init.sls to /tmp/.__tmp_copy/conda/init.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/conda/settings.sls to /tmp/.__tmp_copy/conda/settings.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/top.sls to /tmp/.__tmp_copy/top.sls
DEBUG: Creating directory /tmp/.__tmp_copy/jupyter mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter to /tmp/.__tmp_copy/jupyter
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter' on '34.253.212.200'
DEBUG: Creating directory /tmp/.__tmp_copy/jupyter/templates mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates to /tmp/.__tmp_copy/jupyter/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter/templates' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates/jupyter_notebook_config_py to /tmp/.__tmp_copy/jupyter/templates/jupyter_notebook_config_py
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/templates/jupyter-notebook.conf to /tmp/.__tmp_copy/jupyter/templates/jupyter-notebook.conf
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/init.sls to /tmp/.__tmp_copy/jupyter/init.sls
DEBUG: Creating directory /tmp/.__tmp_copy/jupyter/notebook mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook to /tmp/.__tmp_copy/jupyter/notebook
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/jupyter/notebook' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook/init.sls to /tmp/.__tmp_copy/jupyter/notebook/init.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/notebook/settings.sls to /tmp/.__tmp_copy/jupyter/notebook/settings.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/jupyter/settings.sls to /tmp/.__tmp_copy/jupyter/settings.sls
DEBUG: Creating directory /tmp/.__tmp_copy/java mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java to /tmp/.__tmp_copy/java
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java' on '34.253.212.200'
DEBUG: Creating directory /tmp/.__tmp_copy/java/openjdk mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk to /tmp/.__tmp_copy/java/openjdk
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java/openjdk' on '34.253.212.200'
DEBUG: Creating directory /tmp/.__tmp_copy/java/openjdk/templates mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/templates to /tmp/.__tmp_copy/java/openjdk/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/java/openjdk/templates' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/templates/java.sh to /tmp/.__tmp_copy/java/openjdk/templates/java.sh
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/init.sls to /tmp/.__tmp_copy/java/openjdk/init.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/env.sls to /tmp/.__tmp_copy/java/openjdk/env.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/openjdk/settings.sls to /tmp/.__tmp_copy/java/openjdk/settings.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/debug.sls to /tmp/.__tmp_copy/java/debug.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/init.sls to /tmp/.__tmp_copy/java/init.sls
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/java/settings.sls to /tmp/.__tmp_copy/java/settings.sls
DEBUG: Creating directory /tmp/.__tmp_copy/system mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system to /tmp/.__tmp_copy/system
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/system' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/macros.sls to /tmp/.__tmp_copy/system/macros.sls
DEBUG: Creating directory /tmp/.__tmp_copy/system/templates mode=511
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/templates to /tmp/.__tmp_copy/system/templates
DEBUG: Running command sudo -S bash -c 'mkdir -p /tmp/.__tmp_copy/system/templates' on '34.253.212.200'
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/templates/limits.conf to /tmp/.__tmp_copy/system/templates/limits.conf
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/salt/system/base.sls to /tmp/.__tmp_copy/system/base.sls
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy/* /srv/salt' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
DEBUG: Uploading directory /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/pillar to /srv/pillar
DEBUG: Running command sudo -S bash -c 'mkdir -p /srv/pillar' on '34.253.212.200'
DEBUG: Creating directory /tmp/.__tmp_copy mode=511
DEBUG: Uploading file /home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/formulas/pillar/top.sls to /tmp/.__tmp_copy/top.sls
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy/* /srv/pillar' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
Uploading conda and cluster settings
DEBUG: Uploading file /tmp/tmpgyvuu5cf to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/conda.sls' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
DEBUG: Uploading file /tmp/tmpe6af0crt to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/cluster.sls' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-11 | 6                    | 0               |
+---------+----------------------+-----------------+
DEBUG: Uploading file /tmp/tmpi_4r_utk to /tmp/.__tmp_copy
DEBUG: Running command sudo -S bash -c 'cp -rf /tmp/.__tmp_copy /srv/pillar/dask.sls' on '34.253.212.200'
DEBUG: Running command sudo -S bash -c 'rm -rf /tmp/.__tmp_copy' on '34.253.212.200'
Installing scheduler
+---------+----------------------+-----------------+
| Node ID | # Successful actions | # Failed action |
+=========+======================+=================+
| node-0  | 22                   | 0               |
+---------+----------------------+-----------------+
Installing workers
Traceback (most recent call last):
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 130, in req
    f = urlopen(req, context=con)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/home/mike/anaconda3/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 23, in start
    cli(obj={})
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 201, in up
    dask=dask, notebook=notebook, nprocs=nprocs, source=source)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/main.py", line 347, in provision
    ctx.invoke(dask_install, filepath=filepath, nprocs=nprocs, source=source)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cli/daskd.py", line 80, in dask_install
    output = cluster.salt_call("node-[1-9]*", "state.sls", ["dask.distributed.worker"])
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/cluster.py", line 65, in salt_call
    return self.pepper.local(target, module, args)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 225, in local
    return self.low([low], path='/')
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 200, in low
    return self.req(path, lowstate)
  File "/home/mike/anaconda3/lib/python3.6/site-packages/dask_ec2/libpepper.py", line 142, in req
    raise PepperException('Server error.')
dask_ec2.libpepper.PepperException: Server error.
lionfish0 commented 6 years ago

This might be a duplicate of https://github.com/dask/dask-ec2/issues/30