Supervisor fails to restart on `make update`

[mhkeller]

In automation, Supervisor throws the following error. The following is from running it locally in virtual box but i've also seen this on aws.

TASK: [restart services] ******************************************************
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE service name=redis-server state=restarted
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1444595542.67-45672370239094 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1444595542.67-45672370239094 && echo $HOME/.ansible/tmp/ansible-tmp-1444595542.67-45672370239094'
<127.0.0.1> PUT /var/folders/gp/__h0yj4s4lsc84w79k4b8mnh0000gn/T/tmpkijYFH TO /home/vagrant/.ansible/tmp/ansible-tmp-1444595542.67-45672370239094/service
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=iznhsxohdqnievhvhdzgpyofzjlssfrc] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-iznhsxohdqnievhvhdzgpyofzjlssfrc; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1444595542.67-45672370239094/service; rm -rf /home/vagrant/.ansible/tmp/ansible-tmp-1444595542.67-45672370239094/ >/dev/null 2>&1'"'"''
changed: [newslynx] => (item=redis-server) => {"changed": true, "item": "redis-server", "name": "redis-server", "state": "started"}
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE service name=postgresql state=restarted
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1444595544.0-275620671966763 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1444595544.0-275620671966763 && echo $HOME/.ansible/tmp/ansible-tmp-1444595544.0-275620671966763'
<127.0.0.1> PUT /var/folders/gp/__h0yj4s4lsc84w79k4b8mnh0000gn/T/tmpNgMlSy TO /home/vagrant/.ansible/tmp/ansible-tmp-1444595544.0-275620671966763/service
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=absxkoodsfwrgwvvcbmlsheqnavcrubo] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-absxkoodsfwrgwvvcbmlsheqnavcrubo; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1444595544.0-275620671966763/service; rm -rf /home/vagrant/.ansible/tmp/ansible-tmp-1444595544.0-275620671966763/ >/dev/null 2>&1'"'"''
changed: [newslynx] => (item=postgresql) => {"changed": true, "item": "postgresql", "name": "postgresql", "state": "started"}
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE service name=supervisor state=restarted
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1444595547.64-13136293566710 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1444595547.64-13136293566710 && echo $HOME/.ansible/tmp/ansible-tmp-1444595547.64-13136293566710'
<127.0.0.1> PUT /var/folders/gp/__h0yj4s4lsc84w79k4b8mnh0000gn/T/tmpKx_Tfk TO /home/vagrant/.ansible/tmp/ansible-tmp-1444595547.64-13136293566710/service
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=bkyrxsycxcfbpnojdfwrdvdtpcbjbjqe] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-bkyrxsycxcfbpnojdfwrdvdtpcbjbjqe; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1444595547.64-13136293566710/service; rm -rf /home/vagrant/.ansible/tmp/ansible-tmp-1444595547.64-13136293566710/ >/dev/null 2>&1'"'"''
failed: [newslynx] => (item=supervisor) => {"failed": true, "item": "supervisor"}
msg: Stopping supervisor: supervisord.
Starting supervisor:
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE service name=nginx state=restarted
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1444595547.99-128591121994421 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1444595547.99-128591121994421 && echo $HOME/.ansible/tmp/ansible-tmp-1444595547.99-128591121994421'
<127.0.0.1> PUT /var/folders/gp/__h0yj4s4lsc84w79k4b8mnh0000gn/T/tmpof2J6o TO /home/vagrant/.ansible/tmp/ansible-tmp-1444595547.99-128591121994421/service
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/mike/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/mike/wrk/projects/newslynx/automation/.vagrant/machines/newslynx/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=camqxmxxijzxazceygnltsgzvavwpjij] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-camqxmxxijzxazceygnltsgzvavwpjij; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1444595547.99-128591121994421/service; rm -rf /home/vagrant/.ansible/tmp/ansible-tmp-1444595547.99-128591121994421/ >/dev/null 2>&1'"'"''
changed: [newslynx] => (item=nginx) => {"changed": true, "item": "nginx", "name": "nginx", "state": "started"}

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/Users/mike/update.yaml.retry

newslynx                   : ok=7    changed=6    unreachable=0    failed=1

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

[abelsonlive]

Does the API still come up though? This is weird because it looks from the logs like it actually is restarting...

msg: Stopping supervisor: supervisord.
Starting supervisor:

[mhkeller]

im not sure since i figured if vagrant fails, none of the changes go into effect? or maybe that's not true. what would be a good way to test that it was restarted correctly and not just the old process that's running without having been restarted. i put on the VAGRANT_INFO=info env variable for more logging. i could send that or are you getting the error as well?

On Mon, Oct 12, 2015 at 4:40 PM, Brian Abelson notifications@github.com wrote:

Does the API still come up though? This is weird because it looks from the logs like it actually is restarting...

msg: Stopping supervisor: supervisord. Starting supervisor:

— Reply to this email directly or view it on GitHub https://github.com/newslynx/opportunities/issues/169#issuecomment-147513823 .

@mhkeller mhkeller.com