Improve Ansible quality & runability

[rvagg]

Outstanding item from #873 seems to be related to our Ansible scripts, particularly:

1. Our lack of confidence in their idempotency in general and therefore there's a lack of willingness to rerun the scripts across our cluster to fix problems like #873
2. host_vars setup for non-infra non-jenkins-admins people—you need Jenkins secrets in host_vars in order to fully run the scripts across hosts, build/test people should probably have an easier way to get these & set it up (I think?)

[gibfahn]

host_vars setup for non-infra non-jenkins-admins people—you need Jenkins secrets in host_vars in order to fully run the scripts across hosts, build/test people should probably have an easier way to get these & set it up (I think?)

Having a web frontend like Ansible Tower which auto-manages the secrets for you would be amazing. I'd like to click a button to run a script on a host.

[rvagg]

good news to report, in #964 I have them running repeatedly on Ubuntu, Debian8 and Fedora: ansible-playbook playbooks/jenkins/worker/create.yml --limit test-*ubuntu*-x*,test-*fedora*,test-*debian8-x*. I can run that multiple times without serious problems. Failures are due to some old Fedora machines not being able to fetch updates cause they are EOL.

Also, maybe as a hint. I set up some of my missing host_vars files by doing the following type of thing:

upstart

parallel-ssh -H test-digitalocean-fedora22-x64-1 -H test-digitalocean-fedora23-x64-1 -H test-digitalocean-fedora24-x64-1 -H test-digitalocean-fedora25-x64-1 -H test-digitalocean-fedora25-x64-2 -H test-rackspace-fedora22-x64-1 -H test-rackspace-fedora23-x64-1 -H test-rackspace-fedora24-x64-1 -i 'cat /etc/init/jenkins.conf | grep secret'
parallel-ssh -H test-digitalocean-ubuntu1404-x64-1 -H test-digitalocean-ubuntu1404-x86-1 -H test-softlayer-ubuntu1404-x64-1 -H test-softlayer-ubuntu1404-x86-1 -i 'cat /etc/init/jenkins.conf | grep secret'

systemd

parallel-ssh -H test-digitalocean-ubuntu1604-x86-1 -H test-digitalocean-ubuntu1610-x64-1 -H test-digitalocean-ubuntu1610-x64-2 -H test-nearform_intel-ubuntu1604-x64-1 -H test-nearform_intel-ubuntu1604-x64-2 -H test-rackspace-ubuntu1604-x64-1 -H test-rackspace-ubuntu1604-x64-2 -i 'cat /lib/systemd/system/jenkins.service | grep secret'

then manually doing this type of thing with the results:

echo 'secret: abcxyz1234567890....' > host_vars/test-rackspace-fedora24-x64-1

This could be automated a bit more and we could even make a set of ansible scripts to do this. It'd be easier to let all of build/test get these secrets off existing hosts than trying to get them access via jenkins.

[maclover7]

Below is where we are (or at least what I have been able to do myself) with the different Ansible scripts -- we are slowly migrating away from setup and to ansible, with the main stragglers being Raspberry Pi machines and Windows machines, but we are getting there. I want to try and write some more docs at some point about getting from no setup to running ansible-playbook.

`playbooks/jenkins/docker-host.yaml` - `test-digitalocean-ubuntu1604_docker-x64-1` - `test-digitalocean-ubuntu1604_docker-x64-2` - `test-joyent-ubuntu1604_docker-x64-1` - `test-softlayer-ubuntu1604_docker-x64-1` `playbooks/jenkins/linter.yml` - `test-rackspace-freebsd10-x64-1` - `test-joyent-freebsd10-x64-2` `jenkins/worker/create.yml` - `test-digitalocean-debian8-x64-1` - `test-rackspace-debian8-x64-1` - `test-rackspace-debian8-x64-2` - `test-softlayer-debian8-x86-1` - `test-digitalocean-freebsd10-x64-1` - `test-digitalocean-ubuntu1604-x86-1` - `test-nearform_intel-ubuntu1604-x64-1` - `test-nearform_intel-ubuntu1604-x64-2` - `test-packetnet-ubuntu1604-arm64-1` - `test-packetnet-ubuntu1604-arm64-2` - `test-rackspace-ubuntu1604-x64-1` - `test-rackspace-ubuntu1604-x64-2` - `test-joyent-ubuntu1710-x64-1` - `test-joyent-freebsd10-x64-1` - `test-joyent-ubuntu1710-x64-2` - `test-digitalocean-freebsd11-x64-2` - `test-digitalocean-ubuntu1404-x64-1` - `test-digitalocean-ubuntu1404-x86-1` - `test-softlayer-ubuntu1404-x64-1` - `test-softlayer-ubuntu1404-x86-1` - `test-digitalocean-freebsd11-x64-1` - `test-digitalocean-fedora27-x64-1` - `test-rackspace-fedora27-x64-1` - `test-digitalocean-fedora26-x64-1` - `test-rackspace-fedora26-x64-1` - `test-digitalocean-fedora25-x64-1` - `test-digitalocean-fedora25-x64-2` - `test-digitalocean-fedora24-x64-1` - `test-rackspace-fedora24-x64-1` - `test-rackspace-fedora23-x64-1` - `test-digitalocean-fedora23-x64-1` - `test-packetnet-centos7-arm64-1` - `test-packetnet-centos7-arm64-2` - `test-rackspace-centos7-x64-1` - `test-softlayer-centos7-x64-1`

[juggernaut451]

would love to contribute on this. @gibfahn @maclover7 @rvagg can someone mentor me on this

[maclover7]

Moving this to #1277