convert puppet script from cron job to slave job

[tfoote]

This will require tying each puppet job to the executor, but that shouldn't be a problem to enumerate them at reconfigure time.

As a bonus it will avoid puppet reconfiguring ever colliding with running jobs

[tfoote]

And it will give us visibility into an update failing.

[tfoote]

This is the bash script needed:

#!/bin/bash -x

# exit if anything fails
set -o errexit

cd /etc/puppet
sudo git fetch origin
sudo git reset --hard origin/master
sudo puppet apply /etc/puppet/manifests/site.pp

To do this right there should be one tied to each slave, possibly created by the setup script. It would also require being removed when torn down.

And if this script fails the machine should possibly take itself offline. Though it would be nice if this job could block other jobs, yet still be able to rerun itself.

Another possibility would be to make this job simply block and retry with a timeout to prevent this executor from running other jobs is puppet has failed. (A manual override would be to kill or disable this job)

[tfoote]

Getting closer, need to check all the return values

#!/bin/bash -x

# exit if anything fails
#set +o errexit

cd /etc/puppet

while true;
do

    sudo git fetch origin
    sudo git reset --hard origin/master
    if sudo puppet apply /etc/puppet/manifests/site.pp; 
    then
        echo "breaking out of loop on success"
        break
    fi
    echo "Sleeping to retry later"
    sleep $(( 10 * 60 ))
done

[dirk-thomas]

We will keep this in a cronjob. Closing.