Auto-update for custom nodes

[keen99]

re using the http api to trigger a runSh job - https://github.com/Shippable/support/issues/3857

so I've been using this (for testing, getting to the point of being able to use it for real) for a few hundred+ triggers now I guess.

Intermittently, the runSh jobs don't get the ENV variables (from shipppable.resources.yml or sometimes from the http request) properly defined in the environment. Another trigger and they will be set. No changes to either the rSync repo nor the repo that the runSh job consumes nor the http request occurs between triggers.

To be clear, I have a param defined:

  - name: ansible_deploy_params
    type: params
    version:
      params:
        SSH_USER: "ansible"

and a job that consumes it:

  - name: deploy-via-ansible
    type: runSh
    steps:
      - IN: ansible_master_gitRepo
        switch: off
      ## ssh key integrations for runSh jobs DO NOT add the keys to the agent
      ## you need to ssh-add from /build/IN/name/name_key yourself
      - IN: _ssh_key_integration
        switch: off
      - IN: ansible_deploy_params
        switch: off
      - TASK:
        - script: set -u
        - script: echo "ENV..."
        - script: env |grep -e SSH_USER -e ENVIRONMENT
        - script: echo "Echo..."
        - script: ### this should die out since we set -u if these are missing
        - script: echo ENVIRONMENT $ENVIRONMENT SSH_USER $SSH_USER

(adjusted to reduce complexity of the script for the scope of this ticket)

and an http request to trigger the job:

curl -v -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: apiToken TOKEN' https://api.shippable.com/resources/RESOURCE/triggerNewBuildRequest -d '{"globalEnv":{"PLAYBOOK":"ansible-test.yml", "ENVIRONMENT":"testing", "RDS_SNAPSHOT":"false", "ALLOW_DOWNGRADE":"false", "POPULATIONBUILDER_VERSION":"", "POPULATIONBUILDER_ASSETS_VERSION":"", "BACKOFFICE_ASSETS_VERSION":"" }}'

I've noticed that the PARAM_NAME_PARAMS_VAR variables ARE set for the param defined variables in this case. http request env vars don't have such a variable that I can find.

successful job:

https://app.shippable.com/bitbucket/devexi/jobs/deploy-via-ansible/builds/59d401ba24ca780600332cfe/console

ENV...
SSH_USER=ansible
ENVIRONMENT=testing
ANSIBLE_DEPLOY_PARAMS_PARAMS_SSH_USER=ansible
Echo...
ENVIRONMENT testing SSH_USER ansible

http request vars missing inside job, but seen in UI, param env vars not set

https://app.shippable.com/bitbucket/devexi/jobs/deploy-via-ansible/builds/59d4022b24cffe070025b561/console

ENV...
ANSIBLE_DEPLOY_PARAMS_PARAMS_SSH_USER=ansible
Echo...
./bin/shippable-runner.sh: line 58: ENVIRONMENT: unbound variable
runSh failed with error: 1

http request env vars missing, and param env vars not set:

https://app.shippable.com/bitbucket/devexi/jobs/deploy-via-ansible/builds/59d4016aec6f3b07001641b5/console

ENV...
ANSIBLE_DEPLOY_PARAMS_PARAMS_SSH_USER=ansible
Echo...
./bin/shippable-runner.sh: line 58: ENVIRONMENT: unbound variable
runSh failed with error: 1

[a-murphy]

One of the custom nodes in your subscription has the build runner for version 5.5.4, and is the node where builds are missing environment variables. The params variable and custom variables you are trying to use were both introduced in version 5.7.2. I would suggest clicking on "re-initialize" on the oldest node's page and running the new script it will generate to update that node to the latest version.

[keen99]

thanks - I'll update that node. (I'll update all nodes)

This is a super painful process though, since it can't be automated, and it breaks required docker configs.

Since end users aren't aware of shippable build node releases, nor can we control what version is installed at first creation, this really shouldn't be a manual process (or, it should be optionally manual). Shippable should be pushing updates (via poll for custom nodes).

Additionally I've noticed that the shippable-maintained nodes have different versions across the fleet, which suggests that consistency is....lacking.

Why are nodes not updated automatically?

[a-murphy]

We don't currently update nodes automatically to avoid accidentally changing anything that users have configured on their nodes. In most cases, nodes are added by setting up SSH access to run the initialization script and upgrading existing custom nodes can be done by just clicking the "reinitialize" button. In your case, you decided to copy the scripts to your nodes instead of allowing SSH access, which means that the updated script must be copied to the node again.

I'll label this issue as a feature request and we'll look into updating the nodes automatically.

[mbrodala]

This is also related to #3744 and #3833.