Step-by-step documentation of Vault approle token renewal for werker

[tjtelan]

Just went through this workflow for the first time using internal docs. Given that it seems to be a fairly generic workflow for Vault approles, and our example policies are committed already, we can do our future selves a favor by writing down token renewal as a strict set of steps.

It wouldn't hurt to add in systemd related files into the deploy directory, so we can add copy/paste systemctl commands too.

[tjtelan]

just before this leaves my memory, these were the steps:

This probably doesn't need to be done each time, but for accuracy, this is what I ran

• Create the policy (01_role_create.sh)
• Create the approle (02_approle_add.sh)

The following commands return output we need

• Get the role_id (vault read auth/approle/role/ocelot/role-id)
• Get the secret_id (vault write -field secret_id -f auth/approle/role/ocelot/secret-id)
• Get the werker token (vault write -field token auth/approle/login secret_id=<secret_id> role_id=<role_id>)

(Or the super simple one-liner version)

vault write -field token auth/approle/login secret_id=$(vault write -field secret_id -f auth/approle/role/ocelot/secret-id) role_id=$(vault read -field=role_id auth/approle/role/ocelot/role-id)

Finally...

• Add the werker token to /etc/vaulted/token on the werker vm (This is also not documented in the wiki within the werker server deployment instructions )
• Restart your werker (systemctl restart werker.service)
• If not already the case, enable this service, (systemctl enable werker.service)

[tjtelan]

I had to do this again, and I padded the one-liner. I think I know how I would want to automate this task. But we should consider offering suggestions for managing this proactively, since I seem to be finding out about werkers being locked out after the fact. Slack alerting would be fine, as long as it is documented how an operator might set that up.

But first, this is me updating the token on a host, then restarting werker vault write -field token auth/approle/login secret_id=$(vault write -field secret_id -f auth/approle/role/ocelot/secret-id) role_id=$(vault read -field=role_id auth/approle/role/ocelot/role-id) | ssh werker-user@werker-host "cat | sudo tee /etc/vaulted/token; sudo systemctl restart werker && sudo systemctl status werker"

I think these details need to be configurable:

• vault addr (+ read from environment)
• vault token (+ read from environment)
• Consul host (or better yet, just using Consul as a inventory for werker node vms)
• Token ttl (Being able to set this short or further in the future would be convenient)

The script (probably Ansible) will call the local vault instance, and run the vault commands to get a werker token. Register the output, then ssh into the werkers. Replace the token in the file, then restart the werker service.