Adding Vault and Secrets to our ansible scripts

[cooldracula]

WIP

This will likely end up being a major PR, that adds secrets to our ansible scripts, adjusting how our inventory files and playbooks are organized in the process.

At the moment, it introduces an inventories dir, organized around our services or concerns (right now, using the notification service as an example). Within each concern is an inventory file and the group vars related to it, with the group_vars organized by the same groupings as the inventory file.

For example, we have the dir inventories/notifications_service organized like so:

inventories/notifications_service
├── group_vars
│   ├── all
│   │   ├── vars.yml
│   │   └── vault.yml
│   ├── development
│   │   ├── vars.yml
│   │   └── vault.yml
│   └── production
│       ├── vars.yml
│       └── vault.yml
└── inventory.yml

and the inventory.yml in that dir is structured as:

---
notifications_service:
  hosts:
   notifications.nos.social:
   dev-notifications.nos.social:
  children:
    production:
      hosts:
        notifications.nos.social:
    development:
      hosts:
        dev-notifications.nos.social:

And so every grouping in the inventory corresponds to its own group_vars directory, and each directory contains its own vars.yaml and vault.yml. The vars.yml contain the public variables specific to that grouping, and vault.yml the same but with the secret values.

The vault password is the same for every file, will be shared outside of this pr, and it is expected that you keep that password somewhere safe via a password manager. To use it with these playbooks, set the env var VAULT_PASS to our pass. I've updated the ansible.cfg so that it reads from a password file called .vault_pass and then added the .vault_pass file, which is a bash script that just echoes the above env var. The intention is that we don't have to enter the password every time, but also don't accidentally commit it here.

I want to add inventory files and secrets for all the major services we run currently and additional documentation with example playbooks. The PR does not feel ready until then, but keen for feedback now to make sure this path works with everyone!

[cooldracula]

@mplorentz :

If I were to run something like ansible-playbook -I inventories/notifications_service/inventory.yml how would it know to load all the vars from the various grou_vars subdirectory?

It is a built-in feature of ansible to look for group_vars in the same dir as the inventory file running, and it maps any children in the inventory to the child dirs in group_vars with a matching name. I added an example inventory to this PR to show this in better detail (see readme).

It seems like it would be simpler to just define all the vars in the inventory.yml, is there a reason you aren't doing it that way?

Hahaha, ironically I went this way for improved readability when dealing with the secrets. If we define all the variables in the inventory file, and some of those variables we want to be secret, then we'd need to encrypt the entire inventory file. You would only be able to read and review it by either decrypting it or running ansible-vault view. This makes it harder to review and more easily invites the risk of decrypting the file, changing a secret, and then committing the inventory without remembering to re-encrypt.

The structure in this pr lets us have most everything unencrypted and readable from github, and the vars themselves are known. If we want to change a secret, we are only decrypting a specific secrets file, and this feels less risky.

Another option is that we could define an inventory like so:

 ---
example:
  vars_file:
    - vars/vault.yml
  hosts:
   ansible.fun:
   dev-notifications.nos.social:
  vars:
    port: 1234
    image: docker/busybox
    password: "{{ vault_password }}"
  children:
    lab:
      hosts:
        ansible.fun:
          image_tag: stable
          password: "{{ vault_lab_password }}"
    nostr:
      hosts:
        dev-notifications.nos.social:
          image_tag: latest
          password: "{{ vault_nostr_password }}"

This might be more readable? My initial sense was that this style would be harder to read/maintain with ilarger inventorie, but I might be wrong! I can rewrite the notifications inventory in this style and see if it works well.

[mplorentz]

@cooldracula that all makes sense, thanks. I agree having to manually decrypt/encrypt the inventory file is a no-go.

It does seem like the ansible way is many layers of directories. Personally I would find the single inventory file easier to use. Even if the file gets stupidly large the command line tools I use make it easier to navigate a large file than a hierarchy of folders that need to be kept in sync with the inventory file. Do you prefer the folders? I could go either way, I don't want to spend too much effort swimming upstream.

[cooldracula]

I did some further experimenting and think the latest changes show a fairly readable compromise.

There is still an inventories directory, with our notifications_service inventory file at:

inventories/notifications_service/inventory.yml

But all variables are now included in that inventory file. Any secret variables have the structure of:

some_secret_var: {{ vault_some_secret_var }}

And`` the actual secrets are held in the group_vars of that directory, e.g:

inventories/notifications_service/group_vars/prod/vault.yml

So hopefully you can quickly scan and understand the inventory file and are only dealing with the deeply nested variables files when you are explicitly changing secrets.

Thereason for this structure is to have the most consistent behaviour with our variables. I was mistaken with the yaml in my last comment, you cannot include avars_fileattribute in an inventory file. Inventory files expect thegroup_vars` directory instead. I experimented with having the inventory file at the top level and then just one big group_vars directory organized by topic, but could not get consistent behaviour for the child groups grabbing their appropriate vars. I don't want to risk prod variables intermixing with dev variables in unexpected ways, and so went the slightly more verbose/nested route.

We could use the vars_file attribute in the playbook, but I prefer the variables to live close to the inventories, and not to playbooks. This is more for intuitive reasons than any 'ansible best practice' i could articulate at the moment.

If this structure looks alright, I can adjust the other inventory files in the repo to match it.

[cooldracula]

This accidentally became a large PR as adding in this new inventories and secrets structure required a lot of small changes across our roles and a cleanup/refactor of the root-level inventories and playbooks.

It also now includes the new-do-droplet playbook, as I wanted to make sure this structure would work for creating a wholly new server with the correct service on it.

I have tested the structure with a dev rss service and a dev notifications service. In both cases, I was able to pass in an inventory file to the playbook new-do-droplet plus some additional variables (like domain and the additional roles to run). They successfully went from non-existent server to fully deployed service.

I still need to move our metrics inventories over and double check they work, but it is not likely we'll need to deploy them a bunch so it was low priority.

I am not sure the best way to review this,given the size of the pr. My apologies for this, but maybe a scheduled pair might be good?