`TASK [dare.common.vault_init_config : unseal each vault pod until threshold is reached]` if Vault is not initialised

[mikej888]

Issue encountered while running DARE-SeRP-Dev-Deployment Version: 43f688f (Thu Sep 14 17:15:42 2023 +0100) main branch.

Running monolithic_mk8s_example/1-vm-setup-and-deploy.yaml sometimes failed at:

$ ansible-playbook -i vmware-host.yaml 1-vm-setup-and-deploy.yaml -v
...
TASK [dare.common.vault_init_config : unseal each vault pod until threshold is reached] ***
failed: [single_host] (item=0) => {"ansible_loop_var": "item",
"changed": true, "item": "0", "rc": 2, "return_code": 2, "stderr":
"Error unsealing: Error making API request.\n\nURL: PUT
http://127.0.0.1:8200/v1/sys/unseal\nCode: 400. Errors:\n\n* Vault is
not initialized\n", "stderr_lines": ["Error unsealing: Error making
API request.", "", "URL: PUT http://127.0.0.1:8200/v1/sys/unseal",
"Code: 400. Errors:", "", "* Vault is not initialized"], "stdout": "",
"stdout_lines": []}

A workaround that has worked is the following. Back up dare.common collection vault_init_config role init.yml task:

$ cp ~/.ansible/collections/ansible_collections/dare/common/roles/vault_init_config/tasks/init.yml ~/.ansible/collections/ansible_collections/dare/common/roles/vault_init_config/tasks/init.yml.bak

Edit ~/.ansible/collections/ansible_collections/dare/common/roles/vault_init_config/tasks/init.yml to

1. Remove task to initialise Vault, as it's already initialised.
2. Remove task to set a fact with output from the above.
3. Remove task to save unseal keys to a file, as it's already been done.
4. Add a new task to set a fact with the contents of output/vault/root-unseal.json from the Ansible control node (these correspond to the outputs of the Vault initialisation). i.e.:

   
   ---

• set_fact: vault_init: "{{ lookup('file', vault_config_output_folder + '/root-unseal.json') | from_json }}"

• include_tasks: unseal.yml loop: "{{ vault_pods|flatten(levels=1) }}" loop_control: loop_var: pod_name

  Edit playbook roles to only rerun `dare.common.vault_init_config` and `dare.common.argocd_deploy` i.e. comment out the following roles:
  ```yaml
  ...
  roles:
  # install and configure microk8s
  #    - dare.common.microk8s
  
  # deploy (via helm charts) k8s components that argo depends on
  #    - dare.common.helm_install
  ...

  Rerun playbook:

  $ ansible-playbook -i inventories/eidf-tre-teleport.yaml 1-vm-setup-and-deploy.yaml -v

  It would be good to include some check to ensure that no attempt is made to unseal the Vault until it has initialised.

[alee-x]

Hi @mikej888 ,

I'm a bit confused by this. The error message indicates that Vault isn't initialised, but you mention

Remove task to initialise Vault, as it's already initialised.

the unseal task set is also only accessible via the init task set, so it follows that Vault must have been initialised for ansible to get to the unseal.yml tasks.

if it's a timing issue (i.e. ansible making the unseal request too quickly after vault is initialised) then I suppose we can just add a step to pause for 5-10 seconds to wait for it. @JossWhittle any thoughts?

[mikej888]

Hi @alee-x, I found the message contradictory too. It happened a few times. However, rerunning the playbook as-is definitiely failed as it complained that the Vault was already initialised.

I searched for '"Error unsealing" "Vault is not initialized"' and found that others have had comparable issues...

• Unable to unseal vault post init activity in Vault HA on K8s using S3 Storage hashicorp/vault-helm#226
• vault is not initialized but is initialized hashicorp/vault#9618 This issue has a comment, December 2020, said a fix was 'included in the upcoming 1.6.1 release.'

However, nothing springs out as being a satisfactory explanation with a solution. The Vault version that was installed is 1.13.1.

[mikej888]

Fixed by @alee-x in #106, #107, #108, #109.