Required Index Not Present

[mjnorman]

SUMMARY

Utilizing Approle auth with community.hashi_vault.vault_kv1_get produces error of "msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.vault_kv1_get'. Error was a <class 'hvac.exceptions.UnexpectedError'>, original message: ['required index state not present']

ISSUE TYPE

• Bug Report

COMPONENT NAME

vault_kv1_get

ANSIBLE VERSION

2.11

##### COLLECTION VERSION
<!--- Paste verbatim output from "ansible-galaxy collection list <namespace>.<collection>"  between the quotes
for example: ansible-galaxy collection list community.general
-->
4.1.0

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

  tasks:

    - name: Provide Appdynamics
      vars:
        ansible_hashi_vault_auth_method: approle
        ansible_hashi_vault_role_id: some_role
        ansible_hashi_vault_secret_id: some_secret
        ansible_hashi_vault_engine_mount_point: mount_point
      ansible.builtin.debug:
        msg: "{{ lookup('community.hashi_vault.vault_kv1_get', 'path/to/secret').secret.somesecret}}"

EXPECTED RESULTS

secret should be returned

ACTUAL RESULTS

{
  "msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.vault_kv1_get'. Error was a <class 'hvac.exceptions.UnexpectedError'>, original message: ['required index state not present'], on get https://vault:secret",
  "_ansible_no_log": false
}

{
  "msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.vault_kv1_get'. Error was a <class 'hvac.exceptions.UnexpectedError'>, original message: ['required index state not present'], on get https://vault:secret",
  "_ansible_no_log": false
}

[briantist]

Hi @mjnorman ,thanks for opening this.

Based on the exception (unexpected error), this means the HTTP response code did not fit into any of the ones we would expect to receive from Vault (this is handled in the hvac library: https://hvac.readthedocs.io/en/stable/_modules/hvac/exceptions.html#hvac.exceptions.VaultError.from_status).

As a result, I suspect that this has something to do with your particular Vault instance, in the parameters being set, or possibly a bug in hvac.

I think it's best to first try and figure out what HTTP status code this request is returning.

Are you able to re-run your repro with -vvv and see if we can get an expanded error message out of it? If not, can the error be reproduced either with the Vault CLI or with hvac directly?

---

Couple of other minor things to check since your example code is redacted:

• path/to/secret should be relative to the mount point, not include it
• your error message shows the Vault address as https://vault:secret
  • I don't see the address set in Ansible vars nor in the lookup call, so I suspect it's set via environment variable?
  • I know you have redacted the address, but the existence of a colon : makes me wonder if that was part of the original (the only colon should be after https)

[mjnorman]

I did a ton of reading up on this. The eventual cause appears to be how vault uses active/standby nodes, and the potential for a read to hit a node that does not contain the correct state. The correct fix here would be to have hvac include the below headers which are sent on replies from Vault (https://developer.hashicorp.com/vault/docs/enterprise/consistency#vault-1-7-mitigations):

X-Vault-Index: <base64 value taken from previous response>
X-Vault-Inconsistent: forward-active-node

The workaround for me was to just utilize the retries option to allow time for the nodes to sync. It usually only takes 1 retry in my environment, but others may see more.

[briantist]

Excellent, I was going to recommend the retries (I recommend them for everyone in fact!), if you haven't seen it, we have a section in our user guide that goes into detail about it.

This is also the method we recommend in hvac for now: https://hvac.readthedocs.io/en/stable/advanced_usage.html#retrying-failed-requests

But if you'd like to open an issue or discussion on the hvac repo about implementing the headers in that library, we can discuss it there. I think at a minimum hvac could do with a new exception type that corresponds to that status code so that we can return a better error.

I'll close this issue out here since anything further to address this should be implemented upstream I think. Thanks!