Hashi_vault lookup issues reading data from Kv2 when on ansible version 2.9

[sangeethdba]

SUMMARY

fatal: [xxx.xxx.com]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: The secret secret/data/dbre/elasticsearch/ESBOO/kibanapwd does not contain the field 'content'. for hashi_vault lookup"}

im getting this error when ansible version is 2.9.. the same works when the ansible version sis 2.10.

any way to get hashi_vault lookup work with ansible 2.9 version

here is my lookup command kibana_password: "{{ lookup('hashi_vault', 'secret=secret/data/dbre/elasticsearch/{{ clustername }}/kibanapwd:content auth_method=ldap username={{ vault_user }} password={{ vault_pass }} url={{ vault_url }} validate_certs=True cacert=/cacert/capath/ca.pem') }}"

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault ansible 2.9 valut kv2

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[briantist]

Hi @sangeethdba ! The version of the lookup included in Ansible 2.9 is quite old. When you install 2.10 it contains an updated version of the plugin.

Since most Ansible content including this is now in separate collections, you no longer have to update your core Ansible version to use later content.

This collection is tested against Ansible 2.9 and is supported there.

I recommend that you install the latest collection version alongside 2.9, and update your Ansible configuration to ensure that the collection is found.

Here is the collections guide for 2.9: https://docs.ansible.com/ansible/2.9/user_guide/collections_using.html

And here is the galaxy entry for this collection: https://galaxy.ansible.com/community/hashi_vault

I also recommend checking out the changelog: https://github.com/ansible-collections/community.hashi_vault/blob/main/CHANGELOG.rst

There are some very small breaking changes between the version that was in community.general but they won't affect most use cases. I expect the latest collection ought to work well for you in 2.9.

[briantist]

I might have an idea of what is causing the issue in the 2.9 version, but I can't guarantee supporting it. If possible, post the output from your lookup, but remove the field selector, that is remove :content and post the full result (of course, you can redact any secret information, I'm looking for the data structure of what's returned).

[sangeethdba]

I have awx version 16 ,im facing this issue when i configure my playbok as a templete in tower. if i run the same playbook manually from control server which has ansible version 16 .,lookup with :content is working fine.

[briantist]

@sangeethdba unfortunately I don't have much familiarity with AWX/tower, so the versions don't have much meaning for me. If you tell me the data structure returned as described in my second comment, I may be able to provide a 2.9 workaround.

[sangeethdba]

TASK [elasticsearch : Retrieve secrets from Vault as per Cluster name] ***** fatal: []: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: The secret secret/data/comdbre/elasticsearch/ESBOO/kibanapwd does not contain the field 'content'. for hashi_vault lookup"}

same error from tower template ...with :content and without content. not sure if I'm missing something.

[briantist]

when you remove :content it still gives the error does not contain the field 'content'? That doesn't sound right...

[sangeethdba]

Yes thats true ..

• name: Retrieve secrets from Vault as per Cluster name set_fact: kibana_password: "{{ lookup('hashi_vault', 'secret=secret/data/dbre/elasticsearch/{{ clustername }}/kibanapwd auth_method=ldap username={{ vault_user }} password={{ vault_pass }} url={{ vault_url }} validate_certs=True cacert=/cacert/capath/ca.pem') }}"

• debug: var: kibana_password

and the error:

TASK [elasticsearch : Retrieve secrets from Vault as per Cluster name] ***** fatal: []: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: The secret secret/data/dbre/elasticsearch/ESBOO/kibanapwd does not contain the field 'content'. for hashi_vault lookup"}

[briantist]

I'm afraid I can't explain that; there must be some mistake in what you're executing. You can see where that error is generated in 2.9: https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/plugins/lookup/hashi_vault.py#L203-L207

self.secret_field can only be populated when the secret contains a colon :, so I think something is not persisting in your execution environment.

---

The root of what I suspect the problem is, is that in 2.9 the plugin did not do any special handling for the data structure returned from kv2, so you can't use : to access your field correctly from a kv v2 result in that version of the plugin.

You should be able to do something like this:

kibana_password: "{{ lookup('hashi_vault', 'secret=secret/data/dbre/elasticsearch/{{ clustername }}/kibanapwd auth_method=ldap username={{ vault_user }} password={{ vault_pass }} url={{ vault_url }} validate_certs=True cacert=/cacert/capath/ca.pem')['data']['content'] }}"

But I was asking for the output in order to see the structure definitively.

[sangeethdba]

i take my word back ..

without :content is working and here is the structure.

TASK [elasticsearch : debug] ***************************************************
ok: [XXXXX] => {
    "kibana_password": {
        "data": {
            "content": "XXXXXXXX"
        },
        "metadata": {
            "created_time": "2021-07-08T19:00:59.270370379Z",
            "deletion_time": "",
            "destroyed": false,
            "version": 1
        }
    }
}

[briantist]

perfect! with this I think can say that the workaround I posted above will work:

kibana_password: "{{ lookup('hashi_vault', 'secret=secret/data/dbre/elasticsearch/{{ clustername }}/kibanapwd auth_method=ldap username={{ vault_user }} password={{ vault_pass }} url={{ vault_url }} validate_certs=True cacert=/cacert/capath/ca.pem')['data']['content'] }}"

The important part is at the end: ['data']['content']

Since the result of the lookup is a dictionay, you can reference into the dictionary to get the fields you want.

[sangeethdba]

hope adding ['data']['content']

will work with 2.10 as well

[briantist]

It will not unfortunately, because there was a breaking change between 2.9 and 2.10, where the plugin was modified to take into account kv2 secrets. If you need to support both simultaneously, the best option is to embrace collections, and install a newer collection version.

Otherwise, you can do your own post processing:

- set_fact:
    kibana_password_raw: "{{ lookup('hashi_vault', 'secret=secret/data/dbre/elasticsearch/{{ clustername }}/kibanapwd auth_method=ldap username={{ vault_user }} password={{ vault_pass }} url={{ vault_url }} validate_certs=True cacert=/cacert/capath/ca.pem') }}"

- set_fact:
    kibana_password: "{{
      kibana_password_raw['data']['content'] if 'data' in kibana_password_raw else kibana_password_raw['content']
    }}"

(or you can come up with your own way to do so, like writing a filter plugin 😬)

[sangeethdba]

Awesome thanks for the right pointers.

[sangeethdba]

community.hashi_vault.hashi_vault

kibana_password_raw: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/data/dbre/elasticsearch/{{ clustername }}/kibanapwd:content auth_method=ldap username={{ vault_user }} password={{ vault_pass }} url={{ vault_url }} validate_certs=True cacert=/cacert/capath/ca.pem') }}"

will this work with both 2.9 and 2.10? if i install the latest collection?

[briantist]

Yes, I believe it will!

You'll also have newer features like being able to pass real parameters (not just putting them into the term string with the secret), new environment variables, ansible vars, controlling timeout, supporting retries, and several bugfixes too.

[briantist]

@sangeethdba were you able to use a newer collection locally?

[sangeethdba]

yes

On Tue, Jul 20, 2021 at 4:36 PM Brian Scholer @.***> wrote:

@sangeethdba https://github.com/sangeethdba were you able to use a newer collection locally?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ansible-collections/community.hashi_vault/issues/109#issuecomment-883689580, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5ROKKNWAGG72PR4IURWPTTYXM53ANCNFSM5AUIKCCA .

[briantist]

great news! I hope that worked well for you. I'll close this out for the time being, but feel free to comment or open a new issue if needed.

[sangeethdba]

Thanks appreciate your help

On Tue, Jul 20, 2021 at 4:43 PM Brian Scholer @.***> wrote:

great news! I hope that worked well for you. I'll close this out for the time being, but feel free to comment or open a new issue if needed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ansible-collections/community.hashi_vault/issues/109#issuecomment-883695676, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH5ROKOPUWWYCLTYT76I4UDTYXNXNANCNFSM5AUIKCCA .