vault_kv2_get lookup plugin variables interpolation

SDedik commented 6 months ago

SUMMARY

The vault_kv2_get lookup plugin does not interpolate variables set on the task level. Instead it takes them literally.

ISSUE TYPE

Bug Report

COMPONENT NAME

vault_kv2_get lookup plugin

ANSIBLE VERSION

ansible [core 2.15.10]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True

COLLECTION VERSION

community.hashi_vault:6.2.0

STEPS TO REPRODUCE

Consider following playbook:

- hosts: myhost
  tasks:
    - name: vault test
      vars:
        ansible_hashi_vault_url: "{{ global_hashicorp_vault_url }}"
        ansible_hashi_vault_auth_method: "{{ global_hashicorp_vault_auth_method }}"
        ansible_hashi_vault_role_id: "{{ global_hashicorp_vault_role_id }}"
        ansible_hashi_vault_secret_id: "{{ global_hashicorp_vault_secret_id }}"
      ansible.builtin.set_fact:
        response: "{{ lookup('community.hashi_vault.vault_kv2_get', 'test_secret') }}"

The global_hashicorpvault* variables are defined somewhere else and do have actual and valid values to connect, authenticate and perform a lookup against a vault instance. However, this step fails with the error: Invalid value \"{{ global_hashicorp_vault_auth_method }}\" for configuration option \"plugin_type: lookup plugin: ansible_collections.community.hashi_vault.plugins.lookup.vault_kv2_get setting: auth_method \", valid values are: token, userpass, ldap, approle, aws_iam, azure, jwt, cert, none"

However, if I split variables assignment and lookup action into two tasks it works as expected:

    - name: set hashi_vault parameters
      ansible.builtin.set_fact:
        ansible_hashi_vault_url: "{{ global_hashicorp_vault_url }}"
        ansible_hashi_vault_auth_method: "{{ global_hashicorp_vault_auth_method }}"
        ansible_hashi_vault_role_id: "{{ global_hashicorp_vault_role_id }}"
        ansible_hashi_vault_secret_id: "{{ global_hashicorp_vault_secret_id }}"

    - name: vault test
      ansible.builtin.set_fact:
        response: "{{ lookup('community.hashi_vault.vault_kv2_get', 'test_secret') }}"

EXPECTED RESULTS

The expected result is that ansible variables recognized by the vault_kv2_get lookup plugin and set on the task level are interpolated correctly and resolved values are passed to the plugin thus allowing to define lookup parameters and perform a lookup in a single task.

briantist commented 6 months ago

Hi @SDedik , this interpolation is (supposed to be) handled by ansible itself and affects all lookup plugins, so we cannot really fix it in the collection, see also:

We would also very much like to see that fixed.

SDedik commented 6 months ago

Hello, @briantist, thank you for your response and clarification. Apologies that I decided that interpolation must be performed by the lookup plugin, while researching the issue I was confused by this old post here: https://github.com/ansible/ansible/issues/33738 where they state that Any variables used within the lookup plugin will need to be templated by that plugin (see https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/lookup/template.py as an example). Being that old that post is probably no longer relevant/correct. Thank you anyways!

briantist commented 6 months ago

@SDedik yeah I have also seen that explanation (not in that post but in others). It makes sense for lookups that are accessing variables directly, but imo it's unintuitive behavior for vars that correspond to defined parameters of the lookup. Config manager handles the var->option mapping, and if you pass the option directly to the lookup, the lookup receives the templated value, not the raw template.

vars:
  value: '{{ 1 + 1 }}'
  L: "{{ lookup('plugin', option=value) }}"

But when you map a var to option suddenly the templating doesn't happen. I think there are technical reasons why it doesn't (because these variable->option mappings were designed originally for plugins that are loaded implicitly and early when maybe variables can't be templated yet or not all values are loaded?), but I still think it's very unexpected for it to work like this in lookups especially.

So it's not quite true for me to say that we can't fix it in the collection: we could run our values through self._templar.template(), but it starts to get very tricky:

how do we know if the value is a template or not?
- we cannot tell whether the source of an option value was direct (already templated), via vars (not templated), env var, ini file, etc., without calling an internal API that re-scans all the sources, because even though that source data is considered internally, it is not preserved and passed to the plugin
- we can add an option or (options) that the caller has to specify to say whether or not option values should be run through the templar, but it's not a good UX and it's tough to maintain and test
if core does change it to make it consistent, any workaround changes we make might be broken completely, and we have to figure out how to rollback or workaround those changes with the least harm and breakage

So right now the options mainly are:

define your vars (with templating) and then pass them in as direct options
use set_fact (which can be problematic due to precedence)

You can make the vars->direct method a little more palatable if you always use the same set of options, by using **kwargs passing:

vars:
  opts:
    url: 'https://{{ my_vault }}:8200'
    auth_method: token
    engine_mount_point: '{{ engine }}'
  get_secret: "{{ lookup('community.hashi_vault.vault_kv2_get', path, **opts) }}"

tasks:
  - name: look secrets
    vars:
      engine: this_engine
      path: '{{ item }}'
    loop:
      - path1
      - path2
      - path3
    ansible.builtin.debug:
      msg: 
        - '{{ get_secret }}'
        - "use lookup, pass common opts + additional: {{ lookup('community.hashi_vault.vault_kv2_get', path, token_file='.vault-agent-token', **opts) }}"

Since the templating is still lazy, it works even in loops or with different "sub" values changed in different scopes. But this is only going to be useful in certain contexts where the upfront cost of defining the right variable names and/or structures is worth it.

SDedik commented 6 months ago

Wow, what a great answer! Thank you so much for the detailed explanation, much obliged :)

ansible-collections / community.hashi_vault