Jinja expressions are not evaluated in (not only) credential variables

[mordekasg]

OS / Environment

Debian 12

Ansible version

ansible [core 2.14.17]
  config file = /home/test/host/ansible/ansible.cfg
  configured module search path = ['/home/test/host/ansible/local_modules']
  ansible python module location = /home/test/env.ansible/lib/python3.11/site-packages/ansible
  ansible collection location = /home/test/host/ansible/collections:/usr/share/ansible/collections
  executable location = /home/test/env.ansible/bin/ansible
  python version = 3.11.2 (main, May  2 2024, 11:59:08) [GCC 12.2.0] (/home/test/env.ansible/bin/python)
  jinja version = 3.1.4
  libyaml = True

Environment variables for strategy plugin

ANSIBLE_STRATEGY=mitogen_linear
ANSIBLE_STRATEGY_PLUGINS=/home/test/env.ansible/src/mitogen/ansible_mitogen/plugins/strategy

Mitogen version

v0.3.12

Problem

Jinja expressions are not evaluated by mitogen plugin

How to reproduce

Description

Ansible host: srv1 has credentials configured using Jinja expressions from vault/hostvars variables. Ansible host: srv2 has credentials configured directly as host variables.

Vault file

Path: vault.test.yml

ANSIBLE_LOCALHOST_SRVS_USER: "testusr"
ANSIBLE_LOCALHOST_SRVS_PASSWORD: "okmqwe123"
ANSIBLE_LOCALHOST_SRVS_BECOME_USER: "root"
ANSIBLE_LOCALHOST_SRVS_BECOME_PASSWORD: "okmqwe123"

Inventory file

Path: inventory/local/local.yml

all:
  children:
    test_srvs:
      hosts:
        srv1:
          host_ip: 127.0.0.1
          ansible_host: "{{ host_ip }}"
          ansible_user: "{{ ANSIBLE_LOCALHOST_SRVS_USER }}"
          ansible_password: "{{ ANSIBLE_LOCALHOST_SRVS_PASSWORD }}"
          ansible_become_user: "{{ ANSIBLE_LOCALHOST_SRVS_BECOME_USER }}"
          ansible_become_password: "{{ ANSIBLE_LOCALHOST_SRVS_BECOME_PASSWORD }}"
          ansible_ssh_user: "{{ ansible_user }}"
          ansible_ssh_password: "{{ ansible_password }}"
        srv2:
          ansible_host: 127.0.0.1
          ansible_user: testusr
          ansible_password: okmqwe123
          ansible_become_user: root
          ansible_become_password: okmqwe123
          ansible_ssh_user: testusr
          ansible_ssh_password: okmqwe123

Commands

regular user

ansible -e @vault.test.yml -i inventory/local/local.yml -m ping test_srvs

superuser - become

ansible -b -e @vault.test.yml -i inventory/local/local.yml -m ping test_srvs

Output

regular user

srv1 | UNREACHABLE! => {
    "changed": false,
    "msg": "EOF on stream; last 100 lines received:\nhostname contains invalid characters\r",
    "unreachable": true
}
srv2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

superuser - become

srv1 | UNREACHABLE! => {
    "changed": false,
    "msg": "EOF on stream; last 100 lines received:\nhostname contains invalid characters\r",
    "unreachable": true
}
srv2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Commands with verbose

regular user

ansible -e @vault.test.yml -i inventory/local/local.yml -m ping srv1 -vvvv

superuser - become

ansible -b -e @vault.test.yml -i inventory/local/local.yml -m ping srv1 -vvvv

Output

[mux 871642] 20:15:32.461360 D mitogen.parent: command line for Connection(None): ssh -o "LogLevel ERROR" -l "{{ ansible_user }}" -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -o "GlobalKnownHostsFile /dev/null" -C -o ControlMaster=no -o ControlPersist=60s "{{ host_ip }}" /usr/bin/python3 [...]

Expected result

srv1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
srv2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Source of the problem

Ansible: https://github.com/ansible/ansible/commit/6d2d476113b3a26e46c9917e213f09494fbc0a13

Similar issues

Ansible

Paramiko plugin: https://github.com/ansible/ansible/issues/78509 Ansible: https://github.com/ansible/ansible/pull/76590

Mitogen

905

978

1022

1040

1079

1116

[mordekasg]

Tested on latest release: 0.3.9. Result is still the same.

[golebiewsky]

I have the same issue 👍

[moreati]

With PR #1114 merged into master the Ansible connection password (e.g. ansible_password, ansible_ssh_pass) is templated. Now that I've found a reasonable way to do this, others should follow.

[moreati]

Ansible connection password (e.g. ansible_password, ansible_ssh_pass) is templated

Now released in 0.3.12

[mordekasg]

Ansible connection password (e.g. ansible_password, ansible_ssh_pass) is templated

Now released in 0.3.12

That`s great news!

I have tested on 0.3.12 and templated ansible_password, ansible_ssh_pass are working.

According to this issue: #1116, after new version release, also templated ansible_user, ansible_ssh_user will be working.

I have added PR: #1148, which will allow to template ansible_become_password attribute.

After those 2 PR`s released, feel free to close this issue!

[marbud0]

That`s great news!

I have tested on 0.3.12 and templated ansible_password, ansible_ssh_pass are working.

According to this issue: #1116, after new version release, also templated ansible_user, ansible_ssh_user will be working.

I have added PR: #1148, which will allow to template ansible_become_password attribute.

After those 2 PR`s released, feel free to close this issue!

This problem also affects the ansible_host variable.. It's kind of hinted to in the subject, but not showed in the example.. Can we use this issue to cover this or should I file a new issue?

[mordekasg]

This problem also affects the ansible_host variable.. It's kind of hinted to in the subject, but not showed in the example.. Can we use this issue to cover this or should I file a new issue?

I updated my post and extended example to some extent.

It`s great idea to use this issue to cover also ansible_host variable.

[moreati]

Mitogen 0.3.14 is out. It includes support for templated ansible_become_user, `become_user, etc. Thanks to @mordekasg.

[moreati]

Mitogen 0.3.15 is out. It supports templated SSH arguments and become password.

[moreati]

Mitogen 0.3.16 is out with support for templating the Ansible SSH command (e.g. ansible_ssh_executable), the become command (e.g. ansible_become_exe), and the become command arguments (e.g. ansible_become_flags)

[moreati]

I think that leaves

Become plugins (e.g. sudo)

• [x] become
• [x] become_method

Connection plugins (e.g. ssh)

• [x] host
• [x] host_key_checking
• [x] private_key_file
• [ ] python_path
• [ ] timeout

[moreati]

Mitogen v0.3.17 is out. Templated ssh key, templated host key checking, and templated host address.

[moreati]

Bonus release: 0.3.18 with support for templating the become flag (ansible_become) and the become method (ansible_become_method).

[moreati]

I'll probably target ansible_interpreter_python next. It will probably take longer, a week or two.