dict2items over empty object fails. Even with default({}) safeguard

[netikras]

Summary

basically, `{} | dict2items' fails as it has nothing to iterate over -- dict2items requires a dictionary, got <class 'NoneType'> instead.

I have a playbook that may or may not have key-value pairs in an object (k8s resource labels). And I need to iterate over them if they are present. Problem is, dict2items doesn't like objects that have no keys.

Issue Type

Bug Report

Component Name

dict2items

Ansible Version

$ ansible --version
ansible [core 2.12.10]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/dariusj/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/dariusj/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]
  jinja version = 2.10.1
  libyaml = True

Configuration

# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all

BECOME:
======

CACHE:
=====

CALLBACK:
========

CLICONF:
=======

CONNECTION:
==========

HTTPAPI:
=======

INVENTORY:
=========

LOOKUP:
======

NETCONF:
=======

SHELL:
=====

VARS:
====

OS / Environment

LinuxMint 20.3

Steps to Reproduce

This does not work

- name: Dump example
  vars:
    data:
      pairs:
  debug:
    var: item
  loop: '{{ data.pairs | default({}) | dict2items }}'

This does not work either

- name: Dump example
  vars:
    data:
  debug:
    var: item
  loop: '{{ data.pairs | default({}) | dict2items }}'

Neither does that

- name: Dump example
  vars:
    data:
      pairs: {}
  debug:
    var: item
  loop: '{{ data.pairs | default({}) | dict2items }}'

This works, but it's not what is needed

- name: Dump example
  vars:
    data:
      pairs:
        a: b
  debug:
    var: item
  loop: '{{ data.pairs | default({}) | dict2items }}'

Expected Results

Task successfully completes w/o any iterations done.

Actual Results

fatal: [vm-rocky]: FAILED! => {
    "msg": "Unexpected templating type error occurred on ({{ data.pairs | default({}) | dict2items }}): dict2items requires a dictionary, got <class 'NoneType'> instead."
}

And I'm assuming empty objects should work, because of this example

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description: None

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[mkrizek]

- name: Dump example
  vars:
    data:
      pairs:
  debug:
    var: item
  loop: '{{ data.pairs | default({}) | dict2items }}'

In this example data.pairs is effectively ~ so null or None so the error dict2items requires a dictionary, got <class 'NoneType'> instead is expected.

This does not work either

- name: Dump example
  vars:
    data:
  debug:
    var: item
  loop: '{{ data.pairs | default({}) | dict2items }}'

This should trigger the default filter as pair key is undefined, and the task should be skipped because there is an empty list to loop over.

Neither does that

- name: Dump example
  vars:
    data:
      pairs: {}
  debug:
    var: item
  loop: '{{ data.pairs | default({}) | dict2items }}'

data.pairs is an empty dictionary so the task should be skipped because there is an empty list to loop over.

Testing on the latest version of ansible-core:

ansible [core 2.14.1] (detached HEAD ed924d6493) last updated 2023/01/27 18:41:59 (GMT +200)
  config file = /Users/mkrizek/src/ansible/ansible.cfg
  configured module search path = ['/Users/mkrizek/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/mkrizek/src/ansible/lib/ansible
  ansible collection location = /Users/mkrizek/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/mkrizek/.virtualenvs/ansible-py311/bin/ansible
  python version = 3.11.0 (main, Oct 25 2022, 09:48:29) [Clang 14.0.0 (clang-1400.0.29.102)] (/Users/mkrizek/.virtualenvs/ansible-py311/bin/python)
  jinja version = 3.1.2
  libyaml = True

the playbook:

- hosts: localhost                                                              
  gather_facts: false
  tasks:
    - name: Dump example
      vars:
        data:
          pairs:
      debug:
        var: item
      loop: '{{ data.pairs | default({}) | dict2items }}'
      ignore_errors: yes

    - name: Dump example
      vars:
        data:
      debug:
        var: item
      loop: '{{ data.pairs | default({}) | dict2items }}'

    - name: Dump example
      vars:
        data:
          pairs: {}
      debug:
        var: item
      loop: '{{ data.pairs | default({}) | dict2items }}'

I get expected results:

PLAY [localhost] ********************************************************************************************************************

TASK [Dump example] *****************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ data.pairs | default({}) | dict2items }}): dict2items requires a dictionary, got <class 'NoneType'> instead.. dict2items requires a dictionary, got <class 'NoneType'> instead."}
...ignoring

TASK [Dump example] *****************************************************************************************************************
skipping: [localhost]

TASK [Dump example] *****************************************************************************************************************
skipping: [localhost]

PLAY RECAP **************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=1

The same on ansible-core v2.13.7 (although skipping: [localhost] isn't printed on the screen due to a bug in that version). I have not tested on 2.12 since it only accepts security fixes at this point.

[mkrizek]

Also, you can use builtin undef function as well:

  - vars:                                                                     
      data:
        pairs: "{{ undef() }}"
    debug:
      var: item
    loop: '{{ data.pairs | default({}) | dict2items }}'

[netikras]

@mkrizek Thank you for your answer. I was not aware of the '{undef()}' workaround. However, I have questions

In this example data.pairs is effectively ~ so null or None so the error dict2items requires a dictionary, got <class 'NoneType'> instead is expected.

Which value is None? data.pairs? or data? If it is None, then why doesn't the default({}) filter kick in and default this value to a non-None? It does that when I omit the whole definition of data.

This filter kicks in if in the vars: section I omit the definition of data or data.pairs, which makes them None in a clear and obvious way:

- hosts: localhost
  tasks:
    - name: Dump example
      vars:
      debug:
        var: item
      loop: '{{ data.pairs | default({}) | dict2items }}'

- hosts: localhost
  tasks:
    - name: Dump example
      vars:
        data:
      debug:
        var: item
      loop: '{{ data.pairs | default({}) | dict2items }}'

In both cases, I get a successful execution. Yet, in both the cases the data.pairs was effectively None

TASK [Dump example] ************************
task path: [HIDDEN]/ansible/setup_k8s.yml:10
META: ran handlers
META: ran handlers

So my question is: if I do not define the variable (and it is None) - the default() filter is triggered. If I define the variable but assign no value to it, according to you it is also None, but the default() filter is NOT triggered. Why is that? How is None != None?

In this example data.pairs is effectively ~ so null or None so the error dict2items requires a dictionary, got <class 'NoneType'> instead is expected.

If this is expected, is it mentioned anywhere in the docs? I was looking for it but could not find anything of sorts. Unless this behaviour is documented, I struggle to see how it's expected.

[s-hertel]

Which value is None? data.pairs? or data?

data is a dictionary that contains the key pairs. data.pairs is None.

  vars:
    data:
      pairs:

If it is None, then why doesn't the default({}) filter kick in ...

default applies to undefined variables by default, which is different than None or an empty string/list/dictionary. You can set the second parameter to True to also apply defaults to falsey values https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.default.

If you want to use default with variables that evaluate to false you have to set the second parameter to true:

{{ ''|default('the string was empty', true) }}

[netikras]

@s-hertel @mkrizek Thank you, good souls! That was the missing piece. Now I understand I was missing the concept of undefined. Thank you for the link too, it helped me a lot.

Implementation For the sake of convenience, foo.bar in Jinja does the following things on the Python layer:

check for an attribute called bar on foo (getattr(foo, 'bar'))

if there is not, check for an item 'bar' in foo (foo.getitem('bar'))

if there is not, return an undefined object.

While I understand now what my mistake was and that it's just how jinja2 works, I can't help but argue that 'undefined' might be a semantically incorrect term, as the variable/yaml path is, in fact, defined (that's what I'm doing by writing its path/key), it's just not assigned any value. Intuitively, I'd expect it to be undefined if hasattr(foo, 'bar') returned a falsy answer.

Anyway, my ramblings driven by failed intuition and bruised dev-confidence won't change anything :)

[s-hertel]

Happy to help!

Intuitively, I'd expect it to be undefined if hasattr(foo, 'bar') returned a falsy answer.

Your intuition seems right on. If you load the YAML, it's more obvious the attr exists as None.

>>> import yaml
>>> some_yaml = """\
... vars:
...   data:
...     pairs:
... """
>>> yaml.safe_load(some_yaml)
{'vars': {'data': {'pairs': None}}}

It's the difference between:

>>> class bar:
...     foo = None  # foo is defined
... 
>>> hasattr(bar, 'foo')
True

and

>>> class bar:
...     pass  # foo is undefined
... 
>>> hasattr(bar, 'foo')
False

(Whereas getattr(bar, 'foo', None) would be falsey for both)

[netikras]

Yepp, it was a Friday's HTTP:503 of my concentration :)

I also asked ChatGPT to solve this issue for me, If anyone is interested: https://dev.to/netikras/chatgpt-problem-solving-skills-ansible-yaml-undefined-vs-none-bic