Using arubanetworks.aoscx.aoscx_connection type makes any play to throw an exception at startup

[ppetit]

Hi there!

Unfortunately, aoscx-ansible-collection doesn't work for me when using arubanetworks.aoscx.aoscx connection type. It does work when using network_cli connection type though.

Simple playbook:

---
- name: Backup switch configs
  hosts: aoscx_switches
  collections:
    - arubanetworks.aoscx
  gather_facts: False
  vars:
    ansible_network_os: arubanetworks.aoscx.aoscx
    ansible_connection: arubanetworks.aoscx.aoscx
    ansible_aoscx_validate_certs: False
    ansible_aoscx_use_proxy: False
    ansible_httpapi_validate_certs: False
    ansible_httpapi_use_ssl: True
    ansible_acx_no_proxy: True
    ansible_python_interpreter: /usr/bin/python3

  tasks:
    - name: Backup running config to local JSON file
      aoscx_backup_config:
        config_name: running-config
        output_file: configs/{{ inventory_hostname }}-running-config.json

But really it doesn't matter what you have in the playbook. The simple fact of declaring ansible_connection: arubanetworks.aoscx.aoscx makes any play to throw an exception at startup like so.

$ ansible-playbook ibgp_evpn_vxlan_backup.yml -i vars/commons/inventories/attlab_at_gre/hosts -vvv
ansible-playbook [core 2.13.4]
  config file = /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/ansible.cfg
  configured module search path = ['/home/ppetit/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ppetit/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/ppetit/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ppetit/.local/bin/ansible-playbook
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True
Using /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/ansible.cfg as config file
host_list declined parsing /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/vars/commons/inventories/attlab_at_gre/hosts as it did not pass its verify_file() method
script declined parsing /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/vars/commons/inventories/attlab_at_gre/hosts as it did not pass its verify_file() method
auto declined parsing /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/vars/commons/inventories/attlab_at_gre/hosts as it did not pass its verify_file() method
Parsed /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/vars/commons/inventories/attlab_at_gre/hosts inventory source with ini plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: ibgp_evpn_vxlan_backup.yml *************************************************************************************************
1 plays in ibgp_evpn_vxlan_backup.yml

PLAY [Backup switch configs] *********************************************************************************************************
META: ran handlers

TASK [debug] *************************************************************************************************************************
task path: /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/ibgp_evpn_vxlan_backup.yml:18
The full traceback is:
Traceback (most recent call last):
  File "/home/ppetit/.local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 293, in __getattr__
    return self.__dict__[name]
KeyError: '_nonetype'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ppetit/.local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 1009, in _set_plugin_options
    plugin = getattr(self._connection, '_%s' % plugin_type)
  File "/home/ppetit/.local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 301, in __getattr__
    raise AttributeError("'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
AttributeError: 'Connection' object has no attribute '_nonetype'

$ python3 --version
Python 3.8.10

$ ansible --version
ansible [core 2.13.4]
  config file = /home/ppetit/Workspaces/CMS-Shared-Infra-Aruba/ansible.cfg
  configured module search path = ['/home/ppetit/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ppetit/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/ppetit/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ppetit/.local/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

$ ansible-galaxy collection list

# /home/ppetit/.ansible/collections/ansible_collections
Collection          Version
------------------- -------
ansible.netcommon   3.1.1  
ansible.utils       2.6.1  
arubanetworks.aoscx 4.1.0  

# /home/ppetit/.local/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    3.4.0  
ansible.netcommon             3.1.0  
ansible.posix                 1.4.0  
ansible.utils                 2.6.1  
ansible.windows               1.11.0 
[...]

$ pip list -V | grep aoscx
pyaoscx                       2.3.0

I followed scrupulously the installation process. Thanks in advance for taking a look at this issue.

[tchiapuziowong]

@ppetit can you remove the variable ansible_python_interpreter from your playbook? Also I'm assuming you have an inventory file that defines your host information?

[ppetit]

@tchiapuziowong Removed ansible_python_interpreter but it changes nothing and yes I have a valid inventory which works fine with network_cli BTW.

[tchiapuziowong]

Is it the same above playbook with aoscx_backup_config that works or one using aoscx_config or aoscx_command? Can you execute which python and share the results?

[ppetit]

@tchiapuziowong one using aoscx_command does work

- name: Apply AOS-CX base config
  hosts: aoscx_switches
  collections:
    - arubanetworks.aoscx
  vars:
    ansible_connection: network_cli
    ansible_python_interpreter: /usr/bin/python3

  gather_facts: False

  tasks:
    - name: Generate Template for Device
      template:
        src: "{{ templates_path | default('./') }}iBGP/{{ config_template }}"
        dest: "{{config_path}}{{inventory_hostname}}.conf"
        mode: 0777
      tags:
        - underlay
        - base-config

    - name: Copy generated config to device via SSH
      aoscx_config:
        src: "{{config_path}}{{hostname}}.conf"
        save_when: changed
        backup: True
      tags:
        - underlay
        - base-config

Those based on Rest API modules like below do not work.

- name: Backup switch configs
  hosts: aoscx_switches
  collections:
    - arubanetworks.aoscx
  gather_facts: False
  vars:
    ansible_network_os: arubanetworks.aoscx.aoscx
    ansible_connection: arubanetworks.aoscx.aoscx
    ansible_aoscx_validate_certs: False
    ansible_aoscx_use_proxy: False
    ansible_httpapi_validate_certs: False
    ansible_httpapi_use_ssl: True
    ansible_acx_no_proxy: True
    ansible_python_interpreter: /usr/bin/python3

$ /usr/bin/python3 -V
Python 3.8.10
$ which python
/usr/bin/python
$ /usr/bin/python -V
Python 2.7.18

[jimdavis66]

I am experiencing this exact same issue. network_cli is still working, but I get the same traceback and error when using ansible_connection: arubanetworks.aoscx.aoscx

[tchiapuziowong]

@jimdavis66 Are you using the same Ansible & Python version? Can you share how you installed Ansible?

[tchiapuziowong]

@jimdavis66 Can you also share your operating system?

[jimdavis66]

/usr/bin/python3 -V Python 3.10.6

ansible --version ansible [core 2.13.4] config file = /home/ubuntu/arubaosx/ansible.cfg configured module search path = ['/home/ubuntu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3/dist-packages/ansible ansible collection location = /home/ubuntu/.ansible/collections:/usr/share/ansible/collections executable location = /usr/bin/ansible python version = 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] jinja version = 3.0.3 libyaml = True

/home/ubuntu/.ansible/collections/ansible_collections Collection Version

---

ansible.netcommon 3.1.1 ansible.utils 2.6.1 arubanetworks.aoscx 4.1.0

I install ansible with homebrew on macOS, and with https://github.com/geerlingguy/ansible-role-ansible on Ubuntu. I'm experiencing the problem on both operating systems.

Thanks

[tchiapuziowong]

Thank you both for bringing this to our attention! We're investigating the issue internally and we'll update this issue with any findings/commits with fixes - thank you for your patience!

For others running into this issue - if you could provide versioning/env details that may help us in investigating

[Harsbo]

ubuntu 20.04 with ansible installed through pip3.

ansible [core 2.13.4] python version = 3.8.10 arubanetworks.aoscx 4.1.0

[KevGue]

Updated all my packages, collections, roles today and am also experience this issue with the REST API (SSH works).

Versions: OS: Ubuntu 22.04.1 LTS Ansible: 2.13.5rc1 Python: 3.10.6 ArubaOSCX Collection: 4.1.0

[borgermeister]

Also experiencing the same. Here are the steps to reproduce.

$ python3 -m venv ~/ansible6.5
$ source ~/ansible6.5/bin/activate
$ pip install --upgrade pip setuptools wheel
$ pip install ansible
$ pip install pyaoscx
$ ansible-galaxy collection install arubanetworks.aoscx

Variable file with connection info:

---
# Login variables - SSH/CLI
ansible_user: '{{ vault_ansible_user }}'
ansible_password: '{{ vault_ansible_pass }}'

# Connection variables
ansible_host: '{{ inventory_hostname }}'
ansible_connection: arubanetworks.aoscx.aoscx # REST API via pyaoscx
ansible_network_os: arubanetworks.aoscx.aoscx
ansible_aoscx_validate_certs: false # When ansible_connection=arubanetworks.aoscx.aoscx
ansible_aoscx_use_proxy: false
ansible_acx_no_proxy: true
...

Ansible version

ansible [core 2.13.5]
  config file = /Users/chucknorris/Documents/Azure DevOps/Ansible/ansible.cfg
  configured module search path = ['/Users/chucknorris/ansible6.5/lib64/python3.9/site-packages/napalm_ansible/modules']
  ansible python module location = /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/chucknorris/.ansible/collections:/usr/share/ansible/collections
  executable location = /Library/Frameworks/Python.framework/Versions/3.10/bin/ansible
  python version = 3.10.6 (v3.10.6:9c7b4bd164, Aug  1 2022, 17:13:48) [Clang 13.0.0 (clang-1300.0.29.30)]
  jinja version = 3.1.2
  libyaml = True

PIP version

Package      Version
------------ -------
ansible      6.5.0
ansible-core 2.13.5
cffi         1.15.1
cryptography 38.0.1
Jinja2       3.1.2
MarkupSafe   2.1.1
packaging    21.3
pip          22.2.2
pycparser    2.21
pyparsing    3.0.9
PyYAML       6.0
resolvelib   0.8.1
setuptools   65.4.1
wheel        0.37.1

Ansible configuration file

[defaults]
inventory = ./inventory
host_key_checking = false
gathering = explicit
stdout_callback = yaml
forks = 50
pipelining = true
timeout = 20
strategy = linear
# collections_path = $VIRTUAL_ENV/.ansible/
roles_path = ./roles:~/.ansible/roles
callbacks_enabled = timer, profile_tasks, profile_roles
library = $VIRTUAL_ENV/lib64/python3.9/site-packages/napalm_ansible/modules
action_plugins = $VIRTUAL_ENV/lib64/python3.9/site-packages/napalm_ansible/plugins/action

[persistent_connection]
buffer_read_timeout = 2
connect_timeout = 60
command_timeout = 60
connect_retry_timeout = 30
ssh_type = libssh

Ansible task

---
- name: Configure VLANs - (ArubaOS-CX)
  arubanetworks.aoscx.aoscx_vlan:
    vlan_id: '{{ item.tag }}'
    name: '{{ item.name }}'
    description: '{{ item.name }}'
    state: create
  loop: '{{ vlans }}'
  notify: Write configuration - (ArubaOS-CX)
...

[tchiapuziowong]

Thank you all for assisting us in investigating this issue! We're developing a fix and will be pushing an update soon after completing testing! Thank you for your patience and we'll be updating this issue once the fix has been published 👍

[tchiapuziowong]

We've officially published the release of v4.1.1 of the AOS-CX Ansible collection, the fix for this issue as well as others were a part of the release so please install the latest version of the collection and our Python SDK using the commands below: ansible-galaxy collection install arubanetworks.aoscx -f pip3 install pyaoscx --upgrade

Please let us know if you discover any further issues or unwanted behavior and thank you again for your patience!

[jimdavis66]

Thank you @tchiapuziowong. It is working perfectly for me now.

[KevGue]

Thanks for the update @tchiapuziowong, but now I get the following error on every task in my playbook:

[WARNING]: The "arubanetworks.aoscx.aoscx" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value
instead of the string

In my playbook I am targeting a group, but I also tried targeting a specific host.

Here is an excerpt of the inventory:

all:
  children:
    http:
      children:
        aruba_oscx_http_stage3:
          hosts:
            core-http-aoscx:
              ansible_aoscx_no_proxy: true
              ansible_aoscx_use_proxy: false
              ansible_aoscx_validate_certs: false
              ansible_connection: arubanetworks.aoscx.aoscx
              ansible_network_os: arubanetworks.aoscx.aoscx
              ansible_user: automation

And the beginning my playbook looks like this:

- name: "Backup ArubaOS-CX as JSON"
  hosts: aruba_oscx_http_stage*
  gather_facts: no
  strategy: free
  collections:
    - arubanetworks.aoscx
  vars:
    ansible_python_interpreter: /usr/bin/python3

[DonRhodes]

I too am receiving the [WARNING]: The "arubanetworks.aoscx.aoscx" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value instead of the string message.

Ubuntu 22.04.1 LTS (WSL2) ansible [core 2.13.5rc1] Python 3.10.6

Not sure if it is due to having multiple collections installed after the upgrade?

$ ls -l ~/.ansible/collections/ansible_collections

drwxr-xr-x 4 zadmin zadmin 4096 Oct 25 09:21 ansible
drwxr-xr-x 2 zadmin zadmin 4096 Jul 25 11:44 ansible.netcommon-3.0.1.info
drwxr-xr-x 2 zadmin zadmin 4096 Oct 25 09:21 ansible.netcommon-4.0.0.info
drwxr-xr-x 3 zadmin zadmin 4096 Oct 25 09:14 arubanetworks
drwxr-xr-x 2 zadmin zadmin 4096 Jul 25 11:43 arubanetworks.aoscx-4.0.3.info
drwxr-xr-x 2 zadmin zadmin 4096 Oct 25 09:14 arubanetworks.aoscx-4.1.1.info

[tchiapuziowong]

@KevGue and @DonRhodes please remove any previous versions of the collection and install the latest - you can use this command to install the latest version but you may have to remove the previous version manually: ansible-galaxy collection install arubanetworks.aoscx -f

[DonRhodes]

Will do. I was wondering if that was a possible cause of this.

[DonRhodes]

No luck so far, still getting the error and it is replacing the inventory name with 'inventory_hostname'.

My Ubuntu 20.04.5 LTS box is giving different errors: ansible.module_utils.connection.ConnectionError: Unable to decode JSON from response to get_session(). Received 'None'.

Still looking in to the issue on my end.

[KevGue]

@tchiapuziowong The issue sadly persists.

[tchiapuziowong]

Okay we're looking into it thank you!

[KevGue]

@tchiapuziowong Are there any news on this issue?

[tchiapuziowong]

@KevGue We're still unable to reproduce the issue - can you share your OS, Ansible & Python version? Do you have ansible_host defined in your inventory? If so is that a DNS entry or IPv4/v6 address?

If you're able to escalate to your SE and send a mail to aruba-automation@hpe.com we can work on getting a debug session as well.

[DonRhodes]

@tchiapuziowong Here is my system:

This is a WSL instance:

$ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.1 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.1 LTS (Jammy Jellyfish)"

$ ansible --version ansible [core 2.13.6] config file = /etc/ansible/ansible.cfg configured module search path = ['/home/zadmin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3/dist-packages/ansible ansible collection location = /home/zadmin/.ansible/collections:/usr/share/ansible/collections executable location = /usr/bin/ansible python version = 3.10.6 (main, Nov 2 2022, 18:53:38) [GCC 11.3.0] jinja version = 3.0.3 libyaml = True

$ python3 --version Python 3.10.6

I do not know why this did not occur to me before, but setting ansible_host to an IP playbooks start working correctly. Here is a sample of my inventory file:

all:
  children:
      default:
        hosts:
          office-sw.colgate.edu:
          switch-6300:
            ansible_host: 10.13.0.19
            ansible_aoscx_rest_version: 10.09
      testing:
        hosts:
          test-sw.colgate.edu:
  vars:
    ansible_user: ansiblecxuser
    ansible_password: "{{ ansiblecxuser_pass }}"
    ansible_network_os: arubanetworks.aoscx.aoscx
    ansible_connection: arubanetworks.aoscx.aoscx  # REST API via pyaoscx connection method
    ansible_aoscx_validate_certs: False
    ansible_aoscx_use_proxy: False
    ansible_acx_no_proxy: True
    #
    ansible_ssh_host_key_checking: False
    ansible_paramiko_host_key_checking: False
    ansible_httpapi_validate_certs: False
    ansible_httpapi_use_ssl: True
    connect_timeout: 10

Here is it failing:

PLAY [test-sw.colgate.edu] ******************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************
[WARNING]: The "arubanetworks.aoscx.aoscx" connection plugin has an improperly configured remote target value, forcing "inventory_hostname"
templated value instead of the string
ok: [test-sw.colgate.edu]

TASK [Check for active switch] ******************************************************************************************************************
included: /ansible_git/playbooks/aoscx/tasks/ansible-aoscx-task-check-HTTPS.yml for test-sw.colgate.edu

TASK [__INCLUDED_TASK__ Check if switch can be reached over HTTPS] ******************************************************************************
[WARNING]: The "arubanetworks.aoscx.aoscx" connection plugin has an improperly configured remote target value, forcing "inventory_hostname"
templated value instead of the string
ok: [test-sw.colgate.edu]

TASK [Gather aoscx_facts] ***********************************************************************************************************************
[WARNING]: The "arubanetworks.aoscx.aoscx" connection plugin has an improperly configured remote target value, forcing "inventory_hostname"
templated value instead of the string
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.module_utils.connection.ConnectionError: HTTPSConnectionPool(host='inventory_hostname', port=443): Max retries exceeded with url: /rest/v10.04/login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f232f30f0d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
fatal: [test-sw.colgate.edu]: FAILED! => changed=false
  module_stderr: |-
    Traceback (most recent call last):
      File "/home/zadmin/.ansible/tmp/ansible-local-1963cernk40l/ansible-tmp-1669922365.7616103-2121-105380633511240/AnsiballZ_aoscx_facts.py", line 107, in <module>
        _ansiballz_main()
      File "/home/zadmin/.ansible/tmp/ansible-local-1963cernk40l/ansible-tmp-1669922365.7616103-2121-105380633511240/AnsiballZ_aoscx_facts.py", line 99, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/home/zadmin/.ansible/tmp/ansible-local-1963cernk40l/ansible-tmp-1669922365.7616103-2121-105380633511240/AnsiballZ_aoscx_facts.py", line 47, in invoke_module
        runpy.run_module(mod_name='ansible_collections.arubanetworks.aoscx.plugins.modules.aoscx_facts', init_globals=dict(_module_fqn='ansible_collections.arubanetworks.aoscx.plugins.modules.aoscx_facts', _modlib_path=modlib_path),
      File "/usr/lib/python3.10/runpy.py", line 224, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_aoscx_facts_payload_01c9efm0/ansible_aoscx_facts_payload.zip/ansible_collections/arubanetworks/aoscx/plugins/modules/aoscx_facts.py", line 436, in <module>
      File "/tmp/ansible_aoscx_facts_payload_01c9efm0/ansible_aoscx_facts_payload.zip/ansible_collections/arubanetworks/aoscx/plugins/modules/aoscx_facts.py", line 270, in main
      File "/tmp/ansible_aoscx_facts_payload_01c9efm0/ansible_aoscx_facts_payload.zip/ansible_collections/arubanetworks/aoscx/plugins/module_utils/aoscx_pyaoscx.py", line 82, in get_pyaoscx_session
      File "/tmp/ansible_aoscx_facts_payload_01c9efm0/ansible_aoscx_facts_payload.zip/ansible_collections/arubanetworks/aoscx/plugins/module_utils/aoscx_pyaoscx.py", line 47, in __init__
      File "/tmp/ansible_aoscx_facts_payload_01c9efm0/ansible_aoscx_facts_payload.zip/ansible/module_utils/connection.py", line 200, in __rpc__
    ansible.module_utils.connection.ConnectionError: HTTPSConnectionPool(host='inventory_hostname', port=443): Max retries exceeded with url: /rest/v10.04/login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f232f30f0d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

PLAY RECAP **************************************************************************************************************************************
test-sw.colgate.edu    : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Here it is working:

PLAY [office-sw.colgate.edu] ****************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************
ok: [office-sw.colgate.edu]

TASK [Check for active switch] ******************************************************************************************************************
included: /ansible_git/playbooks/aoscx/tasks/ansible-aoscx-task-check-HTTPS.yml for office-sw.colgate.edu

TASK [__INCLUDED_TASK__ Check if switch can be reached over HTTPS] ******************************************************************************
ok: [office-sw.colgate.edu]

TASK [Gather aoscx_facts] ***********************************************************************************************************************
ok: [office-sw.colgate.edu]

I will contact my SE for you.

[tchiapuziowong]

If you're experiencing a warning please ensure your ansible_host variable is defined - if you design your inventory such that the inventory_hostname is the DNS entry, you can programmatically define the ansible_host variable like so:

all:
  hosts:
    edge-sw.dns-entry.com:    # inventory hostname is set to DNS value
      ansible_host: "{{inventory_hostname}}"     # accessing variable programmatically 
      ansible_user: admin
      ansible_password: password
      ansible_network_os: arubanetworks.aoscx.aoscx
      ansible_connection: arubanetworks.aoscx.aoscx  # REST API via pyaoscx connection method
      ansible_aoscx_validate_certs: False
      ansible_aoscx_use_proxy: False
      ansible_acx_no_proxy: True

[KevGue]

@tchiapuziowong I have sadly only got around to testing this now, but using "{{ inventory_hostname }}" and accessing it via the DNS entry works.

Any reason why this suddenly came up and wasn't an issue before?

[tchiapuziowong]

@KevGue Not sure which was was working previously and not anymore but we've always required ansible_host to be defined in the inventory file, it could be that the Ansible backend we're dependent on changed the way they define/link the variables ansible_host and inventory_hostname