Unable to get the OM collection working (was: "Unable to automatically determine host network os. Please manually configure ansible_network_os value for this host")

[ghost]

Hello,

I fail using the opengear.om collection.

My setup is Ubuntu 20.10 with:

ansible 2.10.8
  config file = /home/mrhode/git/om/ansible.cfg
  configured module search path = ['/home/mrhode/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0]

$ ansible-galaxy collection list

# /home/<user>/.ansible/collections/ansible_collections
Collection  Version
----------- -------
opengear.om 1.0.1 
[...]

Trying to run the demo playbook swap_hostnames

First error:

PLAY [opengear_om] ********************************************************************************************************************************************************
[WARNING]: errors were encountered during the plugin load for meta: ['opengear_om.meta is not a valid collection reference']

Can be fixed with:

  collections:
-    - opengear_om
+    - opengear.om

Now the next error is:

TASK [Get system info] ****************************************************************************************************************************************************
task path: /home/mrhode/git/om/test-playbook.yml:9
redirecting (type: connection) ansible.builtin.httpapi to ansible.netcommon.httpapi
fatal: [om-test]: FAILED! => {}                                                                                                                                                                 

MSG:                                                                                                                                                                                            

Unable to automatically determine host network os. Please manually configure ansible_network_os value for this host                                                                                                                  

Playing around with adding things like ansible_network_os=opengear.om (INI style inventory) didn't succeed.

unable to load API plugin for network_os opengear.om

Tcpdumping doesn't show any network activity from ansible, so this fails before even trying to connect.

REST API itself is working:

{
  "system_version": {
    "rest_api_version": "v2",
    "firmware_version": "21.Q2.1"
  }
}

Thanks a lot!

[ghost]

I did some testing with ubuntu 20.04 and I have the same problems, well apart from the opengear_om problem which works with ansible 2.9.6.

The httpapi plugin is present:

$ ansible-doc -t httpapi -l | grep open
opengear.om.om                     UNDOCUMENTED     

but ansible_network_os=opengear.om.om results in:

redirecting (type: connection) ansible.builtin.httpapi to ansible.netcommon.httpapi
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 613, in _execute
    self._connection = self._get_connection(cvars, templar)
  File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 908, in _get_connection
    connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
  File "/usr/lib/python3/dist-packages/ansible/plugins/loader.py", line 838, in get_with_context
    obj.__init__(instance, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/ansible_collections/ansible/netcommon/plugins/connection/httpapi.py", line 200, in __init__
    self.httpapi = httpapi_loader.get(self._network_os, self)
  File "/usr/lib/python3/dist-packages/ansible/plugins/loader.py", line 792, in get
    return self.get_with_context(name, *args, **kwargs).object
  File "/usr/lib/python3/dist-packages/ansible/plugins/loader.py", line 813, in get_with_context
    self._load_config_defs(name, self._module_cache[path], path)
  File "/usr/lib/python3/dist-packages/ansible/plugins/loader.py", line 388, in _load_config_defs
    add_fragments(dstring, path, fragment_loader=fragment_loader, is_module=(type_name == 'module'))
  File "/usr/lib/python3/dist-packages/ansible/utils/plugin_docs.py", line 124, in add_fragments
    fragments = doc.pop('extends_documentation_fragment', [])
AttributeError: 'AnsibleUnicode' object has no attribute 'pop'
fatal: [om-test]: FAILED! => {}

MSG:

Unexpected failure during module execution.

on ubuntu 20.04 and 21.10.

[ghost]

I was able to get one step further:

  vars:
    ansible_python_interpreter: /usr/bin/python3
    ansible_network_os: ansible.netcommon.restconf

Is this the correct approach? Actually I doubt it... ansible_network_os=opengear.om.om seems more plausible to me. Anyway it fails with in TASK [Get system info]

 The full traceback is:
  File "/tmp/ansible_opengear.om.om_facts_payload_4lbphffe/ansible_opengear.om.om_facts_payload.zip/ansible_collections/ansible/netcommon/plugins/module_utils/network/common/network.py", line 251, in get_capabilities
    capabilities = Connection(module._socket_path).get_capabilities()
  File "/tmp/ansible_opengear.om.om_facts_payload_4lbphffe/ansible_opengear.om.om_facts_payload.zip/ansible/module_utils/connection.py", line 195, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [om-test]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "gather_network_resources": [
                "system"
            ],
            "gather_subset": [
                "min"
            ]
        }
    }
}

MSG:

Method not found

[DavidMorp]

I'm having the same issue trying to connect to a CM7116-2-DAC

  File "/var/folders/y2/msl_81dd5xx6tg0ptr71v_yr0000gn/T/ansible_opengear.om.om_facts_payload_di2dg6ug/ansible_opengear.om.om_facts_payload.zip/ansible_collections/ansible/netcommon/plugins/module_utils/network/common/network.py", line 251, in get_capabilities
    capabilities = Connection(module._socket_path).get_capabilities()
  File "/var/folders/y2/msl_81dd5xx6tg0ptr71v_yr0000gn/T/ansible_opengear.om.om_facts_payload_di2dg6ug/ansible_opengear.om.om_facts_payload.zip/ansible/module_utils/connection.py", line 200, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [console01]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "gather_network_resources": [
                "ports"
            ],
            "gather_subset": [
                "min"
            ]
        }
    },
    "msg": "Method not found"
}

The playbook I run is:

- name: facts
  hosts: console01
  gather_facts: true
  vars:
    ansible_connection: ansible.netcommon.httpapi

  tasks:
    - name: Get port info
      opengear.om.om_facts:
        gather_subset: min
        gather_network_resources: system

[ghost]

@DavidMorp I opened a support ticket with opengear and they have the same problem. I've been told that they're going to fix this but this statement is already several weeks old.

[coolcps]

I get the same "Method not found" error :(

[mattwit]

Looks like the doc string in >plugin>httapi>om.py was causing the issue. The latest commit removes the doc string. We have validated that user_config.yaml and group_config.yaml are indeed working. We are validating and updating the remaining playbooks and will post a new commit when completed.

[ghost]

Dear @mattwit, thanks for taking a look at this!

I still fail - but at least with another error :)

I edited collections/ansible_collections/opengear/om/plugins/httpapi/om.py and commented out the DOCUMENTATON part. Now I receive a new error:

TASK [Add user] ***********************************************************************************************************************************************************
fatal: [om-test]: FAILED! => {
    "changed": false
}

MSG:

HTTP Error 401: Unauthorized

my modified user_ playbook looks as follows:

- hosts: opengear_om
  collections:
    - opengear.om
  connection: httpapi
  become: false
  vars:
    ansible_python_interpreter: /usr/bin/python3
    ansible_network_os: opengear.om.om
    ansible_httpapi_use_ssl: true
    ansible_httpapi_validate_certs: false
    ansible_remote_user: root
  gather_facts: false
  tasks:
    - name: Add user
      opengear.om.om_users:
        config:
        - username: test1
          enabled: no
          no_password: yes
          description: This user was changed.
        - username: test3
          enabled: yes
          no_password: yes
          description: This user was added.
        state:
          overridden

inventory:

[opengear_om]
om-test ansible_host=10.10.42.10

ssh works (ssh key auth):

 $ ssh root@10.10.42.10
net1:dhcp  10.10.42.10/24
root@om1208-8e-l:~# 

any hints?

[ghost]

ok got it: adding ansible_httpapi_password: <password> to the playbook makes it work.

But as storing credentials in plain text in a git repo is not an option I wonder if it's possible to use SSH keys or something else?

[mattwit]

ok got it: adding ansible_httpapi_password: <password> to the playbook makes it work.

But as storing credentials in plain text in a git repo is not an option I wonder if it's possible to use SSH keys or something else?

Have you taken a look at Ansible Vault?

[ghost]

@mattwit thanks for the hint, we use it quite a lot. I was thinking of our CM automation where we disabled passwords and only allow ssh-key auth. But there we work with scp and to push bash scripts and are not using API magic.

[coolcps]

I removed the doc string and get "network os opengear.om.om is not supported"

- hosts: test-tsv
  collections:
    - opengear.om
  connection: httpapi
  become: false
  vars:
    ansible_network_os: opengear.om.om
  gather_facts: false
  tasks:
    - name: Add user
      opengear.om.om_users:
        config:
        - username: test_user
          enabled: no
          no_password: yes
        state:
          merged

EDIT: Disregard, I was able to get it to work on a fresh ansible install. I think this may be a conflict with napalm or something along those lines.

[ghost]

got back to this, another error comes up:

The full traceback is:
Traceback (most recent call last):
  File "/home/mrhode/.ansible/tmp/ansible-local-238027lqb7aca/ansible-tmp-1674650714.4757082-23815-229724681231238/AnsiballZ_om_facts.py", line 107, in <module>
    _ansiballz_main()
  File "/home/mrhode/.ansible/tmp/ansible-local-238027lqb7aca/ansible-tmp-1674650714.4757082-23815-229724681231238/AnsiballZ_om_facts.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/mrhode/.ansible/tmp/ansible-local-238027lqb7aca/ansible-tmp-1674650714.4757082-23815-229724681231238/AnsiballZ_om_facts.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.opengear.om.plugins.modules.om_facts', init_globals=dict(_module_fqn='ansible_collections.opengear.om.plugins.modules.om_facts', _modlib_path=modlib_path),
  File "/usr/lib/python3.10/runpy.py", line 224, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/modules/om_facts.py", line 109, in <module>
  File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/modules/om_facts.py", line 100, in main
  File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/facts/facts.py", line 70, in get_facts
  File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/ansible/netcommon/plugins/module_utils/network/common/facts/facts.py", line 122, in get_network_resources_facts
  File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/facts/system/system.py", line 37, in __init__
  File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/utils/utils.py", line 16, in get_restapi_body_structure
FileNotFoundError: [Errno 2] No such file or directory: '/home/avankat/Documents/automated-device-config/opengear.om/plugins/module_utils/network/om/utils/structure.json'
fatal: [bre1-michael-t1]: FAILED! => changed=false 
  module_stderr: |-
    Traceback (most recent call last):
      File "/home/mrhode/.ansible/tmp/ansible-local-238027lqb7aca/ansible-tmp-1674650714.4757082-23815-229724681231238/AnsiballZ_om_facts.py", line 107, in <module>
        _ansiballz_main()
      File "/home/mrhode/.ansible/tmp/ansible-local-238027lqb7aca/ansible-tmp-1674650714.4757082-23815-229724681231238/AnsiballZ_om_facts.py", line 99, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/home/mrhode/.ansible/tmp/ansible-local-238027lqb7aca/ansible-tmp-1674650714.4757082-23815-229724681231238/AnsiballZ_om_facts.py", line 47, in invoke_module
        runpy.run_module(mod_name='ansible_collections.opengear.om.plugins.modules.om_facts', init_globals=dict(_module_fqn='ansible_collections.opengear.om.plugins.modules.om_facts', _modlib_path=modlib_path),
      File "/usr/lib/python3.10/runpy.py", line 224, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/modules/om_facts.py", line 109, in <module>
      File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/modules/om_facts.py", line 100, in main
      File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/facts/facts.py", line 70, in get_facts
      File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/ansible/netcommon/plugins/module_utils/network/common/facts/facts.py", line 122, in get_network_resources_facts
      File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/facts/system/system.py", line 37, in __init__
      File "/tmp/ansible_opengear.om.om_facts_payload_bzs6zx4w/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/utils/utils.py", line 16, in get_restapi_body_structure
    FileNotFoundError: [Errno 2] No such file or directory: '/home/avankat/Documents/automated-device-config/opengear.om/plugins/module_utils/network/om/utils/structure.json'
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

editing ~/.ansible/collections/ansible_collections/opengear/om/plugins/module_utils/network/om/utils/utils.py and replacing

  - /home/avankat/Documents/automated-device-config/opengear.om
  - collections/ansible_collections/opengear/om/

and placing the whole OM collection in the collections folder of the ansible root dir did the trick.

btw: ansible-galaxy collection is still version 1.0.1 with commit https://github.com/opengear/opengear.om/commit/a7d2fd0df1dd19983d3c61d6447f337d6955444c missing and therefore still broken

[mattwit]

FYI, the ansible-galaxy collection has been updated to version 1.0.2!!!

[richardjohnallsopp]

Hey folks I'm using the latest version of the collections and I'm hitting this error with HTTPS:

PLAY [all] *********************************************************************************************************************
[WARNING]: errors were encountered during the plugin load for meta: ['opengear_om.meta is not a valid collection reference']

TASK [Get system info] *********************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: FileNotFoundError: [Errno 2] No such file or directory: '/home/avankat/Documents/automated-device-config/opengear.om/plugins/module_utils/network/om/utils/structure.json'
fatal: [home-console-01.encom-corp.net]: FAILED! => changed=false 
  module_stderr: |-
    Traceback (most recent call last):
      File "/home/flynn/.ansible/tmp/ansible-local-1499663e52bzo/ansible-tmp-1694172225.1151688-15000-34930516345751/AnsiballZ_om_facts.py", line 107, in <module>
        _ansiballz_main()
      File "/home/flynn/.ansible/tmp/ansible-local-1499663e52bzo/ansible-tmp-1694172225.1151688-15000-34930516345751/AnsiballZ_om_facts.py", line 99, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/home/flynn/.ansible/tmp/ansible-local-1499663e52bzo/ansible-tmp-1694172225.1151688-15000-34930516345751/AnsiballZ_om_facts.py", line 47, in invoke_module
        runpy.run_module(mod_name='ansible_collections.opengear.om.plugins.modules.om_facts', init_globals=dict(_module_fqn='ansible_collections.opengear.om.plugins.modules.om_facts', _modlib_path=modlib_path),
      File "/home/flynn/.pyenv/versions/3.9.11/lib/python3.9/runpy.py", line 210, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/home/flynn/.pyenv/versions/3.9.11/lib/python3.9/runpy.py", line 97, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/home/flynn/.pyenv/versions/3.9.11/lib/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_opengear.om.om_facts_payload_zvtw2aci/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/modules/om_facts.py", line 109, in <module>
      File "/tmp/ansible_opengear.om.om_facts_payload_zvtw2aci/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/modules/om_facts.py", line 100, in main
      File "/tmp/ansible_opengear.om.om_facts_payload_zvtw2aci/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/facts/facts.py", line 70, in get_facts
      File "/tmp/ansible_opengear.om.om_facts_payload_zvtw2aci/ansible_opengear.om.om_facts_payload.zip/ansible_collections/ansible/netcommon/plugins/module_utils/network/common/facts/facts.py", line 124, in get_network_resources_facts
      File "/tmp/ansible_opengear.om.om_facts_payload_zvtw2aci/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/facts/system/system.py", line 37, in __init__
      File "/tmp/ansible_opengear.om.om_facts_payload_zvtw2aci/ansible_opengear.om.om_facts_payload.zip/ansible_collections/opengear/om/plugins/module_utils/network/om/utils/utils.py", line 16, in get_restapi_body_structure
    FileNotFoundError: [Errno 2] No such file or directory: '/home/avankat/Documents/automated-device-config/opengear.om/plugins/module_utils/network/om/utils/structure.json'
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

PLAY RECAP *********************************************************************************************************************
home-console-01.encom-corp.net : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0 

I'm running the following Ansible versions:

ansible==6.0.0
ansible-core==2.13.11
ansible-pylibssh==1.1.0

If I remove these variables:

ansible_httpapi_port: 443
ansible_httpapi_use_ssl: true
ansible_httpapi_validate_certs: false

I get this error instead:

PLAY [all] *********************************************************************************************************************************************************************************************
[WARNING]: errors were encountered during the plugin load for meta: ['opengear_om.meta is not a valid collection reference']

TASK [Get system info] *********************************************************************************************************************************************************************************
fatal: [home-console-01.encom-corp.net]: FAILED! => changed=false 
  msg: 'Expecting value: line 1 column 1 (char 0)'

PLAY RECAP *********************************************************************************************************************************************************************************************
home-console-01.encom-corp.net : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

I have the username and password variables set.

Let me know if you would like a new issue for this, I added it here as it was still open.

Looks like this file is missing:

FileNotFoundError: [Errno 2] No such file or directory: '/home/avankat/Documents/automated-device-config/opengear.om/plugins/module_utils/network/om/utils/structure.json'