Failed to Add Node with ICMP Monitoring - 400 Client Error: Object reference not set to an instance of an object

[Andyjb8]

I am successfully able to add nodes to solarwinds with snmp polling but I am having an issue with adding them with ICMP polling. I am getting a 400 Client Error: Object reference not set to an instance of an object.

Here is my playbook:

- hosts: all
  gather_facts: false

  vars:
    solarwinds_info: &solarwinds_info
      hostname: <server_name>
      username: <my_user>
      password: <omitted>

  tasks:

    - name: Add ICMP node to Solarwinds
      solarwinds.orion.orion_node:
        <<: *solarwinds_info
        state: present
        name: "{{ inventory_hostname }}"
        ip_address: "{{ ansible_host }}"
        #polling_method: ICMP # Also tried with this line and no change
        polling_engine: 1 # If I don't inlucde this I error " get_least_used_polling_engine IndexError: list index out of range"
      delegate_to: localhost

TASK [Add ICMP node to Solarwinds] ****************************************************************************************************************************
task path: /home/<user>/automation/playbooks/orion_icmp_test.yml:26
Friday 04 August 2023  10:53:35 +0000 (0:00:00.072)       0:00:00.072 ********* 
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: <user>
<localhost> EXEC /bin/sh -c 'echo ~<user> && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/<user>/.ansible/tmp `"&& mkdir "` echo /home/<user>/.ansible/tmp/ansible-tmp-1691146415.247887-207253-115825580269209 `" && echo ansible-tmp-1691146415.247887-207253-115825580269209="` echo /home/<user>/.ansible/tmp/ansible-tmp-1691146415.247887-207253-115825580269209 `" ) && sleep 0'
Using module file /home/<user>/.ansible/collections/ansible_collections/solarwinds/orion/plugins/modules/orion_node.py
<localhost> PUT /home/<user>/.ansible/tmp/ansible-local-207245kyy5paem/tmpsw3uh5ur TO /home/<user>/.ansible/tmp/ansible-tmp-1691146415.247887-207253-115825580269209/AnsiballZ_orion_node.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/<user>/.ansible/tmp/ansible-tmp-1691146415.247887-207253-115825580269209/ /home/<user>/.ansible/tmp/ansible-tmp-1691146415.247887-207253-115825580269209/AnsiballZ_orion_node.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/<user>/.ansible/tmp/ansible-tmp-1691146415.247887-207253-115825580269209/AnsiballZ_orion_node.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/<user>/.ansible/tmp/ansible-tmp-1691146415.247887-207253-115825580269209/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/tmp/ansible_solarwinds.orion.orion_node_payload_4uf1xe3n/ansible_solarwinds.orion.orion_node_payload.zip/ansible_collections/solarwinds/orion/plugins/modules/orion_node.py", line 295, in add_node
  File "/home/********/.local/lib/python3.10/site-packages/orionsdk/swisclient.py", line 34, in create
    return self._req(
  File "/home/********/.local/lib/python3.10/site-packages/orionsdk/swisclient.py", line 63, in _req
    resp.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
fatal: [RTR-01 -> localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "hostname": "solarwinds_server",
            "ip_address": "10.x.x.x",
            "name": "RTR-01",
            "node_id": null,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "polling_engine": "1",
            "polling_method": "ICMP",
            "ro_community_string": null,
            "rw_community_string": null,
            "snmp_allow_64": true,
            "snmp_port": "161",
            "snmp_version": null,
            "snmpv3_auth_key": null,
            "snmpv3_auth_key_is_pwd": true,
            "snmpv3_auth_method": "SHA1",
            "snmpv3_priv_key": null,
            "snmpv3_priv_key_is_pwd": true,
            "snmpv3_priv_method": "AES128",
            "snmpv3_username": null,
            "state": "present",
            "unmanage_from": null,
            "unmanage_until": null,
            "username": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "wmi_credentials": null
        }
    },
    "msg": "Failed to create node: 400 Client Error: Object reference not set to an instance of an object. for url: https://solarwinds_server:17778/SolarWinds/InformationService/v3/Json/Create/Orion.Nodes"
}

Even though I get that error, the node shows up in solarwinds. However even though it is there, it stays in an unknown status and will not go into up/green status even forcing polling/rediscover and giving it time. I know the node responds to ping and is reachable because if I manaully add it to solarwinds with ICMP monitoring it immediatly goes into up/green status.

Also, after running the same playbook the second time, it appears runs sucecssfully since the node is already there:

TASK [Add ICMP node to Solarwinds] ****************************************************************************************************************************
task path: /home/<user>/automation/playbooks/orion_icmp_test.yml:26
Friday 04 August 2023  10:56:52 +0000 (0:00:00.074)       0:00:00.074 ********* 
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: <user>
<localhost> EXEC /bin/sh -c 'echo ~<user> && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/<user>/.ansible/tmp `"&& mkdir "` echo /home/<user>/.ansible/tmp/ansible-tmp-1691146612.7231917-207381-32741157161022 `" && echo ansible-tmp-1691146612.7231917-207381-32741157161022="` echo /home/<user>/.ansible/tmp/ansible-tmp-1691146612.7231917-207381-32741157161022 `" ) && sleep 0'
Using module file /home/<user>/.ansible/collections/ansible_collections/solarwinds/orion/plugins/modules/orion_node.py
<localhost> PUT /home/<user>/.ansible/tmp/ansible-local-207370tb6eo8bu/tmpxole4q_c TO /home/<user>/.ansible/tmp/ansible-tmp-1691146612.7231917-207381-32741157161022/AnsiballZ_orion_node.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/<user>/.ansible/tmp/ansible-tmp-1691146612.7231917-207381-32741157161022/ /home/<user>/.ansible/tmp/ansible-tmp-1691146612.7231917-207381-32741157161022/AnsiballZ_orion_node.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/<user>/.ansible/tmp/ansible-tmp-1691146612.7231917-207381-32741157161022/AnsiballZ_orion_node.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/<user>/.ansible/tmp/ansible-tmp-1691146612.7231917-207381-32741157161022/ > /dev/null 2>&1 && sleep 0'
ok: [RTR-01 -> localhost] => {
    "changed": false,
    "invocation": {
        "module_args": {
            "hostname": "solarwinds_server",
            "ip_address": "10.x.x.x",
            "name": "RTR-01",
            "node_id": null,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "polling_engine": "1",
            "polling_method": "ICMP",
            "ro_community_string": null,
            "rw_community_string": null,
            "snmp_allow_64": true,
            "snmp_port": "161",
            "snmp_version": null,
            "snmpv3_auth_key": null,
            "snmpv3_auth_key_is_pwd": true,
            "snmpv3_auth_method": "SHA1",
            "snmpv3_priv_key": null,
            "snmpv3_priv_key_is_pwd": true,
            "snmpv3_priv_method": "AES128",
            "snmpv3_username": null,
            "state": "present",
            "unmanage_from": null,
            "unmanage_until": null,
            "username": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "wmi_credentials": null
        }
    },
    "orion_node": {
        "caption": "RTR-01",
        "ipaddress": "10.x.x.x",
        "netobjectid": "N:1072",
        "nodeid": 1072,
        "objectsubtype": "ICMP",
        "status": 0,
        "statusdescription": "Node status is Unknown.",
        "unmanaged": false,
        "unmanagefrom": "1899-12-30T00:00:00+00:00",
        "unmanageuntil": "1899-12-30T00:00:00+00:00",
        "uri": "swis://solarwinds_server.automation.lab/Orion/Orion.Nodes/NodeID=1072"
    }
}

But as you can see in statusdescription above, the node still shows unknown and won't go up/green. Also if I add the same node with ansible with snmp polling it works. And if I manaually change it to ICMP polling after adding with SNMP polling, that also works, but I can't get ansible to add it with ICMP Polling. This issue was replicated in both our lab and production environments.

[Andyjb8]

Also, if it helps, here is the error log on solarswinds.

2023-08-04 03:53:45,347 [376] INFO SolarWinds.Orion.Web.AuthorizationManager+WindowsAuthorizationManager - (null) (null) Resolved group: AUTOMATION\Domain Admins for identity: AUTOMATION\. 2023-08-04 03:53:46,035 [376] INFO SolarWinds.Orion.Web.OrionMixedModeAuth - (null) (null) Successfully retrieved WindowsIdentity for user AUTOMATION\. 2023-08-04 03:53:46,176 [376] INFO SolarWinds.Orion.Web.AuthorizationManager+WindowsAuthorizationManager - (null) (null) Resolved group: AUTOMATION\Domain Admins for identity: AUTOMATION\. 2023-08-04 03:53:46,285 [376] ERROR SolarWinds.InformationService.Core.CrudProcessor - (null) (null) Create operation failed. System.NullReferenceException: Object reference not set to an instance of an object. at SolarWinds.Data.Providers.Orion.Helpers.SnmpV3CredentialsHelper.UpdateNodesSnmpV3Credentials(Int32[] nodeIds, IDictionary2 columnsToUpdate) at SolarWinds.Data.Providers.Orion.CrudHandlers.NodeCrudHandler.Create(IEnumerable1 infos, Func2 baseCall) at SolarWinds.InformationService.Addons.DataProvider.<Create>d__63.MoveNext() at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at SolarWinds.InformationService.Core.CrudProcessor.<CreateLocal>d__22.MoveNext() at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at SolarWinds.InformationService.Core.CrudProcessor.d__19.MoveNext() at System.Linq.Enumerable.First[TSource](IEnumerable1 source) at SolarWinds.InformationService.Core.CrudProcessor.Create(String entityType, IDictionary2 properties, IQueryExecutionContext context) 2023-08-04 03:53:46,301 [376] ERROR SolarWinds.InformationService.Core.InformationService - (null) (null) Exception for Operation:

10.x.x.x RTR-01 ICMP AES128 true SHA1 true 161 true 1

2023-08-04 03:53:46,301 [376] ERROR SolarWinds.InformationService.Core.InformationService - (null) (null) Exception caught in method SolarWinds.InformationService.Core.InformationService.Create System.NullReferenceException: Object reference not set to an instance of an object. at SolarWinds.Data.Providers.Orion.Helpers.SnmpV3CredentialsHelper.UpdateNodesSnmpV3Credentials(Int32[] nodeIds, IDictionary2 columnsToUpdate) at SolarWinds.Data.Providers.Orion.CrudHandlers.NodeCrudHandler.Create(IEnumerable1 infos, Func2 baseCall) at SolarWinds.InformationService.Addons.DataProvider.<Create>d__63.MoveNext() at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at SolarWinds.InformationService.Core.CrudProcessor.<CreateLocal>d__22.MoveNext() at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at SolarWinds.InformationService.Core.CrudProcessor.d__19.MoveNext() at System.Linq.Enumerable.First[TSource](IEnumerable1 source) at SolarWinds.InformationService.Core.CrudProcessor.Create(String entityType, IDictionary2 properties, IQueryExecutionContext context) at SolarWinds.InformationService.Core.InformationService.Create(String entityType, IDictionary`2 properties)

[Andyjb8]

I figured out it has to do with the null SNMPv3 values being passed even though it is adding it as ICMP. This worked and node went into an up/green status:

    - name: Add ICMP node to Solarwinds
      solarwinds.orion.orion_node:
        <<: *solarwinds_info
        state: present
        name: "{{ inventory_hostname }}"
        ip_address: "{{ ansible_host }}"
        polling_method: ICMP
        polling_engine: 1
        snmp_version: 3
        snmpv3_auth_key: none
        snmpv3_auth_method: SHA1
        snmpv3_priv_key: none
        snmpv3_priv_method: AES128
        snmpv3_username:  none
      delegate_to: localhost

[jeisenbath]

Awesome thanks for testing so much! This is a fixable bug, I believe, by changing the add_node function in orion_node.py to only set the snmpv3 node properties when the polling type is SNMP and SNMPVersion is 3. I will try to test this code today.

def add_node(module, orion):

    props = {
        'IPAddress': module.params['ip_address'],
        'Caption': module.params['name'],
        'ObjectSubType': module.params['polling_method'].upper(),
        'Community': module.params['ro_community_string'],
        'RWCommunity': module.params['rw_community_string'],
        'SNMPVersion': module.params['snmp_version'],
        'AgentPort': module.params['snmp_port'],
        'Allow64BitCounters': module.params['snmp_allow_64'],
        'External': lambda x: True if module.params['polling_method'] == 'EXTERNAL' else False,
    }
...
    if module.params['snmp_version'] == '3' and props['ObjectSubType'] == 'SNMP':
        if module.params['snmpv3_username']:
            props['SNMPV3Username'] = module.params['snmpv3_username']
        if module.params['snmpv3_priv_method']:
            props['SNMPV3PrivMethod'] = module.params['snmpv3_priv_method']
        if module.params['snmpv3_priv_key_is_pwd']:
            props['SNMPV3PrivKeyIsPwd'] = module.params['snmpv3_priv_key_is_pwd']
        if module.params['snmpv3_priv_key']:
            props['SNMPV3PrivKey'] = module.params['snmpv3_priv_key']
        if module.params['snmpv3_auth_method']:
            props['SNMPV3AuthMethod'] = module.params['snmpv3_auth_method']
        if module.params['snmpv3_auth_key_is_pwd']:
            props['SNMPV3AuthKeyIsPwd'] = module.params['snmpv3_auth_key_is_pwd']
        if module.params['snmpv3_auth_key']:
            props['SNMPV3AuthKey'] = module.params['snmpv3_auth_key']

The behavior of the node being added but showing Grey for the status is a common symptom when a node add gets an exception, since the code crashes before it can add the actual ICMP response time/status pollers to the node.

[jeisenbath]

I was able to replicate this error myself @Andyjb8 I have pushed v1.0.2 with these changes, you should no longer need to provide the snmpv3 params when adding an ICMP or snmpv2 node

Thanks again, and also feel free to open an issue on the polling_engine issue you mentioned polling_engine: 1 # If I don't inlucde this I error " get_least_used_polling_engine IndexError: list index out of range"

[Andyjb8]

Ok, it works now, Thanks. But is there a reason is still shows the snmp parameters as being passed in the debug when adding an ICMP node?

Playbook Task:

    - name: Add External node to Solarwinds
      solarwinds.orion.orion_node:
        <<: *solarwinds_info
        state: present
        name: "{{ inventory_hostname }}"
        ip_address: "{{ ansible_host }}"
      delegate_to: localhost

Here is my output: changed: [rtr-01 -> localhost] => { "changed": true, "invocation": { "module_args": { "hostname": "orion_hostname", "ip_address": "10x.x.x", "name": "rtr-01", "node_id": null, "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "polling_engine": null, "polling_method": "ICMP", "ro_community_string": null, "rw_community_string": null, "snmp_allow_64": true, "snmp_port": "161", "snmp_version": null, "snmpv3_auth_key": null, "snmpv3_auth_key_is_pwd": true, "snmpv3_auth_method": "SHA1", "snmpv3_priv_key": null, "snmpv3_priv_key_is_pwd": true, "snmpv3_priv_method": "AES128", "snmpv3_username": null, "state": "present", "unmanage_from": null, "unmanage_until": null, "username": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "wmi_credentials": null } }, "orion_node": { "caption": "rtr-01", "ipaddress": "10.x.x.x", "netobjectid": "N:1081", "nodeid": 1081, "objectsubtype": "ICMP", "status": 0, "statusdescription": "Node status is Unknown.", "unmanaged": false, "unmanagefrom": "1899-12-30T00:00:00+00:00", "unmanageuntil": "1899-12-30T00:00:00+00:00", "uri": "swis://orion_hostname/Orion/Orion.Nodes/NodeID=1081" } }

[jeisenbath]

It looks like I still have some default values for some of the SNMPv3 parameters, so that would be why. I'll correct that in an upcoming bugfix version.