configuration_item_info fails to transform a ServiceNow record

[Klaas-]

SUMMARY

I think the module is unable to get all configuration items from our service now instance

ISSUE TYPE

• Bug Report

COMPONENT NAME

servicenow.itsm.configuration_item_info

ANSIBLE VERSION

ansible 2.9.14
  config file = /home/me/ansible/ansible.cfg
  configured module search path = [u'/home/me/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Mar 20 2020, 17:08:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

CONFIGURATION

ANSIBLE_SSH_ARGS(/home/me/ansible/ansible.cfg) = -C -o ControlMaster=auto -o ControlPersist=60s -o ServerAliveInterval=60
CACHE_PLUGIN(/home/me/ansible/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/home/me/ansible/ansible.cfg) = $HOME/ansible/facts
CACHE_PLUGIN_TIMEOUT(/home/me/ansible/ansible.cfg) = 600
DEFAULT_CALLBACK_WHITELIST(/home/me/ansible/ansible.cfg) = [u'profile_tasks', u'mail']
DEFAULT_FORKS(/home/me/ansible/ansible.cfg) = 30
DEFAULT_NO_TARGET_SYSLOG(/home/me/ansible/ansible.cfg) = True
DEFAULT_ROLES_PATH(/home/me/ansible/ansible.cfg) = [u'/etc/ansible/roles', u'/usr/share/ansible/roles']
DIFF_ALWAYS(/home/me/ansible/ansible.cfg) = True

OS / ENVIRONMENT

RHEL 7.9 targets a service now paris-06-24-2020__patch7b-04-26-2021_05-04-2021_2116

STEPS TO REPRODUCE

    - name: Fetch configuration item we will update
      servicenow.itsm.configuration_item_info:
        instance:
          host: "{{ servicenow_host }}"
          username: "{{ servicenow_username }}"
          password: "{{ servicenow_password }}"
        sys_class_name: cmdb_ci_computer
      register: result

EXPECTED RESULTS

Have all my computers in a variable

ACTUAL RESULTS

The error was: AttributeError: 'unicode' object has no attribute 'items'

File \"/tmp/ansible_servicenow.itsm.configuration_item_info_payload_moxe2f/ansible_servicenow.itsm.configuration_item_info_payload.zip/ansible_collections/servicenow/itsm/plugins/module_utils/utils.py\", line 65, in _transform\nAttributeError: 'unicode' object has no attribute 'items'\n"

I looked at the code and thought maybe it's a weird computer that is missing something, so I changed the code to check for the existence of 'items' and if it doesn't exist skip the computer. That will then show me that the last computer is always empty. I first thought it's a specific computer but when checking with result.records | length I noticed the number of items always differs. So I am guessing this is a problem in paginating or something like that? Of my ~20k elements in computer only ~5k get to ansible before it throws an empty element.

[Akasurde]

@Klaas- Thanks for reporting this issue. Could you please paste the output with -vvvv so that we can see the stack trace presented? Thanks.

[Klaas-]

yeah sure, full stacktrace:

The full traceback is:
Traceback (most recent call last):
  File "/home/me/.ansible/tmp/ansible-tmp-1624280778.46-89590-247879270742260/AnsiballZ_configuration_item_info.py", line 102, in <module>
    _ansiballz_main()
  File "/home/me/.ansible/tmp/ansible-tmp-1624280778.46-89590-247879270742260/AnsiballZ_configuration_item_info.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/me/.ansible/tmp/ansible-tmp-1624280778.46-89590-247879270742260/AnsiballZ_configuration_item_info.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.servicenow.itsm.plugins.modules.configuration_item_info', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib64/python2.7/runpy.py", line 176, in run_module
    fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/ansible_servicenow.itsm.configuration_item_info_payload_qgo33G/ansible_servicenow.itsm.configuration_item_info_payload.zip/ansible_collections/servicenow/itsm/plugins/modules/configuration_item_info.py", line 188, in <module>
  File "/tmp/ansible_servicenow.itsm.configuration_item_info_payload_qgo33G/ansible_servicenow.itsm.configuration_item_info_payload.zip/ansible_collections/servicenow/itsm/plugins/modules/configuration_item_info.py", line 181, in main
  File "/tmp/ansible_servicenow.itsm.configuration_item_info_payload_qgo33G/ansible_servicenow.itsm.configuration_item_info_payload.zip/ansible_collections/servicenow/itsm/plugins/modules/configuration_item_info.py", line 163, in run
  File "/tmp/ansible_servicenow.itsm.configuration_item_info_payload_qgo33G/ansible_servicenow.itsm.configuration_item_info_payload.zip/ansible_collections/servicenow/itsm/plugins/module_utils/utils.py", line 73, in to_ansible
  File "/tmp/ansible_servicenow.itsm.configuration_item_info_payload_qgo33G/ansible_servicenow.itsm.configuration_item_info_payload.zip/ansible_collections/servicenow/itsm/plugins/module_utils/utils.py", line 65, in _transform
AttributeError: 'unicode' object has no attribute 'items'

[tadeboro]

This error indicates that the data returned from the ServiceNow instance is corrupted: instead of the record map, we somehow ended up with a string in the result.

@Klaas- Can you please post the output of the following command:

curl \
  --header "Accept:application/json" \
  --user 'your_user':'yourp_password' \
  'https://your.instance-address.here/api/now/table/cmdb_ci_computer'

If everything goes as planned, you should get back something similar to this (but in a bit less readable format):

{
  "result": [
    {
      "os_address_width": "",
      "attested_date": "",
      "operational_status": "1",
      "os_service_pack": "",
      "cpu_core_thread": "",
      "cpu_manufacturer": "",
      "sys_updated_on": "2021-03-01 22:53:40",
      "discovery_source": "",
      "first_discovered": "",
      "due_in": "",
      # More data here
      "lease_id": ""
    }
  ]
}

[Klaas-]

Hi, curl shows it's being aborted serverside :)

Result is like this (run through python -m json.tool)

{
    "error": {
        "detail": "Transaction cancelled: maximum execution time exceeded Check logs for error trace or enable glide.rest.debug property to verify REST request processing",
        "message": "Transaction cancelled: maximum execution time exceeded"
    },
    "result": [
        {
[... a lot of assets ...]
        },
        ""
    ],
    "status": "failure"
}

The last asset is an empty one which creates the problems with utils.py, but I am guessing the more correct way of addressing this is catching the error and printing that message.

[tadeboro]

Hmm, maybe we should reduce the batch size to get around this problem. Right now, we use the default value of 1000, which could be problematic. We added support for pagination in #82, but we still use the default value of 1000 records per request there. If we expose this as an instance parameter, playbook authors can take control of this value.

But in any case, we need to add more safeguards to the table client. We naively assumed that if the response code is 200, things are OK. But we were clearly wrong ;)

[Klaas-]

Maybe I am misunderstanding you, but the curl get you gave me is not paginated, it simply returns all elements it can fetch until the query is cancelled server side. But I do agree that if you pageinate you will most likely get around the issue but it'll simply transform the single large query into smaller pieces that are manageable within the maximum transaction time.

[tadeboro]

Maybe I am misunderstanding you, but the curl get you gave me is not paginated, it simply returns all elements it can fetch until the query is cancelled server side.

It is implicitly paginated. If there is no sysparm_limit query parameter, ServiceNow treats it as 1000 (go to https://docs.servicenow.com/bundle/quebec-application-development/page/integrate/inbound-rest/concept/c_TableAPI.html#c_TableAPI and search for sysparm_limit on that page).

[Akasurde]

@Klaas- Could you please check if this issue still persists and let us know? Thanks.

needs_info

[tima]

Closing because of activity.