Trouble ingest

[FunnyZebra174]

Hi, I'm using netbox 3.5.9 and netdoc 3.5.21. It detects devices perfectly and connects to them, but it can't add anything to the database. At first I thought that the problem was in the rights, but after looking at the log log, I saw that the problem was at the stage of ingestion. He gives such an error and I do not know what to do with it, tell me how to solve it. Thank you in advance. The hardware I used for the tests are cisco c9300 and 2960 cx switches.

[dainok]

Can you post the log 5 so I can see why the parser is failing?

[FunnyZebra174]

This error occurs when using a script (add and discover) and (ingest). These logs were used when using the ingest script. Sep 23 13:35:53 localhost python3[29457]: 08:35:53 default: extras.scripts.run_script(commit=True, data={'discoverables': <RestrictedQuerySet [<Discoverable: 10.8.175.21 via netmi..., job=, request=<utilities.utils.NetBoxFakeRequest object at 0x7fbf9071ff70>) (d3ebfdcd-2848-46fc-99b8-e8b9ee960f63) Sep 23 13:36:02 localhost python3[64460]: An exception occurred: AttributeError: 'NoneType' object has no attribute 'lower' Sep 23 13:36:02 localhost python3[64460]: Sep 23 13:36:02 localhost python3[64460]: Traceback (most recent call last): Sep 23 13:36:02 localhost python3[64460]: File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script Sep 23 13:36:02 localhost python3[64460]: script.output = script.run(data=data, commit=commit) Sep 23 13:36:02 localhost python3[64460]: File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 357, in run Sep 23 13:36:02 localhost python3[64460]: log_ingest(log) Sep 23 13:36:02 localhost python3[64460]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 407, in log_ingest Sep 23 13:36:02 localhost python3[64460]: module.ingest(log) Sep 23 13:36:02 localhost python3[64460]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/ingestors/netmiko_cisco_ios_show_cdp_neighbors_detail.py", line 14, in ingest Sep 23 13:36:02 localhost python3[64460]: neighbors_per_interface = utils.count_interface_neighbors( Sep 23 13:36:02 localhost python3[64460]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 82, in count_interface_neighbors Sep 23 13:36:02 localhost python3[64460]: label = normalize_interface_label(item.get(key)) Sep 23 13:36:02 localhost python3[64460]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 465, in normalize_interface_label Sep 23 13:36:02 localhost python3[64460]: name = name.lower().strip() Sep 23 13:36:02 localhost python3[64460]: AttributeError: 'NoneType' object has no attribute 'lower' Sep 23 13:36:02 localhost python3[64460]: Sep 23 13:36:02 localhost python3[64460]: Exception raised during script execution: 'NoneType' object has no attribute 'lower' Sep 23 13:36:02 localhost python3[64460]: 08:36:02 default: Job OK (d3ebfdcd-2848-46fc-99b8-e8b9ee960f63) Sep 23 13:36:02 localhost python3[64460]: 08:36:02 Result is kept for 500 seconds Sep 23 13:36:02 localhost python3[29457]: 08:36:02 Cleaning registries for queue: high Sep 23 13:36:02 localhost python3[29457]: 08:36:02 Cleaning registries for queue: default Sep 23 13:36:02 localhost python3[29457]: 08:36:02 Cleaning registries for queue: low Sep 23 13:36:35 localhost python3[29457]: 08:36:35 default: extras.scripts.run_script(commit=True, data={'credential': , 'mode': 'netmiko_cisco_ios', 'site':..., job=<Job: 28502ec0-03ed-4ae1-8d9c-9b10320db2eb>, request=<utilities.utils.NetBoxFakeRequest object at 0x7fbf905c8100>) (28502ec0-03ed-4ae1-8d9c-9b10320db2eb)

[FunnyZebra174]

And this is the start and end of the logs when using the script add and discover.
Sep 23 13:36:35 localhost python3[64711]: ['10.8.175.21'] Sep 23 13:36:35 localhost python3[64711]: dict_keys(['10.8.175.21']) Sep 23 13:36:35 localhost python3[64711]: No Netmiko Allied Telesis AW+ (allied_telesis_awplus) device found Sep 23 13:36:35 localhost python3[64711]: No Netmiko Aruba OSCX (aruba_osswitch) device found Sep 23 13:36:39 localhost python3[64711]: multiple_tasks** Sep 23 13:36:39 localhost python3[64711]: * 10.8.175.21 ** changed : False *** Sep 23 13:36:39 localhost python3[64711]: vvvv multiple_tasks changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO Sep 23 13:36:39 localhost python3[64711]: ---- {"command": "show version", "template": "HOSTNAME", "enable": true, "order": 0} changed : False INFO Sep 23 13:36:39 localhost python3[64711]: Cisco IOS XE Software, Version 17.06.01 Sep 23 13:36:39 localhost python3[64711]: Cisco IOS Software [Bengaluru], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.6.1, RELEASE SOFTWARE (fc6) Sep 23 13:36:39 localhost python3[64711]: Technical Support: http://www.cisco.com/techsupport

---- {"command": "show ip route rip", "template": "show ip route", "enable": true, "order": 103} changed : False INFO Sep 23 13:36:42 localhost python3[64711]: ---- {"command": "show ip route bgp", "template": "show ip route", "enable": true, "order": 104} changed : False INFO Sep 23 13:36:42 localhost python3[64711]: ---- {"command": "show ip route eigrp", "template": "show ip route", "enable": true, "order": 105} changed : False INFO Sep 23 13:36:42 localhost python3[64711]: ---- {"command": "show ip route ospf", "template": "show ip route", "enable": true, "order": 106} changed : False INFO Sep 23 13:36:42 localhost python3[64711]: ---- {"command": "show ip route isis", "template": "show ip route", "enable": true, "order": 107} ** changed : False INFO Sep 23 13:36:42 localhost python3[64711]: ^^^^ END additional_tasks ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sep 23 13:36:42 localhost python3[64711]: No Netmiko Cisco IOS XE (Telnet) (cisco_ios_telnet) device found Sep 23 13:36:42 localhost python3[64711]: No Netmiko Cisco NX-OS (cisco_nxos) device found Sep 23 13:36:42 localhost python3[64711]: No Netmiko Cisco XR (cisco_xr) device found Sep 23 13:36:42 localhost python3[64711]: No Netmiko HPE Comware (hp_comware) device found Sep 23 13:36:42 localhost python3[64711]: No Netmiko HPE Procurve (hp_procurve) device found Sep 23 13:36:42 localhost python3[64711]: No Netmiko HPE Procurve (Telnet) (hp_procurve_telnet) device found Sep 23 13:36:42 localhost python3[64711]: No Netmiko Huawei VRP (huawei) device found Sep 23 13:36:42 localhost python3[64711]: No Netmiko Linux (linux) device found Sep 23 13:36:42 localhost python3[64711]: No VMware vSphere (vmware_vsphere) device found Sep 23 13:36:42 localhost python3[64711]: No Palo Alto Networks NGFW (panw_ngfw) device found Sep 23 13:36:43 localhost python3[64711]: 08:36:43 default: Job OK (28502ec0-03ed-4ae1-8d9c-9b10320db2eb) Sep 23 13:36:43 localhost python3[64711]: 08:36:43 Result is kept for 500 seconds Sep 23 13:36:43 localhost python3[29457]: 08:36:43 default: extras.scripts.run_script(data={}, job=<Job: 739678ac-1727-4986-8d6e-f0d72ee29bfc>, request=<utilities.utils.NetBoxFakeRequest object at 0x7f6becda9a00>) (739678ac-1727-4986-8d6e-f0d72ee29bfc) Sep 23 13:36:53 localhost python3[64765]: An exception occurred: AttributeError: 'NoneType' object has no attribute 'lower' Sep 23 13:36:53 localhost python3[64765]: Sep 23 13:36:53 localhost python3[64765]: Traceback (most recent call last): Sep 23 13:36:53 localhost python3[64765]: File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script Sep 23 13:36:53 localhost python3[64765]: script.output = script.run(data=data, commit=commit) Sep 23 13:36:53 localhost python3[64765]: File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 357, in run Sep 23 13:36:53 localhost python3[64765]: log_ingest(log) Sep 23 13:36:53 localhost python3[64765]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 407, in log_ingest Sep 23 13:36:53 localhost python3[64765]: module.ingest(log) Sep 23 13:36:53 localhost python3[64765]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/ingestors/netmiko_cisco_ios_show_cdp_neighbors_detail.py", line 14, in ingest Sep 23 13:36:53 localhost python3[64765]: neighbors_per_interface = utils.count_interface_neighbors( Sep 23 13:36:53 localhost python3[64765]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 82, in count_interface_neighbors Sep 23 13:36:53 localhost python3[64765]: label = normalize_interface_label(item.get(key)) Sep 23 13:36:53 localhost python3[64765]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 465, in normalize_interface_label Sep 23 13:36:53 localhost python3[64765]: name = name.lower().strip() Sep 23 13:36:53 localhost python3[64765]: AttributeError: 'NoneType' object has no attribute 'lower' Sep 23 13:36:53 localhost python3[64765]: Sep 23 13:36:53 localhost python3[64765]: Exception raised during script execution: 'NoneType' object has no attribute 'lower' Sep 23 13:36:53 localhost python3[64765]: 08:36:53 default: Job OK (739678ac-1727-4986-8d6e-f0d72ee29bfc)

[dainok]

See the last FAQ about exporting logs: https://github.com/dainok/netdoc/wiki/FAQ#netdoc-is-not-parsingingesting-data

[FunnyZebra174]

The log with the number 137 just fits the description from that question 137.json

[cmong-nextech]

Looks like your instance is trying to normalize an interface label that's being created via the "show cdp/lldp neighbors" command. The "'NoneType' object has no attribute lower" is poor error handling for stating it can't use that method on a null value (the interface wasn't created earlier in the process. I'd bet that this is due to a missing line in your textfsm template for the show lldp/cdp neighbors.

Try adding this regex string under line 28 in "cisco_ios_show_lldp_neighbors_detail.textfsm" ^Local\s+Intf\s+service\s+instance:\s+-\s*$$

[FunnyZebra174]

Thank you, your solution helped, but now I'm faced with another error. Sep 24 09:06:33 localhost python3[1273]: 04:06:33 default: extras.scripts.run_script(commit=True, data={'discoverables': <RestrictedQuerySet [<Discoverable: 10.8.175.21 via netmi..., job=<Job: 864bb884-385d-4afb-a529-3ab4fab32aa3>, request=<utilities.utils.NetBoxFakeRequest object at 0x7fe5efd27670>) (864bb884-385d-4afb-a529-3ab4fab32aa3) Sep 24 09:06:47 localhost python3[37492]: An exception occurred: TypeError: expected string or bytes-like object Sep 24 09:06:47 localhost python3[37492]: Sep 24 09:06:47 localhost python3[37492]: Traceback (most recent call last): Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script Sep 24 09:06:47 localhost python3[37492]: script.output = script.run(data=data, commit=commit) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 357, in run Sep 24 09:06:47 localhost python3[37492]: log_ingest(log) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 407, in log_ingest Sep 24 09:06:47 localhost python3[37492]: module.ingest(log) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/ingestors/netmiko_cisco_ios_show_cdp_neighbors_detail.py", line 14, in ingest Sep 24 09:06:47 localhost python3[37492]: neighbors_per_interface = utils.count_interface_neighbors( Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 82, in count_interface_neighbors Sep 24 09:06:47 localhost python3[37492]: label = normalize_interface_label(item.get(key)) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 468, in normalize_interface_label Sep 24 09:06:47 localhost python3[37492]: if re.match(r".*-trk\d*$", str(name)): Sep 24 09:06:47 localhost python3[37492]: File "/usr/lib64/python3.9/re.py", line 191, in match Sep 24 09:06:47 localhost python3[37492]: return _compile(pattern, flags).match(string) Sep 24 09:06:47 localhost python3[37492]: TypeError: expected string or bytes-like object Sep 24 09:06:47 localhost python3[37492]: Sep 24 09:06:47 localhost python3[37492]: Exception raised during script execution: expected string or bytes-like object Sep 24 09:06:47 localhost python3[37492]: 04:06:47 default: Job OK (864bb884-385d-4afb-a529-3ab4fab32aa3) Sep 24 09:06:47 localhost python3[37492]: 04:06:47 Result is kept for 500 seconds

[dainok]

Templates are maintained by NetworkToCode (see https://github.com/networktocode/ntc-templates). I decided to raise errors so users know that parsing is failing. Said that the only way to get help is to post failing logs. Now the 95 is failing.

[FunnyZebra174]

I found out what the reason was, netdoc was trying to get data by keys that are not in the dictionary at the entrance. In file файл /opt/netbox/venv/lib64/python3.9/site-packages/netdoc/ingestors/netmiko_cisco_ios_show_cdp_neighbors_detail.py. I found out what the reason was, netdoc was trying to get data using keys that are not in the dictionary at the entrance. I replaced local_port with local_interface (line 15 and 20) and destination_host with neighbor_name(line 25)

[dainok]

Maybe NTC has implemented breaking changes. NetDoc includes by default a working release of templates. If you don’t need latest updates maybe you can use the embedded ones.

Sent from Proton Mail for iOS

On Tue, Sep 24, 2024 at 06:12, FunnyZebra174 @.***(mailto:On Tue, Sep 24, 2024 at 06:12, FunnyZebra174 < wrote:

Thank you, your solution helped, but now I'm faced with another error. image.png (view on web) Sep 24 09:06:33 localhost python3[1273]: 04:06:33 default: extras.scripts.run_script(commit=True, data={'discoverables': <RestrictedQuerySet [<Discoverable: 10.8.175.21 via netmi..., job=<Job: 864bb884-385d-4afb-a529-3ab4fab32aa3>, request=<utilities.utils.NetBoxFakeRequest object at 0x7fe5efd27670>) (864bb884-385d-4afb-a529-3ab4fab32aa3) Sep 24 09:06:47 localhost python3[37492]: An exception occurred: TypeError: expected string or bytes-like object Sep 24 09:06:47 localhost python3[37492]: Sep 24 09:06:47 localhost python3[37492]: Traceback (most recent call last): Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script Sep 24 09:06:47 localhost python3[37492]: script.output = script.run(data=data, commit=commit) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 357, in run Sep 24 09:06:47 localhost python3[37492]: log_ingest(log) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 407, in log_ingest Sep 24 09:06:47 localhost python3[37492]: module.ingest(log) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/ingestors/netmiko_cisco_ios_show_cdp_neighbors_detail.py", line 14, in ingest Sep 24 09:06:47 localhost python3[37492]: neighbors_per_interface = utils.count_interface_neighbors( Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 82, in count_interface_neighbors Sep 24 09:06:47 localhost python3[37492]: label = normalize_interface_label(item.get(key)) Sep 24 09:06:47 localhost python3[37492]: File "/opt/netbox/venv/lib64/python3.9/site-packages/netdoc/utils.py", line 468, in normalize_interface_label Sep 24 09:06:47 localhost python3[37492]: if re.match(r".-trk\d$", str(name)): Sep 24 09:06:47 localhost python3[37492]: File "/usr/lib64/python3.9/re.py", line 191, in match Sep 24 09:06:47 localhost python3[37492]: return _compile(pattern, flags).match(string) Sep 24 09:06:47 localhost python3[37492]: TypeError: expected string or bytes-like object Sep 24 09:06:47 localhost python3[37492]: Sep 24 09:06:47 localhost python3[37492]: Exception raised during script execution: expected string or bytes-like object Sep 24 09:06:47 localhost python3[37492]: 04:06:47 default: Job OK (864bb884-385d-4afb-a529-3ab4fab32aa3) Sep 24 09:06:47 localhost python3[37492]: 04:06:47 Result is kept for 500 seconds

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>