search

dainok / netdoc

Automatic Network Documentation plugin for NetBox
GNU General Public License v3.0
89 stars 13 forks source link

Ingest via netmiko_cisco_ios error #74

Closed gabrielat closed 11 months ago

gabrielat commented 11 months ago

Hi! I'm running a brand new installation of Netbox with Netdoc (I have followed the steps described on https://github.com/dainok/netdoc/wiki/NetBox-with-NetDoc-installation) and I tried to run "Add and discover" script without creating anything but Site. I'm getting the following error:

An exception occurred: ValueError: The discoverable 143.54.0.5 does not have an attached device thus logs cannot be ingested. Check if logs with priority 0 are ingested or if Device is attached to a different Discoverable.

Traceback (most recent call last):
  File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 353, in run
    log_ingest(log)
  File "/opt/netbox/venv/lib/python3.10/site-packages/netdoc/utils.py", line 572, in log_ingest
    raise ValueError(
ValueError: The discoverable 143.54.0.5 does not have an attached device thus logs cannot be ingested. Check if logs with priority 0 are ingested or if Device is attached to a different Discoverable.

So, after I had this error, I've created the Manufacturer Cisco, Device Type WS-C4506-E, Device Router with a virtual interface that has the same IP address as the Discoverable, but still got the same error.

Any thoughts on how can I get it to work?

Thank you!

dainok commented 11 months ago

Can you post a screenshot of the discoverable? Can you also check discovery logs? You should find a log with hostname template. Check why this is not ingested.

ismailkalolwala commented 11 months ago

Hi there, this is what i am getting

image

dainok commented 11 months ago

Seems that the migration to Netbox 3.5 brings a lot of unexpected errors, even if I tested NetDoc on my lab. I need to make additional tests. If you are hurry, please use NetDoc 0.10.x with NetBox 3.4.10. Stay tuned

ismailkalolwala commented 11 months ago

image

dainok commented 11 months ago

https://github.com/dainok/netdoc/pull/78 should fix the issue. Please upgrade to 3.5.4 and let me know if anything changes.

proebstle commented 11 months ago

Hi, same issue.... new insalled with Debian 12.....

Okt 03 14:34:29 KK-NETBOX-01 python3[13550]: 12:34:29 default: Job OK (d04d5c3d-af23-4d1b-9614-ba77cb75b770) Okt 03 14:34:29 KK-NETBOX-01 python3[13550]: 12:34:29 Result is kept for 500 seconds Okt 03 14:34:29 KK-NETBOX-01 python3[13451]: 12:34:29 Cleaning registries for queue: high Okt 03 14:34:29 KK-NETBOX-01 python3[13451]: 12:34:29 Cleaning registries for queue: default Okt 03 14:34:29 KK-NETBOX-01 python3[13451]: 12:34:29 Cleaning registries for queue: low Okt 03 14:34:29 KK-NETBOX-01 python3[13451]: 12:34:29 default: extras.scripts.run_script(data={}, job=<Job: 9e0baa10-768e-462d-bb8d-88e340b1bd0b>, request=<utilities.utils.NetBoxFakeRequest object at 0x7fd5b875f6d0>) (9e0baa10-768e-462d-bb8d-88e340b1bd0b)

Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: An exception occurred: TypeError: Device() got unexpected keyword arguments: 'device_role_id' Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: Traceback (most recent call last): Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: script.output = script.run(data=data, commit=commit) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 353, in run Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: log_ingest(log) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/venv/lib/python3.11/site-packages/netdoc/utils.py", line 578, in log_ingest Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: module.ingest(log) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/venv/lib/python3.11/site-packages/netdoc/ingestors/netmiko_hp_procurve_hostname.py", line 38, in ingest Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: device_o = device.create(**data) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: ^^^^^^^^^^^^^^^^^^^^^ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/venv/lib/python3.11/site-packages/netdoc/schemas/device.py", line 108, in create Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: obj = utils.object_create(Device, **kwargs) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/venv/lib/python3.11/site-packages/netdoc/utils.py", line 1055, in object_create Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: return model_o.objects.create(**kwargs) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: return getattr(self.get_queryset(), name)(*args, **kwargs) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py", line 656, in create Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: obj = self.model(**kwargs) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: ^^^^^^^^^^^^^^^^^^^^ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/netbox/utilities/tracking.py", line 41, in __init__ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: super().__init__(*args, **kwargs) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/base.py", line 567, in __init__ Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: raise TypeError( Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: TypeError: Device() got unexpected keyword arguments: 'device_role_id' Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: Exception raised during script execution: Device() got unexpected keyword arguments: 'device_role_id' Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: 12:34:30 default: Job OK (9e0baa10-768e-462d-bb8d-88e340b1bd0b) Okt 03 14:34:30 KK-NETBOX-01 python3[13555]: 12:34:30 Result is kept for 500 seconds

image

dainok commented 11 months ago

@proebstle Can you describe, step by step, how to reproduce the error? I'm not able to.

dainok commented 11 months ago

NetDoc 3.5 requires NetBox 3.5.

gabrielat commented 10 months ago

Hi! Sorry for the delay here, I was on vacation. I'm running Netbox 3.5.8 and I get this error on ingest:

An exception occurred: ValueError: The discoverable 192.168.0.5 does not have an attached device thus logs cannot be ingested. Check if logs with priority 0 are ingested or if Device is attached to a different Discoverable.

Traceback (most recent call last):
  File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 353, in run
    log_ingest(log)
  File "/opt/netbox/venv/lib/python3.10/site-packages/netdoc/utils.py", line 572, in log_ingest
    raise ValueError(
ValueError: The discoverable 192.168.0.5 does not have an attached device thus logs cannot be ingested. Check if logs with priority 0 are ingested or if Device is attached to a different Discoverable.

On the first run, I didn't have any device created. However, I did create it after the first time I had this error and now I do have a Device with the same primary IP, but it seems that it is not possible to attach the results with the device..

Thank you!

dainok commented 10 months ago

Upgrade Netdoc to latest and recheck. If issue is still happening, please check why logs with priority 0 are not ingesting.

gabrielat commented 10 months ago

Hi.. The error has changed after upgrading netdoc to 3.5.10. This is what happens:

Script Log
Line    Level   Message
1   Info    
This is the parent ingest job

2   Info    
123 logs to be ingested

3   Info    
Limiting ingesting to 25 logs

4   Info    
Ingesting log 278 with command show version on device 192.168.0.5 via netmiko_cisco_ios

5   Info    
Ingesting log 282 with command show version on device 192.168.0.5 via netmiko_cisco_ios

6   Info    
Ingesting log 280 with command show version on device 192.168.0.4 via netmiko_cisco_ios

7   Info    
Ingesting log 84 with command show version on device 192.168.0.5 via netmiko_cisco_ios

8   Info    
Ingesting log 201 with command show version on device 192.168.0.4 via netmiko_cisco_ios

9   Info    
Ingesting log 162 with command show version on device 192.168.0.5 via netmiko_cisco_ios

10  Info    
Ingesting log 45 with command show version on device 192.168.0.4 via netmiko_cisco_ios

11  Info    
Ingesting log 123 with command show version on device 192.168.0.5 via netmiko_cisco_ios

12  Info    
Ingesting log 240 with command show version on device 192.168.0.5 via netmiko_cisco_ios

13  Info    
Ingesting log 164 with command show interfaces on device 192.168.0.5 via netmiko_cisco_ios

14  Info    
Ingesting log 47 with command show interfaces on device 192.168.0.4 via netmiko_cisco_ios

15  Info    
Ingesting log 125 with command show interfaces on device 192.168.0.5 via netmiko_cisco_ios

16  Info    
Ingesting log 86 with command show interfaces on device 192.168.0.5 via netmiko_cisco_ios

17  Info    
Ingesting log 242 with command show interfaces on device 192.168.0.5 via netmiko_cisco_ios

18  Info    
Ingesting log 203 with command show interfaces on device 192.168.0.4 via netmiko_cisco_ios

19  Info    
Ingesting log 48 with command show cdp neighbors detail on device 192.168.0.4 via netmiko_cisco_ios

20  Info    
Ingesting log 204 with command show cdp neighbors detail on device 192.168.0.4 via netmiko_cisco_ios

21  Info    
Ingesting log 127 with command show lldp neighbors detail on device 192.168.0.5 via netmiko_cisco_ios

22  Info    
Ingesting log 49 with command show lldp neighbors detail on device 192.168.0.4 via netmiko_cisco_ios

23  Failure 
An exception occurred: IntegrityError: Multiple neighbors on ROUT-CC-CETEL2-01:GigabitEthernet2/7 (gi2/7) or SW-CC-SECOM-01:gi1/0/28 (gi1/0/28)

Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.10/site-packages/netdoc/schemas/cable.py", line 87, in link
    cable_o = Cable.objects.filter(terminations__interface=left_interface_o).get(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/query.py", line 650, in get
    raise self.model.DoesNotExist(
dcim.models.cables.Cable.DoesNotExist: Cable matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "dcim_cabletermination_unique_termination"
DETAIL:  Key (termination_type_id, termination_id)=(38, 534) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.10/site-packages/netdoc/schemas/cable.py", line 96, in link
    CableTermination.objects.create(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/query.py", line 671, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/netbox/netbox/dcim/models/cables.py", line 319, in save
    super().save(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 812, in save
    self.save_base(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 863, in save_base
    updated = self._save_table(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1006, in _save_table
    results = self._do_insert(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1047, in _do_insert
    return manager._insert(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1791, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "dcim_cabletermination_unique_termination"
DETAIL:  Key (termination_type_id, termination_id)=(38, 534) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 353, in run
    log_ingest(log)
  File "/opt/netbox/venv/lib/python3.10/site-packages/netdoc/utils.py", line 578, in log_ingest
    module.ingest(log)
  File "/opt/netbox/venv/lib/python3.10/site-packages/netdoc/ingestors/netmiko_cisco_ios_show_lldp_neighbors_detail.py", line 100, in ingest
    cable.link(
  File "/opt/netbox/venv/lib/python3.10/site-packages/netdoc/schemas/cable.py", line 111, in link
    raise IntegrityError(
django.db.utils.IntegrityError: Multiple neighbors on ROUT-CC-CETEL2-01:GigabitEthernet2/7 (gi2/7) or SW-CC-SECOM-01:gi1/0/28 (gi1/0/28)

24  Info    
Database changes have been reverted due to error.
dainok commented 10 months ago

Hi! Don't use closed issues, I'm not monitor them. BTW check the FAQ https://github.com/dainok/netdoc/wiki/FAQ