Telnet support and various changes

andymillernz commented 1 year ago

Various Small Changes

Support Telnet as an access method
Implement default max log ingestions per run (50)
Add missing OSPF and vrrp_engine route types
Add option to bypass duplicate/existing devices (duplicate discoverables)
Fix for NXOS Hostname detection
Improvement for Cisco device type detection

dainok commented 1 year ago

Hi @andymillernz, thank you for your huge work! I have a couple of questions:

RAISE_ON_DISCOVERABLE_NOT_ATTACHED: during ingestion phase, HOSTNAME logs are parsed at first, the device is created and bound to the discoverable. Logs that are ingested after HOSTNAME, require discoverable.device to be set. Otherwise ingestion phase won't know where attach objects. Which is the use case you considered?
Deleted interfaces: I prefer to drop deleted interfaces in the ingestion phase (like I did for netmiko_cisco_ios_show_ip_interface.py). This avoid NetBox to be filled with dirty data. Where did you find deleted interfaces?

andymillernz commented 1 year ago

Hi @andymillernz, thank you for your huge work! I have a couple of questions:

RAISE_ON_DISCOVERABLE_NOT_ATTACHED: during ingestion phase, HOSTNAME logs are parsed at first, the device is created and bound to the discoverable. Logs that are ingested after HOSTNAME, require discoverable.device to be set. Otherwise ingestion phase won't know where attach objects. Which is the use case you considered?

I saw this when there were multiple discoverables (with different IP addresses) pointing to the same device. Happens after new discoverables are automatically added - then edited to made them "discoverable" and fresh discover run down on them.

Deleted interfaces: I prefer to drop deleted interfaces in the ingestion phase (like I did for netmiko_cisco_ios_show_ip_interface.py). This avoid NetBox to be filled with dirty data. Where did you find deleted interfaces?

Happy to delete the interface in ingestion -- I made the current change just to get things running - almost certainly from an NXOS device..

dainok commented 1 year ago

@andymillernz yes it can happen when you are using a specific management IP address but CDP/LLDP report a different address. To maintain a discoverable list clean, I could add a DONT_ADD_DISCOVERABLE=True (default False) setting. This could solve your issue?

andymillernz commented 1 year ago

@andymillernz yes it can happen when you are using a specific management IP address but CDP/LLDP report a different address. To maintain a discoverable list clean, I could add a DONT_ADD_DISCOVERABLE=True (default False) setting. This could solve your issue?

I think that is a good idea - however both are needed as one of the key use cases (I believe) is having NetDoc discover unknown devices.

It does raise a set of possible enhancements with regards to discoverables & devices - I'll create a new issue to discuss..

dainok commented 1 year ago

I'm dividing the PR into items to discuss and integrate them. I'm revisiting how NetDoc is discovering so it's easy for me to manually merge changes one by one.

Changes:

RAISE_ON_DISCOVERABLE_NOT_ATTACHED: not merged, maybe we can improve. The reason is that log ingestion requires device to be set, so not having an attached device means not ingesting logs, and discovery will be useless.
cisco_ios_telnet: to be merged, very interesting approach!
show hostname on NXOS: to be merged, thank you for the suggestion, I didn't aware about this command.
not attached_interface on Cisco IOS PO: info needed. With this change missing attached interfaces won't be created anymore. This should not be an issue untile show etherchannel cames after show interfaces. Why you need that?
netmiko_cisco_ios_show_inventory: seems ok. Can you spend a couple of words describing your use case?
MAX_INGESTED_LOGS: I suspect I wrote a bug on that, and maybe you found it. Need to test but seems ok, thank you!
deleted interfaces: I prefer ingestors skip deleted interfaces to maintain NetBox clean.
route types: to be merged.
vlan range: to be merged.

andymillernz commented 1 year ago

not attached_interface on Cisco IOS PO: info needed. With this change missing attached interfaces won't be created anymore. This should not be an issue untile show etherchannel cames after show interfaces. Why you need that?

Probably dealing with a Cisco bug. Have an IOS device that the "show ether channel summary" includes:

387    Po387(SU)        -        Gi3/1(P)       Gi3/2(P)       Gi3/3(P)
                                 Gi3/4(P)       Gi3/5(P)       Gi3/6(P)

However none of those interfaces (Po387, Gi3/x) exist - suspect they did exist and were deleted, and/or hardware failed/removed

netmiko_cisco_ios_show_inventory: seems ok. Can you spend a couple of words describing your use case?

The previous code only worked with "Chassis" based routers - so any device that was a single 'box' (Cisco 2960) was not being identified correctly and was added as a Cisco/Unknown Device. In all cases I tested the first entry in inventory is the hardware description. Could possibly remove the "chassis" check altogether?

show inventory
NAME: "1", DESCR: "WS-C2960XR-48TS-I"
PID: WS-C2960XR-48TS-I , VID: V06  , SN: xxxxx

NAME: "Switch 1 - Power Supply 1", DESCR: "FRU Power Supply"
PID: PWR-C2-250WAC     , VID: V04Q , SN: xxxx

dainok commented 1 year ago

Changes:

route types: merged with https://github.com/dainok/netdoc/pull/63
vlan range: merged with https://github.com/dainok/netdoc/pull/63
show hostname on NXOS: merged with https://github.com/dainok/netdoc/pull/59
MAX_INGESTED_LOGS: merged with https://github.com/dainok/netdoc/pull/64
netmiko_cisco_ios_show_inventory: merged with https://github.com/dainok/netdoc/pull/65
cisco_ios_telnet: merged with https://github.com/dainok/netdoc/pull/66

Not to be merged:

RAISE_ON_DISCOVERABLE_NOT_ATTACHED: as discussed, we are opening a different thread
not attached_interface: it was just a debugging test
deleted interfaces. to be moved in ingestion phase

dainok / netdoc

Telnet support and various changes #58