Can we try to add Huawei support?

gabrielat commented 11 months ago

Hello, I saw that there was an issue to add Huawei support, but it didn't follow through. Is this all you need? How can I help?

To add a new supported device, we need, at least:

A 2 node lab (see https://github.com/dainok/netdoc/tree/master/netdoc/tests/netmiko_cisco_ios/lab1)
Check if supported by Netmiko (see http://ktbyers.github.io/netmiko/PLATFORMS.html)
List of commands
Add missing NTC Templates (see https://github.com/networktocode/ntc-templates/tree/master/tests)

Originally posted by @dainok in https://github.com/dainok/netdoc/issues/2#issuecomment-1580211135

dainok commented 11 months ago

Best is if you are able to write discoverers, templates, ingestors. If not provide a lab and I'll do the best. Consider sponsoring :)

gabrielat commented 11 months ago

The lab is created only with text files? Or did you export this files from a software?

dainok commented 11 months ago

You can use a simulator but at the end you need a set of text and YAML files. Text files contain the output of "display" commands, YAML files contain what you expect to be ingested into Netbox. See https://github.com/dainok/netdoc/tree/master/netdoc/tests/netmiko_cisco_xr/lab1/logs

You can prepare a physical or a virtual lab then export logs, see https://github.com/dainok/netdoc/blob/master/scripts/lab_export.py

dainok commented 11 months ago

Obviously lab files will be for public (testing purpose).

dainok commented 10 months ago

Still interested? Can you provide the requested files?

gabrielat commented 10 months ago

Hi! Yes, I am still interested. I’m trying to do what you’ve asked!

Enviado do meu iPhone

Em 19 de nov. de 2023, à(s) 06:16, Andrea Dainese @.***> escreveu:

Still interested? Can you provide the requested files?

— Reply to this email directly, view it on GitHubhttps://github.com/dainok/netdoc/issues/102#issuecomment-1817796383, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRCQMJMBPMHLDAYXTWNDZ3YFHE6DAVCNFSM6AAAAAA7BQ7M5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJXG44TMMZYGM. You are receiving this because you authored the thread.Message ID: @.***>

dainok commented 10 months ago

Ok, thank you for the feedback.

gabrielat commented 10 months ago

Hi @dainok ! I think I managed to do the lab. Here are the files netmiko_huawei_vrp.zip

dainok commented 10 months ago

Thank you @gabrielat I'm working on that in the next weeks.

dainok commented 10 months ago

@gabrielat one question: which netmiko tye should I use?

Huawei
Huawei OLT
Huawei SmartAX

gabrielat commented 10 months ago

@dainok Huawei

dainok commented 10 months ago

@gabrielat few questions:

can you confirm the command "display arp all"? (you reported "display arp" only)
can you provide the output of "display ip routing-table verbose" from node1?
can you provide the output of "display vlan all" from node1?

dainok commented 10 months ago

See https://github.com/dainok/netdoc/pull/112

gabrielat commented 10 months ago

Sure, I’ll do that later today. But there were a few commands that didn’t have the “all” option. But I’ll confirm that and get back to you.

Best regards Enviado do meu iPhone

Em 23 de nov. de 2023, à(s) 09:22, Andrea Dainese @.***> escreveu:

@gabrielathttps://github.com/gabrielat few questions:

can you confirm the command "display arp all"? (you reported "display arp" only)
can you provide the output of "display ip routing-table verbose" from node1?
can you provide the output of "display vlan all" from node1?

— Reply to this email directly, view it on GitHubhttps://github.com/dainok/netdoc/issues/102#issuecomment-1824340657, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADRCQMO34RGZG35GLU77NCLYF45XDAVCNFSM6AAAAAA7BQ7M5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRUGM2DANRVG4. You are receiving this because you were mentioned.Message ID: @.***>

dainok commented 10 months ago

@gabrielat

I will go with the following commands, please confirm:

display current-configuration | include sysname display current-configuration display device manufacture-info display lldp neighbor display vlan all display ip vpn-instance display interface display eth-trunk display ip interface display ip routing-table verbose display lldp neighbor brief display arp all display mac-address

No commands are required on a per vrf bassis. ARP and routing are included for all VRFs in the above commands. After you confirm, I'll start to write and adapt NTC templates (most of them are missing or not working as expected).

dainok commented 10 months ago

I need from your side:

the output of "display ip routing-table verbose" from node1?
the output of "display vlan all" from node1?
the output of "display interface" with a trunk with more than one aggregated interface (add one interface Eth-Trunk0, no matter if it's down). Please provide also "display eth-trunks"

gabrielat commented 10 months ago

@gabrielat

I will go with the following commands, please confirm:

display current-configuration | include sysname display current-configuration display device manufacture-info display lldp neighbor display vlan all display ip vpn-instance display interface display eth-trunk display ip interface display ip routing-table verbose display lldp neighbor brief display arp all display mac-address

No commands are required on a per vrf bassis. ARP and routing are included for all VRFs in the above commands. After you confirm, I'll start to write and adapt NTC templates (most of them are missing or not working as expected).

Hi! All commands listed above are ok, except "display vlan all". This is what Huawei displays on vlan:

display vlan ? INTEGER<1-4094> VLAN ID pool VLAN pool summary Summary information of VLAN vlan-name Specify VLAN name | Matching output

gabrielat commented 10 months ago

3. display interface

I've disassembled the lab I did before, can I sent you the results for a real switch?

display ip routing-table verbose: Route Flags: R - relay, D - download to fib, T - to vpn-instance

Routing Tables: Public Destinations : 363 Routes : 363

Destination: 0.0.0.0/0 Protocol: O_ASE Process ID: 1 Preference: 150 Cost: 1 NextHop: 10.10.0.1 Neighbour: 0.0.0.0 State: Active Adv Age: 30d22h50m33s Tag: 0 Priority: high Label: NULL QoSInfo: 0x0 IndirectID: 0x0
RelayNextHop: 0.0.0.0 Interface: Vlanif10 TunnelID: 0x0 Flags: D

Destination: 10.2.8.0/21 Protocol: OSPF Process ID: 19200 Preference: 10 Cost: 81 NextHop: 10.10.0.1 Neighbour: 0.0.0.0 State: Active Adv Age: 30d22h50m33s Tag: 0 Priority: medium Label: NULL QoSInfo: 0x0 IndirectID: 0x0
RelayNextHop: 0.0.0.0 Interface: Vlanif10 TunnelID: 0x0 Flags: D

...

display interfacve with eth-trunk: Eth-Trunk2 current state : UP Line protocol current state : UP Description:link - core-saude Switch Port, Link-type : trunk(configured), PVID : 1, Hash arithmetic : According to SIP-XOR-DIP,Maximal BW: 40G, Current BW: 40G, The Maximum Frame Length is 9216 IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 6413-ab94-59a0 Current system time: 2023-11-27 08:18:42-03:00 Last 300 seconds input rate 380298448 bits/sec, 38240 packets/sec Last 300 seconds output rate 64235528 bits/sec, 20093 packets/sec Input: 48825888039 packets, 52742246838697 bytes Unicast: 47966460072, Multicast: 795402201 Broadcast: 64025766, Jumbo: 21818227555 Discard: 0, Pause: 0 Frames: 0

Total Error: 0 CRC: 0, Giants: 0 Jabbers: 0, Fragments: 0 Runts: 0, DropEvents: 0 Alignments: 0, Symbols: 0 Ignoreds: 0

Output: 29386300826 packets, 21021916296143 bytes Unicast: 29088080133, Multicast: 206571011 Broadcast: 91649683, Jumbo: 9291051991 Discard: 0, Pause: 0

Total Error: 0 Collisions: 0, ExcessiveCollisions: 0 Late Collisions: 0, Deferreds: 0 Buffers Purged: 0

Input bandwidth utilization  : 0.95%
Output bandwidth utilization : 0.16%

PortName Status Weight

40GE0/0/1 UP 1

The Number of Ports in Trunk : 1 The Number of UP Ports in Trunk : 1

LoopBack0 current state : UP Line protocol current state : UP (spoofing) Description:lo0-vrf default Route Port,The Maximum Transmit Unit is 1500 Internet Address is 10.10.0.13/32 Physical is Loopback Current system time: 2023-11-27 08:18:53-03:00 Last 300 seconds input rate 0 bits/sec, 0 packets/sec Last 300 seconds output rate 0 bits/sec, 0 packets/sec Realtime 0 seconds input rate 0 bits/sec, 0 packets/sec Realtime 0 seconds output rate 0 bits/sec, 0 packets/sec Input: 0 packets,0 bytes 0 unicast,0 broadcast,0 multicast 0 errors,0 unknownprotocol Output:0 packets,0 bytes 0 unicast,0 broadcast,0 multicast 0 errors Input bandwidth utilization : 0% Output bandwidth utilization : 0%

dainok commented 10 months ago

Sure, please send in txt/raw format as a separated file. About VLANs: the command is "display vlan", and the output is the one you sent me in the past days?

The total number of VLANs is: 3
--------------------------------------------------------------------------------
U: Up;         D: Down;         TG: Tagged;         UT: Untagged;
MP: Vlan-mapping;               ST: Vlan-stacking;
#: ProtocolTransparent-vlan;    *: Management-vlan;
--------------------------------------------------------------------------------

VID  Type    Ports
--------------------------------------------------------------------------------
1    common  UT:40GE0/0/1(D)    40GE0/0/2(D)    40GE0/0/3(D)    40GE0/0/4(D)
                40GE0/0/5(D)    40GE0/0/6(D)    XGE0/0/1(D)     XGE0/0/3(D)

gabrielat commented 10 months ago

Sure, please send in txt/raw format as a separated file. About VLANs: the command is "display vlan", and the output is the one you sent me in the past days?

The total number of VLANs is: 3
--------------------------------------------------------------------------------
U: Up;         D: Down;         TG: Tagged;         UT: Untagged;
MP: Vlan-mapping;               ST: Vlan-stacking;
#: ProtocolTransparent-vlan;    *: Management-vlan;
--------------------------------------------------------------------------------

VID  Type    Ports
--------------------------------------------------------------------------------
1    common  UT:40GE0/0/1(D)    40GE0/0/2(D)    40GE0/0/3(D)    40GE0/0/4(D)
                40GE0/0/5(D)    40GE0/0/6(D)    XGE0/0/1(D)     XGE0/0/3(D)

Yes, that's it

gabrielat commented 10 months ago

I need from your side:

the output of "display ip routing-table verbose" from node1?

the output of "display vlan all" from node1?

the output of "display interface" with a trunk with more than one aggregated interface (add one interface Eth-Trunk0, no matter if it's down). Please provide also "display eth-trunks"

Here are the files ypu've asked for :-)

display interface.txt display ip routing-table verbose.txt

dainok commented 10 months ago

@gabrielat parsing display vlan is very complex: I'm not able to parse tagged and untagged interfaces. Any chance I can get which VLAN is passing and how on each interface? (display int or display vlan detail?

gabrielat commented 10 months ago

@gabrielat parsing display vlan is very complex: I'm not able to parse tagged and untagged interfaces. Any chance I can get which VLAN is passing and how on each interface? (display int or display vlan detail?

Hi! I've looked the commands available but nothing seems to work properly.. Huawei doesn't have 'display vlan detail' and doesn't show the tagged and untagged vlans on 'display interface'.

From 'display vlan' I could see that tagged ports are preceeded with TG and untagged with UT. For instance:

<rout> display vlan 234
--------------------------------------------------------------------------------
U: Up;         D: Down;         TG: Tagged;         UT: Untagged;
MP: Vlan-mapping;               ST: Vlan-stacking;
#: ProtocolTransparent-vlan;    *: Management-vlan;
--------------------------------------------------------------------------------

VID  Type    Ports                                                          
--------------------------------------------------------------------------------
234  common  UT:XGE1/0/4(U)     XGE1/0/5(U)                                     
             TG:XGE1/0/24(D)                                                    

VID  Status  Property      MAC-LRN Statistics Description      
--------------------------------------------------------------------------------
234  enable  default       enable  disable    vlan234

Note that in section VID Type Ports There is

234  common  UT:XGE1/0/4(U)     XGE1/0/5(U)                                     
             TG:XGE1/0/24(D)

Saying that VLAN 234 is UNTAGGED on ports XGE1/0/4 and on XGE1/0/5, both in Up state, and also TAGGED on por XGE1/0/24 in Down state.

As we can see in:

<rout>display current-configuration interface XGigabitEthernet 1/0/4
#
interface XGigabitEthernet1/0/4
 port link-type access
 port default vlan 234
#
return
<rout>display current-configuration interface XGigabitEthernet 1/0/5
#
interface XGigabitEthernet1/0/5
 port link-type access
 port default vlan 234
#
return
<rout-cs-dc-saude-01>display current-configuration interface XGigabitEthernet 1/0/24
#
interface XGigabitEthernet1/0/24
 port link-type trunk
 port trunk allow-pass vlan 100 234 1823
#
return

I think it's not ideal, but this is the best way I could find..

dainok commented 10 months ago

It’s not ideal, it’s a nightmare! Any other display something that can help? What if you display vlan 10?

On Wed, Nov 29, 2023 at 19:52, gabrielat @.***(mailto:On Wed, Nov 29, 2023 at 19:52, gabrielat < wrote:

@.***(https://github.com/gabrielat) parsing display vlan is very complex: I'm not able to parse tagged and untagged interfaces. Any chance I can get which VLAN is passing and how on each interface? (display int or display vlan detail?

Hi! I've looked the commands available but nothing seems to work properly.. Huawei doesn't have 'display vlan detail' and doesn't show the tagged and untagged vlans on 'display interface'.

From 'display vlan' I could see that tagged ports are preceeded with TG and untagged with UT. For instance:
display vlan 234 -------------------------------------------------------------------------------- U: Up; D: Down; TG: Tagged; UT: Untagged; MP: Vlan-mapping; ST: Vlan-stacking; #: ProtocolTransparent-vlan; *: Management-vlan; -------------------------------------------------------------------------------- VID Type Ports -------------------------------------------------------------------------------- 234 common UT:XGE1/0/4(U) XGE1/0/5(U) TG:XGE1/0/24(D) VID Status Property MAC-LRN Statistics Description -------------------------------------------------------------------------------- 234 enable default enable disable vlan234 Note that in section VID Type Ports There is 234 common UT:XGE1/0/4(U) XGE1/0/5(U) TG:XGE1/0/24(D) Saying that VLAN 234 is UNTAGGED on ports XGE1/0/4 and on XGE1/0/5, both in Up state, and also TAGGED on por XGE1/0/24 in Down state. As we can see in: display current-configuration interface XGigabitEthernet 1/0/4 # interface XGigabitEthernet1/0/4 port link-type access port default vlan 234 # return display current-configuration interface XGigabitEthernet 1/0/5 # interface XGigabitEthernet1/0/5 port link-type access port default vlan 234 # return display current-configuration interface XGigabitEthernet 1/0/24 # interface XGigabitEthernet1/0/24 port link-type trunk port trunk allow-pass vlan 100 234 1823 # return I think it's not ideal, but this is the best way I could find.. — Reply to this email directly, [view it on GitHub](https://github.com/dainok/netdoc/issues/102#issuecomment-1832511460), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ACGI3ZD44AWEMLV2FAUNJ3DYG6AAXAVCNFSM6AAAAAA7BQ7M5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSGUYTCNBWGA). You are receiving this because you were mentioned.Message ID: ***@***.***>

dainok commented 10 months ago

@gabrielat seems I successfully parsed VLANs. Now VRFs (VPN in Huawei language). Where I can see which VRF is associated to each interface?

dainok commented 10 months ago

@gabrielat another question: in display mac / arp / route, how are displayed 40GE and Trunk interfaces? Can you post a short snippet?

dainok commented 10 months ago

@gabrielat I completed most of the required tasks. I need:

a VRF example: where I can find VRF - interface association? I display interface / display ip route / display arp with a VRF
in display mac / arp / route, how are displayed 40GE and Trunk interfaces? Can you post a short snippet?

dainok commented 10 months ago

@gabrielat based on templates, you should have this command: display port vlan Please send me the output from ROUTER1.

gabrielat commented 10 months ago

@gabrielat based on templates, you should have this command: display port vlan Please send me the output from ROUTER1.

here is the display port vlan return display port vlan.txt

gabrielat commented 10 months ago

@gabrielat I completed most of the required tasks. I need:

a VRF example: where I can find VRF - interface association? I display interface / display ip route / display arp with a VRF

in display mac / arp / route, how are displayed 40GE and Trunk interfaces? Can you post a short snippet?

These are the options on display interface

<router1> display interface ?
  40GE              40GE interface
  Eth-Trunk         Ethernet-Trunk interface
  Ethernet          Ethernet interface
  LoopBack          LoopBack interface
  MEth              MEth interface
  NULL              NULL interface
  Vlanif            Vlan interface
  XGigabitEthernet  XGigabitEthernet interface
  brief             Summary information about the interface status and
                    configuration
  description       Interface description
  main              Main interface
  slot              Slot number
  |                 Matching output
  <cr>

display ip routing-table vpn-instance tel.txt display arp vpn-instance tel.txt

dainok commented 10 months ago

If you assign an interface to a vrf, can you see the VRF on display ip interface or in display interface?

dainok commented 10 months ago

@gabrielat Beside the question about VRF (see above) I finished. I need you:

set up a testing instance
try to discover and ingest your infrastructure
report any error

You must use:

https://github.com/dainok/netdoc/tree/add-huawei (add-huawei branch) for NetDoc
https://github.com/dainok/ntc-templates/tree/improve-huawei-vrp (improve-huawei branch) for NTC Templates

After you confirm everything is ok, I can create a PR for NTC templates and release NetDoc.

gabrielat commented 10 months ago

If you assign an interface to a vrf, can you see the VRF on display ip interface or in display interface?

I've added an IP and VPN-Instance on a port and got this:

[router1-40GE0/0/3]display this 
#
interface 40GE0/0/3
 undo portswitch
 description livre
 ip binding vpn-instance met
 ip address 172.20.40.1 255.255.255.252
#
return

[router1-40GE0/0/3]display interface 40GE 0/0/3   
40GE0/0/3 current state : DOWN
Line protocol current state : DOWN
Description:livre
Route Port,The Maximum Frame Length is 9216
Internet Address is 172.20.40.1/30
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 6413-ab94-59a4
Last physical up time   : -
Last physical down time : 2023-10-27 09:24:35 UTC-03:00
Current system time: 2023-12-05 08:16:02-03:00
Port Mode: COMMON FIBER
Speed : 40000,  Loopback: NONE
Duplex: FULL,   Negotiation: DISABLE
Mdi   : -,      Flow-control: DISABLE
Last 300 seconds input rate 0 bits/sec, 0 packets/sec
Last 300 seconds output rate 0 bits/sec, 0 packets/sec
Input peak rate 0 bits/sec, Record time: -
Output peak rate 0 bits/sec, Record time: -

Input:  0 packets, 0 bytes
  Unicast:                          0,  Multicast:                           0
  Broadcast:                        0,  Jumbo:                               0
  Discard:                          0,  Pause:                               0

  Total Error:                      0
  CRC:                              0,  Giants:                              0
  Runts:                            0,  Fragments:                           0
  Alignments:                       0,  Symbols:                             0
  Ignoreds:                         0

Output:  0 packets, 0 bytes
  Unicast:                          0,  Multicast:                           0
  Broadcast:                        0,  Jumbo:                               0
  Discard:                          0,  Pause:                               0

    Input bandwidth utilization threshold : 80.00%
    Output bandwidth utilization threshold: 80.00%
    Input bandwidth utilization  :    0%
    Output bandwidth utilization :    0%

[router1-40GE0/0/3]display ip interface 40GE 0/0/3
40GE0/0/3 current state : DOWN 
Line protocol current state : DOWN
The Maximum Transmit Unit : 1500 bytes
input packets : 0, bytes : 0, multicasts : 0
output packets : 0, bytes : 0, multicasts : 0
Directed-broadcast packets:
 received packets:            0, sent packets:            0 
 forwarded packets:           0, dropped packets:           0 
ARP packet input number:           0
  Request packet:                  0
  Reply packet:                    0
  Unknown packet:                  0
Internet Address is 172.20.40.1/30 
Broadcast address : 172.20.40.3
TTL being 1 packet number:         0
TTL invalid packet number:         0
ICMP packet input number:          0
  Echo reply:                      0
  Unreachable:                     0
  Source quench:                   0
  Routing redirect:                0
  Echo request:                    0
  Router advert:                   0
  Router solicit:                  0
  Time exceed:                     0      
  IP header bad:                   0
  Timestamp request:               0
  Timestamp reply:                 0
  Information request:             0
  Information reply:               0
  Netmask request:                 0
  Netmask reply:                   0
  Unknown type:                    0

gabrielat commented 10 months ago

I found this command that may be helpfull:

[router1-40GE0/0/3]display ip vpn-instance interface 
 Total VPN-Instances configured      : 3

 VPN-Instance Name and ID : cftv, 1
  Interface Number : 3 
  Interface list : Vlanif1823, 
                   Vlanif1847, 
                   LoopBack5

 VPN-Instance Name and ID : met, 2
  Interface Number : 5 
  Interface list : 40GE0/0/3, 
                   Vlanif1774, 
                   Vlanif1797, 
                   LoopBack2, 
                   Vlanif1769

 VPN-Instance Name and ID : tel, 3
  Interface Number : 4 
  Interface list : Vlanif755, 
                   Vlanif773, 
                   Vlanif797, 
                   LoopBack1

dainok commented 10 months ago

This is useful, thank you. Let me integrate it before you start the tests

dainok commented 10 months ago

@gabrielat in the example you posted, if you type "display ip routing-table verbose", do you see the line "Routing Table(s): VRFNAME" for each VRF?

gabrielat commented 10 months ago

@gabrielat in the example you posted, if you type "display ip routing-table verbose", do you see the line "Routing Table(s): VRFNAME" for each VRF?

No, it shows only "Routing Tables: Public"

gabrielat commented 10 months ago

But you can view separatelly using:

display ip routing-table vpn-instance VRFNAME verbose

dainok commented 10 months ago

@gabrielat ok go with tests. You can experiment errors on discover phase, I couldn't test it on a lab.

gabrielat commented 10 months ago

Hi @dainok ! Sorry, but I don't know how to add these on my installation. Can you tell me what I need to do? Thanks

dainok commented 10 months ago

Follow https://github.com/dainok/netdoc/wiki/Developing-NetDoc But on NTC template, use my url and my branch instead of the official one.

gabrielat commented 10 months ago

@dainok trying to run for the first time, got this error:

An exception occurred: `ValidationError: ['display current-configuration | include sysname', 'HOSTNAME'] is not of type 'string'

Failed validating 'type' in schema['properties']['command']: {'type': 'string'}

On instance['command']: ['display current-configuration | include sysname', 'HOSTNAME']`

Traceback (most recent call last):
  File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 264, in run
    output = discovery(
  File "/root/src/netdoc/netdoc/tasks.py", line 118, in discovery
    module.discovery(filtered_devices, filters=filters, filter_type=filter_type)
  File "/root/src/netdoc/netdoc/discoverers/netmiko_huawei_vrp.py", line 65, in discovery
    discoverylog.create(
  File "/root/src/netdoc/netdoc/schemas/discoverylog.py", line 93, in create
    validate(kwargs, get_schema_create(), format_checker=FormatChecker())
  File "/opt/netbox/venv/lib/python3.10/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: ['display current-configuration | include sysname', 'HOSTNAME'] is not of type 'string'

Failed validating 'type' in schema['properties']['command']:
    {'type': 'string'}

On instance['command']:
    ['display current-configuration | include sysname', 'HOSTNAME']

dainok commented 10 months ago

@gabrielat fixed (hopefully). can you update netdoc and test again?

gabrielat commented 10 months ago

@gabrielat fixed (hopefully). can you update netdoc and test again?

@dainok it's running now

dainok commented 10 months ago

@gabrielat everything is working? any error?

gabrielat commented 10 months ago

@dainok the discovery ran ok, the ingest returned an error due to IPv6:

An exception occurred: `ValidationError: '2804:1f20::20' is not a 'ipv4'

Failed validating 'format' in schema['properties']['address']: {'format': 'ipv4', 'type': 'string'}

On instance['address']: '2804:1f20::20'`

Traceback (most recent call last):
  File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 357, in run
    log_ingest(log)
  File "/root/src/netdoc/netdoc/utils.py", line 399, in log_ingest
    module.ingest(log)
  File "/root/src/netdoc/netdoc/ingestors/netmiko_huawei_vrp_display_lldp_neighbor.py", line 86, in ingest
    remote_discoverable_list += discoverable.get_list(
  File "/root/src/netdoc/netdoc/schemas/discoverable.py", line 91, in get_list
    validate(kwargs, get_schema(), format_checker=FormatChecker())
  File "/opt/netbox/venv/lib/python3.10/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: '2804:1f20::20' is not a 'ipv4'

Failed validating 'format' in schema['properties']['address']:
    {'format': 'ipv4', 'type': 'string'}

On instance['address']:
    '2804:1f20::20'

dainok commented 10 months ago

@gabrielat I see you are using IPv6. I'm not implemented it yet in NetDoc. IPv4 can be fine for now?

gabrielat commented 10 months ago

@gabrielat I see you are using IPv6. I'm not implemented it yet in NetDoc. IPv4 can be fine for now?

Yes, but I ran netdoc on IPv4 addresses..

Running task 'multiple_tasks' with args {} on 7 hosts
Host '143.54.0.7': running task 'multiple_tasks'
Host '143.54.0.7': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.12': running task 'multiple_tasks'
Host '143.54.0.12': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.13': running task 'multiple_tasks'
Host '143.54.0.13': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.17': running task 'multiple_tasks'
Host '143.54.0.17': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.22': running task 'multiple_tasks'
Host '143.54.0.22': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.23': running task 'multiple_tasks'
Host '143.54.0.23': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.24': running task 'multiple_tasks'
Host '143.54.0.24': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.7': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.12': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.17': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.13': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.23': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.22': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.24': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.17': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.17': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.12': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.17': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.12': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.17': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.23': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.12': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.23': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.12': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.23': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.24': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.23': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.24': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.24': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.24': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.22': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.7': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.22': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.13': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.7': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.22': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.12': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.23': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.13': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.7': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.12': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.17': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.13': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.22': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.12': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.7': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.23': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.17': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.23': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.17': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.24': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.17': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.24': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.13': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.17': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.24': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.12': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.17': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.12': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.23': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.23': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.12': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.17': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.12': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.24': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.24': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.24': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.23': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.24': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.23': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.22': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.22': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.22': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.22': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.7': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.22': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.7': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.22': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.7': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.22': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.7': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.13': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.13': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.13': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.7': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.7': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.7': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.13': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.13': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.13': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.13': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.7': running task 'additional_tasks'
Host '143.54.0.7': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.12': running task 'additional_tasks'
Host '143.54.0.12': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.13': running task 'additional_tasks'
Host '143.54.0.13': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.17': running task 'additional_tasks'
Host '143.54.0.17': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.22': running task 'additional_tasks'
Host '143.54.0.22': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.23': running task 'additional_tasks'
Host '143.54.0.23': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.24': running task 'additional_tasks'
Host '143.54.0.24': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'

dainok commented 10 months ago

ok, let me think how I can skip IPv6 for now. In the meanwhile you can ingest all logs from CLI using a netbox shell and https://github.com/dainok/netdoc/blob/master/scripts/log_ingest.py

dainok commented 10 months ago

@gabrielat update netdoc and try again

dainok / netdoc