search

netdisco / netdisco

A web-based network management tool.
http://netdisco.org/
BSD 3-Clause "New" or "Revised" License
700 stars 81 forks source link

arpnip support for unifi via linux ssh #1267

Open lameslime opened 3 months ago

lameslime commented 3 months ago 
device_auth:
  - tag: snmp3-dev (main for other devices)
    user: snmp3_dev
    auth:
      pass: password
      proto: SHA
    priv:
       pass: crypt
       proto: AES
  - tag: unifi-devices (collector for unifi ap's)
    # driver: 'SNMP::Info::Layer2::Ubiquiti'
    arpnip_no:
     - 0.0.0.0/0
    user: snmp3-unifi
    auth:
       pass: password
      proto: SHA
    priv:
      pass: password
      proto: AES
  - tag: unifi-devices-ssh (want arpnip from this)
    driver: cli
    platform: Linux
    only: 'os:UniFi'
    username: username
    password: password
    arp_command: '/sbin/arp'
    ssh_master_opts:
     - '-o'
     - 'StrictHostKeyChecking=no'

Snmp discovery works fine, but for unifi ap's it fails to do arpnip and macsuck (finds interfaces but no macs). So I set up a ssh collector, but i always get skip: arp table data supplied by other source. If i run arp via ssh manually I get results.

Would it be possible that my config is wrong? I also tried 'os:UniFi' instead of '0.0.0.0/0'.

I also get this error when i added the ssh collector.

Can't locate object method "macsuck" via package "App::Netdisco::SSHCollector::Platform::Linux" at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SSH.pm line 58.
debug ⬅ (error) Can't locate object method "macsuck" via package "App::Netdisco::SSHCollector::Platform::Linux" at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SSH.pm line 58.

Info:

Software    Version
App::Netdisco   2.80.3
SNMP::Info  3.972.2
DB Schema   89
PostgreSQL  13.00.4
Perl / Python   5.38.2 / 3.12.7

Most of the config is default, except some auth parameters for proxy authentication. If you need any other info let me know.

ollyg commented 2 months ago

Thanks @lameslime I've opened this as a bug report, so we can see what's going wrong. Are you able to share the output of netdisco-do arpnip -D for the device?

lameslime commented 2 months ago

I also ran with other debug parameters (-DISC) and I only got more detailed SNMP collection logs. SSH collection fails early with (info) skip: arp table data supplied by other source.

~ $ netdisco-do arpnip -D -d 172.16.10.101
[35052] 2024-12-18 07:57:22  info App::Netdisco version 2.080003 loaded.
[35052] 2024-12-18 07:57:22  info arpnip: [172.16.10.101] started at Wed Dec 18 07:57:22 2024
[35052] 2024-12-18 07:57:22 debug arpnip: running with timeout 600s
[35052] 2024-12-18 07:57:22 debug //// CHECK \\\\ phase
[35052] 2024-12-18 07:57:22 debug ⮕ worker Internal::BackendFQDN p1000000
[35052] 2024-12-18 07:57:22 debug ⮕ worker Internal::SNMPFastDiscover p1000000
[35052] 2024-12-18 07:57:22 debug running with configured SNMP timeouts
[35052] 2024-12-18 07:57:22 debug ⮕ worker Arpnip p0
[35052] 2024-12-18 07:57:22 debug ⬅ (done) arpnip is able to run
[35052] 2024-12-18 07:57:22 debug //// EARLY \\\\ phase
[35052] 2024-12-18 07:57:22 debug ⮕ worker Arpnip::Nodes p0 "prepare common data"
[35052] 2024-12-18 07:57:22 debug //// MAIN \\\\ phase
[35052] 2024-12-18 07:57:22 debug ⮕ worker Arpnip::Nodes p1000000
[35052] 2024-12-18 07:57:22 debug ⬅ (info) skip: arp table data supplied by other source
[35052] 2024-12-18 07:57:22 debug ⮕ worker Arpnip::Nodes p200
[35052] 2024-12-18 07:57:22 debug cli session cache warm: [172.16.10.101]
[35052] 2024-12-18 07:57:23 debug 172.16.10.101 35052 arpnip()
[35052] 2024-12-18 07:57:38 debug ⬅ (done) Gathered arp caches from 172.16.10.101
[35052] 2024-12-18 07:57:38 debug ⮕ worker Arpnip::Nodes p100
[35052] 2024-12-18 07:57:38 debug ⬅ (info) skip: namespace passed at higher priority
[35052] 2024-12-18 07:57:38 debug ⮕ worker Arpnip::Subnets p100
[35052] 2024-12-18 07:57:38 debug snmp reader cache warm: [172.16.10.101]
[35052] 2024-12-18 07:57:38 debug [172.16.10.101:161] try_connect with v: 3, t: 0.2, r: 0, class: SNMP::Info::Layer2::Ubiquiti, comm: <hidden>
[35052] 2024-12-18 07:57:38 debug  [172.16.10.101] arpnip - found subnet 172.16.10.0/24
[35052] 2024-12-18 07:57:38 debug ⬅ (info)  [172.16.10.101] arpnip - processed 1 Subnet entries
[35052] 2024-12-18 07:57:38 debug //// STORE \\\\ phase
[35052] 2024-12-18 07:57:38 debug ⮕ worker Arpnip::Nodes p0
[35052] 2024-12-18 07:57:38 debug  resolving 0 ARP entries with max 50 outstanding requests
[35052] 2024-12-18 07:57:38 debug  [172.16.10.101] arpnip - processed 0 ARP Cache entries
[35052] 2024-12-18 07:57:38 debug  [172.16.10.101] arpnip - processed 0 IPv6 Neighbor Cache entries
[35052] 2024-12-18 07:57:38 debug ⬅ (done) Ended arpnip for 172.16.10.101
[35052] 2024-12-18 07:57:38 debug //// LATE \\\\ phase
[35052] 2024-12-18 07:57:38 debug ⮕ worker Arpnip::Hooks p0
[35052] 2024-12-18 07:57:38 debug ⬅ (info)  [172.16.10.101] hooks - 0 queued
[35052] 2024-12-18 07:57:38  info arpnip: finished at Wed Dec 18 07:57:38 2024
[35052] 2024-12-18 07:57:38  info arpnip: status done: Ended arpnip for 172.16.10.101
ollyg commented 2 months ago

Hi @lameslime thanks for the debug output. Actually it looks like Netdisco is working fine, and for some reason no ARP data is coming back from the command line or SSH module.

You can ignore the "skip" lines, they are for other methods of gathering ARP data (like from a file or SNMP). The SSH module is running:

[35052] 2024-12-18 07:57:22 debug ⮕ worker Arpnip::Nodes p200
[35052] 2024-12-18 07:57:22 debug cli session cache warm: [172.16.10.101]
[35052] 2024-12-18 07:57:23 debug 172.16.10.101 35052 arpnip()
[35052] 2024-12-18 07:57:38 debug ⬅ (done) Gathered arp caches from 172.16.10.101

Looking at the source of the Linux SSH module, it runs /sbin/arp -n | tail -n +2 and then matches the output against regular expression ([0-9\.]+)\s+ether\s+([a-fA-F0-9:]+) so I wonder if either of those is failing.

I'm sorry the debug on SSH isn't great. I hope we can improve it as we migrate from the Perl backend to Python but that's a bit of work, still.

lameslime commented 2 months ago

Thanks for some info. I didn't realize Unifi's arp response is different, so it failed the regex. I made an awk that converts the output to look like regular arp and with regex test, it also matches. /sbin/arp | awk '{gsub(/[\(\)]/, "", $2); print $2, "ether", $4, "C", $7}'

image

But it still doesn't seem to do anything, am I being dumb and forgot that i need macsuck for arpnip to work? Definitely takes a bit longer (on debug 172.16.10.101 35523 arpnip()) for arpnip command to finish now, logs look identical. Edit: netdisco-do show -d ip -e arp doesn't list anything

ollyg commented 1 month ago

Hi @lameslime no you don't need macsuck for arpnip to work. They work independently and put separate info into the database. If you want to search on an IP and find a device port, you'll need both. But you can still search on ARP IP data or MAC data independently.

The other thing I could do to help is provide me the raw output of the command line show command on Unifi. I might be able to mock it up here, I'm not sure.

lameslime commented 1 month ago

Sure, model shouldn't matter as they all basically run the same os. I have ap management on untaged vlan and 2 wifi networks on tagged 10 and 11 vlans. There are a lot of interfaces, but seems like everything other than brX can be ignored. Let me know if you need more cli outputs.

NAME-AP-A4-BZ.7.0.103# arp
? (10.10.10.105) at 50:98:39:e1:85:f6 [ether]  on br0.10
? (10.10.10.115) at 5c:d0:6e:e7:c7:aa [ether]  on br0.10
? (10.10.11.194) at 7c:87:ce:8c:c8:07 [ether]  on br0.11
? (10.10.11.189) at e0:5a:1b:60:99:3c [ether]  on br0.11
? (10.10.10.108) at f8:63:3f:9c:2d:59 [ether]  on br0.10
? (10.10.11.201) at e4:65:b8:58:48:30 [ether]  on br0.11
? (10.10.100.1) at 2c:c8:1b:97:09:3c [ether]  on br0
? (10.10.10.110) at 04:7c:16:c5:7a:40 [ether]  on br0.10
NAME-AP-A4-BZ.7.0.103# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: miireg: <> mtu 0 qdisc noop state DOWN group default qlen 1000
    link/generic 
3: eth0: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb master br0 state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:41 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9e05:d6ff:fe49:3741/64 scope link 
       valid_lft forever preferred_lft forever
4: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
5: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
6: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
    link/ether de:a7:ae:f6:a2:bc brd ff:ff:ff:ff:ff:ff
7: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
    link/ether 7e:90:1f:d6:42:70 brd ff:ff:ff:ff:ff:ff
8: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
9: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
11: ip6gre0@NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default qlen 1000
    link/gre6 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
12: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 66:9e:32:85:53:cf brd ff:ff:ff:ff:ff:ff
13: teql0: <NOARP> mtu 1500 qdisc noop state DOWN group default qlen 100
    link/void 
14: mld-wifi0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
15: wifi0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 2699
    link/ieee802.11 9c:05:d6:49:37:42 brd ff:ff:ff:ff:ff:ff
16: soc0: <> mtu 0 qdisc noop state DOWN group default qlen 1000
    link/ieee802.11 
17: wifi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 2699
    link/ieee802.11 9c:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
18: wifi2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 2699
    link/ieee802.11 9c:05:d6:49:37:44 brd ff:ff:ff:ff:ff:ff
19: soc1: <> mtu 0 qdisc noop state DOWN group default qlen 1000
    link/ieee802.11 
20: ath4: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb master br0 state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:42 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9e05:d6ff:fe49:3742/64 scope link 
       valid_lft forever preferred_lft forever
21: ath5: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
    link/ether a2:05:d6:49:37:42 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a005:d6ff:fe49:3742/64 scope link 
       valid_lft forever preferred_lft forever
22: vwire2: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb master br0 state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
23: ath1: <BROADCAST,MULTICAST,ALLMULTI,PROMISC> mtu 1500 qdisc htb master br0 state DOWN group default qlen 1000
    link/ether a2:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
24: ath0: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb master br0 state UP group default qlen 1000
    link/ether a6:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a405:d6ff:fe49:3743/64 scope link 
       valid_lft forever preferred_lft forever
25: ath7: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb master br0 state UP group default qlen 1000
    link/ether aa:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a805:d6ff:fe49:3743/64 scope link 
       valid_lft forever preferred_lft forever
26: vwire3: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb master br0 state UP group default qlen 1000
    link/ether 9e:05:d6:49:37:49 brd ff:ff:ff:ff:ff:ff
27: ath6: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb master br0 state UP group default qlen 1000
    link/ether 9e:05:d6:49:37:44 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9c05:d6ff:fe49:3744/64 scope link 
       valid_lft forever preferred_lft forever
28: ath8: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
    link/ether 9e:05:d6:49:37:45 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9c05:d6ff:fe49:3745/64 scope link 
       valid_lft forever preferred_lft forever
29: eth0.10@eth0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.10 state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:41 brd ff:ff:ff:ff:ff:ff
30: ath1.10@ath1: <BROADCAST,MULTICAST,ALLMULTI,M-DOWN> mtu 1500 qdisc noqueue master br0.10 state DOWN group default qlen 1000
    link/ether a2:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
31: vwire2.10@vwire2: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.10 state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
32: vwire3.10@vwire3: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.10 state UP group default qlen 1000
    link/ether 9e:05:d6:49:37:49 brd ff:ff:ff:ff:ff:ff
33: eth0.11@eth0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.11 state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:41 brd ff:ff:ff:ff:ff:ff
34: ath1.11@ath1: <BROADCAST,MULTICAST,ALLMULTI,M-DOWN> mtu 1500 qdisc noqueue master br0.11 state DOWN group default qlen 1000
    link/ether a2:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
35: vwire2.11@vwire2: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.11 state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
36: vwire3.11@vwire3: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.11 state UP group default qlen 1000
    link/ether 9e:05:d6:49:37:49 brd ff:ff:ff:ff:ff:ff
37: br0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:41 brd ff:ff:ff:ff:ff:ff
    inet 10.10.100.104/24 brd 10.10.100.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::9e05:d6ff:fe49:3741/64 scope link 
       valid_lft forever preferred_lft forever
38: br0.11: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:41 brd ff:ff:ff:ff:ff:ff
39: br0.10: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:05:d6:49:37:41 brd ff:ff:ff:ff:ff:ff
40: pd99: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ieee802.11/radiotap a6:05:d6:49:37:42 brd ff:ff:ff:ff:ff:ff
41: ath8.10@ath8: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.10 state UP group default qlen 1000
    link/ether 9e:05:d6:49:37:45 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9c05:d6ff:fe49:3745/64 scope link 
       valid_lft forever preferred_lft forever
42: ath5.11@ath5: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.11 state UP group default qlen 1000
    link/ether a2:05:d6:49:37:42 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a005:d6ff:fe49:3742/64 scope link 
       valid_lft forever preferred_lft forever
44: ath7.10@ath7: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.10 state UP group default qlen 1000
    link/ether aa:05:d6:49:37:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a805:d6ff:fe49:3743/64 scope link 
       valid_lft forever preferred_lft forever
ollyg commented 1 month ago

Hi @lameslime I am releasing a new version 2.082002 of netdisco with something to try...

You'll need to remove your existing device_auth stanza for the unifi box ("unifi-devices-ssh") and add:

device_auth:
  - tag: unifi-devices-ssh
    driver: cli
    only: 172.16.10.101
    action: 'arpnip::nodes'
    username: foo
    password: bar
    device_type: linux

you will also need to add the following config to your config file:

extra_python_worker_package_namespace: 'netdisco'
extra_python_worker_plugins:
  - 'arpnip.nodes.main.cli.linux':
      only: 'os:UniFi'

run arpnip with debug and let me know how it goes!

if it doesn't go well, you could try adding the following configuration to get verbose logging of the cli stuff:

netmiko_logging: true
paramiko_logging: true