Enable NMAP prodding to double-check device Status to circumvent arp cache

[jokob-sk]

          I have got similar situation. Version: v23.9.11 (hosted on docker).

Kernel:Linux

System:GNU/Linux

Operating System:Debian GNU/Linux 11 (bullseye)

Uname:Linux portainer 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:14:14 UTC 2023 x86_64 GNU/Linux
Architecture:x86_64

That particular device is currently down. But interface shows it's online.

While I do a scan:

Starting Nmap 7.80 ( https://nmap.org ) at 2023-09-12 08:59 UTC Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn Nmap done: 1 IP address (0 hosts up) scanned in 3.06 seconds

In my case I feel like first scan of network just stucks and shows always the same info. Way I run Pi.Alert on docker is:

docker run -d --network=host \ --name=pialert \ -v /home/zawier/Docker/PiAlert/config:/home/pi/pialert/config \ -v /home/zawier/Docker/PiAlert/db:/home/pi/pialert/db \ -e TZ=Europe/Warsaw \ -e PORT=20211 \ -e HOST_USER_ID=1000 \ -e HOST_USER_GID=1000 \ jokobsk/pi.alert:latest

To be sure I have pulled from docker hub lastes version (which was updated yesterday).

Originally posted by @zawier-git in https://github.com/jokob-sk/Pi.Alert/issues/428#issuecomment-1715340971

[jokob-sk]

Hey @zawier-git!

What scan methods are you using?

1. Ideally, post your pialert.conf file with personal info redacted.
2. Please post pialert log of the section where the scan is performed.

Thanks, j

[ace16vitamine]

I saw this in another way today: The device was up but reported as down... :-) But I was not sure if this comes from an active Qualys MAP Scan, can be that my NIC was busy

[zawier-git]

Hi Jokob :)

First of all thank you for your awesome work!!

What scan methods are you using?

In my example was fast scan:

Starting Nmap 7.80 ( https://nmap.org ) at 2023-09-12 11:53 UTC
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.07 seconds

Normal scan returns as well:

Starting Nmap 7.80 ( https://nmap.org ) at 2023-09-12 11:56 UTC
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.06 seconds

Detailed the same :

Starting Nmap 7.80 ( https://nmap.org ) at 2023-09-12 11:57 UTC
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.57 seconds

1. Ideally, post your pialert.conf file with personal info redacted.

pialert_conf.txt

2. Please post pialert log of the section where the scan is performed.

pialert_log.txt

In above log ip & mac adresses were also edited. Please let me know if that's helpfull.If you need futher information, just don't hesitate to ask ;) . I would be happy to help.

Last thing which I believe is interesting is fact that when I have run WatchYourLan image from docker hub (to compare results with Pi.Alert) still I have my offline device as online:

So it might be something wrong with my network I guess..

[jokob-sk]

Hi @zawier-git !

Thanks for the awesome logs & details. This helps a lot.

It looks like the plugin that is marking the device as online is ARPSCAN:

14:06:51 [Plugins] ---------------------------------------------
14:06:51 [Plugins] display_name: Arp-Scan (Network scan)
14:06:51 [Plugins] Executing: python3 /home/pi/pialert/front/plugins/arp_scan/script.py userSubnets={subnets}
14:06:51 [ARP Scan] In script 
14:06:51 [ARP Scan] values.userSubnets: ["userSubnets=b'someHash='"]
14:06:51 [ARP Scan] userSubnetsParamBase64: 'someHash='
14:06:51 [ARP Scan] userSubnetsParam: XX.XX.10.0/24 --interface=myinft
14:07:02 [ARP Scan] arpscan_output: Interface: myinft, type: EN10MB, MAC: 11:22:33:44:55:66, IPv4: XX.XX.10.100
Starting arp-scan 1.9.7 with 256 hosts (https://github.com/royhills/arp-scan)
XX.XX.10.2  a1:a1:a1:a1:a1:a1   GIGA-BYTE TECHNOLOGY CO.,LTD.
XX.XX.10.4  d1:d1:d1:d1:d1:d1   (Unknown: locally administered)
XX.XX.10.1  b1:b1:b1:b1:b1:b1   TP-Link Corporation Limited
XX.XX.10.3  c1:c1:c1:c1:c1:c1   (Unknown: locally administered)
XX.XX.10.5  g1:g1:g1:g1:g1:g1    Dell Inc.
XX.XX.10.10 aa:bb:cc:dd:ee:ff   GIGA-BYTE TECHNOLOGY CO.,LTD.
XX.XX.10.22 bb:cc:dd:ee:ff:gg   Shenzhen Century Xinyang Technology Co., Ltd

21 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9.7: 256 hosts scanned in 11.181 seconds (22.90 hosts/sec). 7 responded

14:07:02 [ARP Scan] Found: Devices without duplicates 7

I assume it's one of these two devices:

XX.XX.10.2  a1:a1:a1:a1:a1:a1   GIGA-BYTE TECHNOLOGY CO.,LTD.
XX.XX.10.10 aa:bb:cc:dd:ee:ff   GIGA-BYTE TECHNOLOGY CO.,LTD.

You can try to run arp-scan directly in the container and play around with the parameters, but I can't do much if arp-scan itself is reporting the device as online:

sudo arp-scan  --ignoredups --retry=6 192.168.1.0/24 --interface eth1

I could implement a special "NMAP prodding" mechanism to double-check if a host really is online in the future.

I also checked and apparently, arp-scan uses some kind of cache - try to experiment with this and let me know as I can improve the accuracy of the results later:

arp-scan primarily relies on the ARP cache to identify devices on a local network. However, if you want to perform an ARP scan without relying on the ARP cache, you can use the following options:

1. Clear the ARP Cache: Before running arp-scan, you can clear the ARP cache on your system. This ensures that the ARP cache is empty, and arp-scan will build its own list of devices based on the ARP responses it receives during the scan. You can clear the ARP cache using the arp command on Linux:

   sudo ip -s -s neigh flush all

   After running this command, the ARP cache will be cleared, and you can proceed with arp-scan.

2. Disable ARP Cache: Some operating systems allow you to temporarily disable ARP caching. On Linux, you can do this with the following command:

   sudo sysctl -w net.ipv4.neigh.default.gc_stale_time=0

   This disables the ARP cache, and ARP requests and responses will be used exclusively by arp-scan to identify devices. However, keep in mind that this approach may affect normal network operations, so use it with caution and reset the ARP cache settings when you're done.

Please note that manipulating the ARP cache in this way can have network-wide implications, so be careful when using these commands, especially on production networks. Always ensure that you have the necessary permissions to perform such operations, as they typically require administrative privileges (sudo).

Additionally, running arp-scan without an ARP cache may generate a high amount of ARP traffic on your network, which could potentially cause network congestion or other issues, so use it judiciously.

[zawier-git]

Thank you @jokob-sk for a detailed instruction. I have done some test as you adviced, and here I come with results:

`root@portainer:/#  ip -s -s neigh flush all
10.10.10.22 dev ens18 lladdr mac_addr_1 ref 1 used 19/3/19 probes 4 REACHABLE
2.17.0.6 dev docker0 lladdr mac_addr_2  used 1423/1483/1423 probes 0 STALE
2.17.0.4 dev docker0  used 1427/1490/1427 probes 6 FAILED
2.17.0.2 dev docker0 lladdr mac_addr_3 used 1398/1398/1362 probes 1 STALE
10.10.10.10 dev ens18 lladdr mac_addr_4 ref 1 used 22/0/21 probes 1 REACHABLE
2.18.0.2 dev br-16aa0d4bf229 lladdr mac_addr_5 used 1329/1329/1309 probes 1 STALE
2.17.0.7 dev docker0 lladdr mac_addr_6  used 1418/1418/1383 probes 1 STALE
2.17.0.3 dev docker0 lladdr mac_addr_7 used 113/113/72 probes 1 STALE
10.10.10.1 dev ens18 lladdr mac_addr_8  ref 1 used 19/3/19 probes 1 REACHABLE
Failed to send flush request: Operation not permitted
root@portainer:/#  arp-scan  --ignoredups --retry=6 10.10.10.0/24 --interface ens18
Interface: ens18, type: EN10MB, MAC: mac_addr_13 IPv4: 10.10.10.100
Starting arp-scan 1.9.7 with 256 hosts (https://github.com/royhills/arp-scan)
10.10.10.2    mac_addr_3      GIGA-BYTE TECHNOLOGY CO.,LTD.
10.10.10.1     mac_addr_8        TP-Link Corporation Limited
10.10.10.3     mac_addr_9     (Unknown: locally administered)
10.10.10.4     mac_addr_10      (Unknown: locally administered)
10.10.10.5     mac_addr_11      Dell Inc.
10.10.10.10    mac_addr_4       GIGA-BYTE TECHNOLOGY CO.,LTD.
10.10.10.22    mac_addr_ 12     Shenzhen Century Xinyang Technology Co., Ltd`

Another device (10.10.10.22) is currently offline, yet it remains discoverable. Initially, I suspected this anomaly might be due to residual power consumption on the mainboard. This could explain why the 'Shenzhen Century Xinyang Technology Co., Ltd' WiFi dongle was still detectable. Some of my devices are configured to enter low-power modes for energy conservation (to enable Wake-On-LAN functionality). To test this hypothesis, I disconnected the dongle and conducted another test, which yielded the same result. I then unplugged the power cord from the wall socket, and again, the device remained visible.

So straight to conclusion - it's pure magic after clearing a cache:)

I also considered whether network topology could be a contributing factor. In my setup, I have multiple routers, with each preceding one serving as the WAN interface for the next. This cascading configuration repeats several times, and each router connects to various other devices. One of these routers, specifically a NanoPi R4S running FriendlyWrt (a fork of OpenWrt) with Docker and another instance of Pi.Alert, presented a particular challenge. Another instance of Pi.Alert, located on a different subnet under Proxmox, running under Ubuntu and managed via Portainer, is hidden behind firewalls. While this specific device can establish a connection to the NanoPi, communication in the opposite direction appears to be restricted.

It's possible that the complex network topology is causing difficulties, as there may be challenges in establishing and maintaining all required network paths."

I could implement a special "NMAP prodding" mechanism to double-check if a host really is online in the future.

A such dubble check is more than welcome!

[jokob-sk]

Alright, so to sum up - clearing the cache is difficult and even then results are not correct. The only way to accurately discover offline devices is (As far as I know) NMAP prodding (a quick NMAP scan). Agreed?

[zawier-git]

Agreed!

[bwirt]

I'm also having this issue. I am using Unifi scan (though I think ARP is also running), and I have a switch that shows up as "online" even though it hasn't been plugged in for months. This switch has never been plugged in since I installed pi.alert, so I'm confused why it shows as online.

Let me know if I can be of assistance testing.

[jokob-sk]

@bwirt I think that's a slightly different issue. The first step is to figure out which scan is reporting the device as online. Try running only one of the plugins to determine that. Also, try downloading the _dev image as there are improvements to the UNIFI import plugin. If I remember correctly, in the production image, all devices ar imported and this is reported as online. In the next version, there is an option to first import all UNIFI devices and then report only on online ones.

Check how to use the _dev image, work in progress and what comes next in this post: https://github.com/jokob-sk/Pi.Alert/issues/138

[bwirt]

If I remember correctly, in the production image, all devices ar imported and this is reported as online. In the next version, there is an option to first import all UNIFI devices and then report only on online ones.

Sounds like that's the issue then. I'll try the next version and see if that clears it up.

[jokob-sk]

Might be solved with #645 - please check and of not will reopen