NMAP Scan for Device Detection

stevenengland commented 5 months ago

Is there an existing issue for this?

[X] I have searched the existing open and closed issues

Is your feature request related to a problem? Please describe

Active detection of devices seems to be only possible via ARP scanning.

Describe the solution you'd like

Is there a chance that NMAP scans could be one source for detecting devices as well? With a unobtrusive scan as default option. That would allow me to scan neighboured networks of the NetAlertX instance that it has no ARP access to.

Describe alternatives you've considered

Since NMAP is already on board I did not consider another option yet.

Anything else?

Maybe later.

jokob-sk commented 5 months ago

This should be available in the next release. If you can, please have a look at the netalertx-dev docker image, in about 15 minutes (or after the last action finishes) from now.

It would be great if you could test this (backup everything first or use a new container) on your end by switching to the above image and letting me know if the issue was resolved.

Thanks in advance, j

stevenengland commented 5 months ago

Will check that tomorrow :-)

Thanks a lot!

stevenengland commented 5 months ago

Hi @jokob-sk ,

thanks a lot, definitely goes into the right direction.

A few thoughts:

Calling nmap with -sn flags as default flags is a reasonable approach to keep the scan under the radar. Everyone wanting more can apply this via settings. Nice.
The scan via UI does not lead to new discovered devices.
Scanning via cli (python3 /app/front/plugins/nmap_dev_scan/nmap_dev.py) leads to this error:
```
...
Nmap done: 256 IP addresses (30 hosts up) scanned in 35.94 seconds
```

Traceback (most recent call last): File "/app/front/plugins/nmap_dev_scan/nmap_dev.py", line 144, in main() File "/app/front/plugins/nmap_dev_scan/nmap_dev.py", line 53, in main unique_devices = execute_scan(subnets, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/front/plugins/nmap_dev_scan/nmap_dev.py", line 113, in execute_scan devices_list.append([mac_addresses[i], ip_address, name, vendors[i], interface])


IndexError: list index out of range
netalertx:/# 
```

So far. If you need more information just tell me ;)

jokob-sk commented 5 months ago

Thanks for testing, can you send me the debug log output? I'd need the raw data to see what's being processed.

Search your app.log for:

[NMAPDEV] scan_output:

And paste the redacted version of that output or send it to jokob@duck.com. Thanks in advance, j

stevenengland commented 5 months ago

Sent it, please have a look if it reached your mailbox.

jokob-sk commented 5 months ago

OK, I updated the code. If you can, please have a look at the netalertx-dev docker image, in about 15 minutes (or after the last action finishes) from now.

stevenengland commented 5 months ago

Unfortunately I am now on a business trip with limited resources. But maybe I'll manage it somehow before Friday

vladaurosh commented 5 months ago

Hey @jokob-sk

I've tested nmap with latest dev image. Looking in app.log it does work fine, but new devices are not added to the list. I am scanning neighboured networks, and in app.log I can see nmap is finding devices in those networks. In Plugins -> NMAP device discovery I see just devices from my main network, but no devices from neighboured networks.

And one suggestion regarding nmap scanner. In Settings I had to modify arguments and append networks: sudo nmap -sn 192.168.2.0/24 192.168.3.0/24 And in log I see: 01:35:47 [NMAPDEV] subnets: ['192.168.1.0/24 --interface=eth0'] 01:35:47 [NMAPDEV] scan_args: ['sudo', 'nmap', '-sn', '192.168.2.0/24', '192.168.3.0/24', '192.168.1.0/24'] So it scans my main network by default. Can this be removed?

Also, maybe update UI so nmap has Scan Subnets field where we enter subnets instead of updating arguments.

jokob-sk commented 5 months ago

Hi @vladaurosh ,

Thanks for testing!

I will look into it.

The idea is to make this easy to configure so SCAN_SUBNETS is used for all network scanners.

Any reason why scanning the main network is an issue?

The results are collated and if the devices are already available, nothing gets overwritten AFAIK.

jokob-sk commented 5 months ago

I checked the latest dev container and it seems to be working on my end:

I however don't have multiple subnets to test this on - maybe there is a bug if multiple subnets are added

jokob-sk commented 5 months ago

Can you try to add the subnets you have added to the namp command sudo nmap -sn 192.168.2.0/24 192.168.3.0/24 from here to the main SCAN_SUBNETS setting? I think that will solve your issue

vladaurosh commented 5 months ago

Hey @jokob-sk

I've tried with 2 neighboured subnets into SCAN_SUBNETS, same result. To add some details:

in app.log I can see that nmap scan was performed against neighboured subnets, hosts were found, all looks good there, but I don't see those hosts in device list in UI.
I've checked last_result.log and similar to Plugins -> NMAP device discovery I just see my main network.
Now, there's a difference if for nmap scan all subnets are specified in SCAN_SUBNETS. nmap will run 3 times in my case, once for each subnet. On the other hand if I specify 2 neighboured subnets in arguments of nmap command, it will do single run scanning all 3 subnets.

I can understand that for simplicity all subnets should be in SCAN_SUBNETS, but probably not optimal. For example, in my case of 1 main subnet and 2 neighboured subnets, there will be 6 scans, 3 arp scans (if arp scan is even started for neighboured subnets, for sure it's not needed as it doesn't work) plus 3 nmap scans (nmap scan of main subnet not needed here as it's already scanned in arp scan). To be honest, not sure if this could be done in simple way, to somehow have an option what to scan for each scan type.

jokob-sk commented 5 months ago

Hi @vladaurosh ,

Thanks for testing 🙏

Could you please email me the last_result.logand app.log to jokob@duck.com so I can have a closer look? I think what's happening is I'm always checking 4 lines of the result at a time and with multiple subnets it gets out-of-sync. (feel free to help me debug this as well 😅):

https://github.com/jokob-sk/NetAlertX/blob/d26d6c8b0bae84c9c673f14734e1219f04c52de7/front/plugins/nmap_dev_scan/nmap_dev.py#L98

I'll think about having another Subnets setting for nmap but as you said - it's getting now complicated. Should every scanner using subnets have an override? It's starting to get complex and it's already difficult to get first time users set up the correct settings 🤔

I'd rather trade a bit of slowness (running namp 3x for 3 subnets) and consistency than overcomplicating the setup and to have edge cases for each plugin.

Anyway, let me hear your thoughts 💭

vladaurosh commented 5 months ago

Hi @jokob-sk

Logs have been sent.

If I have an idea about subnets, will let you know for sure. For me this way is not a problem, but not sure if someone has multiple subnets with a lot of hosts, duplicate scans will not help.

jokob-sk commented 5 months ago

Hi @vladaurosh ,

Thanks for the logs!

I added some additional logging. Can you try to re-download the dev image in ~10 min? I also think I fixed some issues with parsing the output so you might already get better results and if not, we'll get at least better debug information to investigate further.

vladaurosh commented 5 months ago

Hi @jokob-sk

Just tried latest dev, there are some changes in app.log but still doesn't work.

15:26:20 [NMAPDEV] Skipped (Not enough info in output): ['Starting Nmap 7.94 ( https://nmap.org ) at 2024-04-27 15:26 IST', '']
15:26:20 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' 192.168.3.1', 'Host is up (0.00060s latency).', '']
15:26:20 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' host1.domain.com (192.168.3.2)', 'Host is up (0.00048s latency).', '']
15:26:20 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' host2.domain.com (192.168.3.3)', 'Host is up (0.00053s latency).', '']
15:26:20 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' host3.domain.com (192.168.3.4)', 'Host is up (0.00045s latency).', 'Nmap done: 8 IP addresses (4 hosts up) scanned in 1.51 seconds', '']

Hope this helps.

jokob-sk commented 5 months ago

Hi @vladaurosh ,

Thanks for the logs, this helps.

I think the next dev build is better. Can you give it a go?

You need to you update the NMAPDEV_ARGS setting to sudo nmap -sn -PR -n, as I changed the default value in the json.

Previously I was not forcing MAC address output and only passing the mask. Now I included the interface and forced MAC address output.

Thanks in advance, j

vladaurosh commented 5 months ago

Hey @jokob-sk

Still doesn't work. I've ran nmap -sn -PR -n 192.168.3.0/29 -e eth0 from the container and got:

Starting Nmap 7.94 ( https://nmap.org ) at 2024-04-28 17:05 IST Nmap scan report for 192.168.3.1 Host is up (0.00062s latency). Nmap scan report for 192.168.3.2 Host is up (0.00057s latency). Nmap scan report for 192.168.3.3 Host is up (0.00080s latency). Nmap scan report for 192.168.3.4 Host is up (0.00077s latency). Nmap done: 8 IP addresses (4 hosts up) scanned in 1.53 seconds

So we're not getting mac addresses here for neighboured subnets, and getting them correctly for main subnet. That's not the issue with nmap just a limit of network layout. But not getting mac addresses for neighboured subnets I guess that breaks your code as you expect mac addresses in the output? Maybe in that case have something different written in mac field, similar as for Internet device?

Oh yeah, I see this in app.log for neighboured subnet:

7:10:50 [NMAPDEV] Skipped (Not enough info in output): ['Starting Nmap 7.94 ( https://nmap.org ) at 2024-04-28 17:10 IST', ''] 17:10:50 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' 192.168.3.1', 'Host is up (0.00064s latency).', ''] 17:10:50 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' 192.168.3.2', 'Host is up (0.00076s latency).', ''] 17:10:50 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' 192.168.3.3', 'Host is up (0.00051s latency).', ''] 17:10:50 [NMAPDEV] Skipped (Couldn't parse MAC or IP): [' 192.168.3.4', 'Host is up (0.00073s latency).', 'Nmap done: 8 IP addresses (4 hosts up) scanned in 1.51 seconds', '']

jokob-sk commented 5 months ago

Hi @vladaurosh ,

The issue of writing something into the MAC field is that if the device is discovered via other means and a proper MAC is discovered then this will detect 2 devices instead of correctly merging it into one based on the MAC address.

You could argue to use the IP address only, but that would require an internal code rewrite on how the matching is done right now (e..g the MAC is used for Events, Presence, Plugin data matching).

Another approach woudl be to use the IPs only for presence detection - but again - not a trivial internal change.

Not sure if we can force NMAP to report MAC addresses on neighbored subnets? That would be the best solution if possible.

Another alternative is to run 2 scanners - Arp scan and Nmap which gives better coverage I think as NMAP is more accurate detecting the online/offline state of a device?

vladaurosh commented 5 months ago

Hi @jokob-sk

Makes perfect sense. Let me see if I am able to get MAC addresses from the hosts in other subnets.

mh166 commented 5 months ago

It generally is not possible to retrieve the MAC address of a neighbouring subnet because the traffic between them is routed which happens on the IP layer.

You can use MAC addresses just within your routed domain or "area": to reach a PC within your own subnet, you are able to talk directly to it, no router needed. Because you're sending packets to the very device you want to reach, of course you need their MAC address (OSI Layer 2).

However if you want to reach another subnet, your packet is routed from one IP to another, on OSI Layer 3. So the information of what's the other device's MAC address is kind of "lost" (at least from your perspective). Instead, the router replaces that MAC address with it's own as he is the "mediator": he needs to receive the packet on behalf of that device in order to forward it to the correct PC on the other subnet.

It may work if your NetAlertX instance is running on the router that is connecting those subnets, depending on your specific setup. Other than that, there are some ways to overcome this basic (and useful) restriction of how networking works, but (at least according to my knowledge) none of them are trivial and often may have side-effects which could potentially negatively impact your network.

With that in mind (and as long as everything keeps revolving around the MAC address as source of truth) I would see two options:

Allow this setting to be configured as it is right now but add a warning to it. You could explain that adjusting the parameters might now result in expected behaviour. Especially with regards to other subnets, as currently NetAlertX requires access to the device's MAC address to function correctly.
Provide a "proxy agent": a small script/binary/docker image/whatever, that can live in the foreign network and execute the network scans from within that neighbouring subnet. The results can then be sent back to the main NetAlertX instance and be used as if the data had been sourced locally. I'd suggest adding a new field or at least a note to all such devices to indicate that they have been discovered by a proxy agent. Maybe even say which proxy agent it was in particular as multiple could be added for multiple subnets.

mh166 commented 5 months ago

On a side note: I would strongly suggest not to simply parse the plain text output of nmap to get the information. As we saw just now, this is very error prone even in the best circumstances.

Instead, please use the -oX option to output XML. The output can be directed to stdout using -oX - or written to a file by specifying -oX filename. As there are plenty of XML libraries out there, this would allow easy and, more importantly, reliable parsing of all the information needed.

The result for nmap -oX - -sn 192.168.0.0/24 would look something like that.

Example scan (click to show)

**NB:** The output has been shortened and neatly indented by me for nicer reading. ```XML

```

jokob-sk commented 5 months ago

Hey,

rewritten into parsing the XML output if someone wants to test the dev image,

Thanks, j

mh166 commented 5 months ago

Looks good, thank you! I'm really impressed by your fast turn around time in this project. Keep up the awesome work! 🥳👍

jokob-sk commented 5 months ago

Thanks, it's only possible with the plugin and auto-generating setting & UI system in place. Happy for anyone to contribute :) https://github.com/jokob-sk/NetAlertX/tree/main/front/plugins

Regarding 2: Providing a proxy agent - that sounds too big of an investment I'm currently not willing to make. There are other scan methods that can substitute it and you could theoretically run multiple instances of the container. I will keep it in the back of my mind for now.

jokob-sk commented 5 months ago

released -> closing for now

jokob-sk / NetAlertX