[setup] nmap find devices but doesn't add them to the database

azertylr commented 2 weeks ago

Is there an existing issue for this?

[X] I have searched the existing open and closed issues and I checked the docs https://github.com/jokob-sk/NetAlertX/tree/main/docs

Current Behavior

Hello,

I have an internet router, subnet is 192.168.1.0/24 and I have another router with another subnet 192.168.2.0/24, connected to the main router. NetAlertX is running on the second router, on subnet 192.168.2.0/24. I'm using nmap to detect device on main network, and it does find the devices (see logs) but doesn't add them to the database.

Example

17:31:15 [NMAPDEV] nm[host]: {'hostnames': [{'name': '', 'type': ''}], 'addresses': {'ipv4': '192.168.1.102'}, 'vendor': {}, 'status': {'state': 'up', 'reason': 'echo-reply'}}
17:31:15 [NMAPDEV] Hostname: (unknown), IP: 192.168.1.102, MAC: , Vendor: 
17:31:15 [NMAPDEV] Skipping: (unknown), IP or MAC missing

Thanks !

Expected Behavior

add 192.168.1.102 to database of found devices

Steps To Reproduce

No response

app.conf

#-----------------AUTOGENERATED FILE-----------------#
#                                                    #
#         Generated:  2024-09-02_17-28-26            #
#                                                    #
#   Config file for the LAN intruder detection app:  #
#      https://github.com/jokob-sk/NetAlertX         #
#                                                    #
#-----------------AUTOGENERATED FILE-----------------#

# General
#---------------------------
LOADED_PLUGINS=['ARPSCAN','CSVBCKP','DBCLNP','DDNS','DHCPLSS','DHCPSRVS','INTRNT','INTRSPD','MAINT','NBTSCAN','NMAP','NMAPDEV','NSLOOKUP','NTFPRCS','OMDSDN','SETPWD','SNMPDSC','SYNC','UI','UNDIS','UNFIMP','VNDRPDT']
LOADED_PLUGINS__metadata="                      {}\n                    "
SCAN_SUBNETS=['192.168.1.0/24 --interface=ens18']
SCAN_SUBNETS__metadata="                      {}\n                    "
LOG_LEVEL='verbose'
LOG_LEVEL__metadata="                      {}\n                    "
TIMEZONE='Europe/Paris'
TIMEZONE__metadata="                      {}\n                    "
PLUGINS_KEEP_HIST=250
PLUGINS_KEEP_HIST__metadata="                      {}\n                    "
REPORT_DASHBOARD_URL='http://netalertx'
REPORT_DASHBOARD_URL__metadata="                      {}\n                    "
DAYS_TO_KEEP_EVENTS=90
DAYS_TO_KEEP_EVENTS__metadata="                      {}\n                    "
HRS_TO_KEEP_NEWDEV=0
HRS_TO_KEEP_NEWDEV__metadata="                      {}\n                    "
API_CUSTOM_SQL='SELECT * FROM Devices WHERE dev_PresentLastScan = 0'
API_CUSTOM_SQL__metadata="                      {}\n                    "
NETWORK_DEVICE_TYPES=['AP','Gateway','Firewall','Hypervisor','Powerline','Switch','WLAN','PLC','Router','USB LAN Adapter','USB WIFI Adapter','Internet']
NETWORK_DEVICE_TYPES__metadata="                      {}\n                    "

docker-compose.yml

services:
  netalertx:
    image: jokobsk/netalertx:latest
    container_name: netalertx
    restart: unless-stopped
    network_mode: host
    environment:
      - TZ=Europe/Paris
    volumes:
      - /home/docker/netalertx/config:/app/config
      - /home/docker/netalertx/db:/app/db

What branch are you running?

Production

app.log

17:30:55 [Plugin utils] display_name: NMAP Device discovery 17:30:55 [Plugins] Executing: python3 /app/front/plugins/nmap_dev_scan/nmap_dev.py 17:30:55 [NMAPDEV] In script 17:30:55 [Database] Opening DB 17:30:55 [NMAPDEV] subnets: ['192.168.1.0/24 --interface=ens18'] 17:30:55 [NMAPDEV] scan_args: ['sudo', 'nmap', '-sn', '-PR', '-oX', '-', '192.168.1.0/24', '-e', 'ens18'] 17:31:15 [NMAPDEV] Number of hosts: 4 17:31:15 [NMAPDEV] nm[host]: {'hostnames': [{'name': '', 'type': ''}], 'addresses': {'ipv4': '192.168.1.100'}, 'vendor': {}, 'status': {'state': 'up', 'reason': 'echo-reply'}} 17:31:15 [NMAPDEV] Hostname: (unknown), IP: 192.168.1.100, MAC: , Vendor: 17:31:15 [NMAPDEV] Skipping: (unknown), IP or MAC missing 17:31:15 [NMAPDEV] nm[host]: {'hostnames': [{'name': '', 'type': ''}], 'addresses': {'ipv4': '192.168.1.101'}, 'vendor': {}, 'status': {'state': 'up', 'reason': 'echo-reply'}} 17:31:15 [NMAPDEV] Hostname: (unknown), IP: 192.168.1.101, MAC: , Vendor: 17:31:15 [NMAPDEV] Skipping: (unknown), IP or MAC missing 17:31:15 [NMAPDEV] nm[host]: {'hostnames': [{'name': '', 'type': ''}], 'addresses': {'ipv4': '192.168.1.102'}, 'vendor': {}, 'status': {'state': 'up', 'reason': 'echo-reply'}} 17:31:15 [NMAPDEV] Hostname: (unknown), IP: 192.168.1.102, MAC: , Vendor: 17:31:15 [NMAPDEV] Skipping: (unknown), IP or MAC missing 17:31:15 [NMAPDEV] nm[host]: {'hostnames': [{'name': '', 'type': ''}], 'addresses': {'ipv4': '192.168.1.103'}, 'vendor': {}, 'status': {'state': 'up', 'reason': 'echo-reply'}} 17:31:15 [NMAPDEV] Hostname: (unknown), IP: 192.168.1.103, MAC: , Vendor: 17:31:15 [NMAPDEV] Skipping: (unknown), IP or MAC missing 17:31:15 [NMAPDEV] Devices found: 0 17:31:15 [NMAPDEV] Script finished 17:31:15 [Plugins] No output received from the plugin NMAPDEV - enable LOG_LEVEL=debug and check logs 17:31:15 [Plugin utils] --------------------------------------------- 17:31:15 [Plugin utils] display_name: DB cleanup

Debug enabled

[X] I have read and followed the steps in the wiki link above and provided the required debug logs and the log section covers the time when the issue occurs.

azertylr commented 2 weeks ago

To complete this issue, If I ran nmap on the subnet where NetAlertX is connected (192.168.2.0/24), it is working fine

17:50:54 [Plugin utils] display_name: NMAP Device discovery 17:50:54 [Plugins] Executing: python3 /app/front/plugins/nmap_dev_scan/nmap_dev.py 17:50:54 [NMAPDEV] In script 17:50:54 [Database] Opening DB 17:50:54 [NMAPDEV] subnets: ['192.168.2.0/24 --interface=ens18'] 17:50:54 [NMAPDEV] scan_args: ['sudo', 'nmap', '-sn', '-PR', '-oX', '-', '192.168.2.0/24', '-e', 'ens18'] 17:51:03 [NMAPDEV] Number of hosts: 1 17:51:03 [NMAPDEV] nm[host]: {'hostnames': [{'name': '', 'type': ''}], 'addresses': {'ipv4': '192.168.2.1', 'mac':'E4:F4:C6:0A:CC:64'}, 'vendor': {'E4:F4:C6:0A:CC:63': 'Netgear'}, 'status': {'state': 'up', 'reason': 'arp-response'}} 17:51:03 [NMAPDEV] entry: {'E4:F4:C6:0A:CC:64': 'Netgear'} 17:51:03 [NMAPDEV] Hostname: (unknown), IP: 192.168.2.1, MAC: E4:F4:C6:0A:CC:64, Vendor: Netgear

jokob-sk commented 2 weeks ago

Hi @azertylr,

This behaviour is by design and an Nmap limitation. A valid Mac address is needed for every device and Nmap is only able to discover Mac addresses on the host subnet. You can use a additional scanner, such as arp scan, to discover these devices.

Edit: I also clarified the README to make this NMAP limitation more obvious:

J

azertylr commented 2 weeks ago

Hello,

Thanks for your details. In fact I was using arp scan but it didn't find devices on other subnet. (Even if I run the scan manually). That's why I used nmap as it found my devices in manual. Is there a block in my router firewall that prevent arp scanning the subnet?

jokob-sk commented 2 weeks ago

Hi @azertylr,

Thank might be the case but I'm not an expert on firewalls. I would recommend checking the logs when running arp-scan if there are some errors. You can also try to ssh into the container and run the arp-scan command manually to see if you can debug the issue that way.

Hope this helps, j

jokob-sk commented 6 days ago

no answer for a week -> closing for now

azertylr commented 3 days ago

Hello,

Sorry I didn't had time to check earlier. I've checked and ran:

arp-scan 192.168.1.0/24 --interface=ens18

and it doesn't find any devices. I've tried to disable the firewall of my ddwrt router but it's the same

jokob-sk commented 3 days ago

Hi @azertylr ,

Unfortunately, I'm not an expert on networks, and not sure how your home network is configured.

If arp-scan isn't returning results then there might be access issues to the other networks.

Can you please confirm you are running the scan inside the container? These scans need to be working in the container in order for the app to have access to the results.

Also, have you read the limitations of VLAN access as per https://github.com/jokob-sk/NetAlertX/blob/main/docs/SUBNETS.md?

You can also try setting up multiple instances that would scan the networks independently and sync the results into one: https://github.com/jokob-sk/NetAlertX/blob/main/front/plugins/sync/README.md (there is also pull support from the nodes available in that plugin in the netalertx-dev image).

Hope the above helps, j

azertylr commented 2 days ago

Hi @jokob-sk

Thanks for you help. Yes I did run the command from the container. I've also test from the host machine with the same results. (no devices found). I don't have VLAN Unfortunatly I don't have another instance that I could run from the other subnet :(

jokob-sk commented 2 days ago

Hummm, I would double check that:

NAX is running in host mode
You have the correct interfaces specified for the each subnet

Try running the scans from the container.
If the above doesn't help, I would consider using an alternative way to discover devices. You can check the table of available plugins and pick a dev scanner (🔍) that doesn't need access to the network. For example the SNMPDSC or DHCPLSS plugins might help if your router supports these protocols.
If your browser doesn't support any of the available plugins, consider helping me write a custom plugin for your router.

jokob-sk / NetAlertX