jokob-sk / NetAlertX

🖧🔍 WIFI / LAN intruder detector. Scans for devices connected to your network and alerts you if new and unknown devices are found.
GNU General Public License v3.0
3.15k stars 186 forks source link

Network scan crashes docker container regularly #482

Closed ftrueck closed 1 year ago

ftrueck commented 1 year ago

I navigate through pi.alert and all of a sudden the web is not reacting anymore. When I check the status of the docker container then I see a stacktrace from network scan. In the following I attach the Log when the crash happened.

Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/pi/pialert/pialert/main.py", line 179, in sys.exit(main())
File "/home/pi/pialert/pialert/main.py", line 136, in main process_scan(db) File "/home/pi/pialert/pialert/networkscan.py", line 41, in process_scan update_devices_data_from_scan (db) File "/home/pi/pialert/pialert/device.py", line 193, in update_devices_data_from_scan sql.execute ("""UPDATE Devices sqlite3.IntegrityError: NOT NULL constraint failed: Devices.dev_Name 23:19:49 [Notification] included sections: ['new_devices', 'down_devices', 'events'] 23:19:49 [Notification] New Devices sections done. 23:19:49 [Notification] Down Devices sections done. 23:19:49 [Notification] Events sections done. 23:19:49 [Notification] No changes to report 23:19:49 [Notification] Notifications changes: 0 23:19:49 [MAIN] Process: Wait 23:19:54 [MAIN] waiting to start next loop 23:19:59 [MAIN] waiting to start next loop 23:20:05 [MAIN] waiting to start next loop 23:20:10 [MAIN] waiting to start next loop 23:20:15 [MAIN] waiting to start next loop 23:20:20 [MAIN] waiting to start next loop 23:20:25 [MAIN] waiting to start next loop 23:20:30 [MAIN] waiting to start next loop 23:20:35 [MAIN] waiting to start next loop 23:20:40 [MAIN] waiting to start next loop 23:20:45 [MAIN] waiting to start next loop 23:20:50 [Notification] Check if something to report 23:20:50 [Notification] Open text Template 23:20:50 [Notification] Open html Template 23:20:50 [Notification] Using template/back/report_template.html 23:20:50 [Notification] included sections: ['new_devices', 'down_devices', 'events'] 23:20:50 [Notification] New Devices sections done. 23:20:50 [Notification] Down Devices sections done. 23:20:50 [Notification] Events sections done. 23:20:50 [Notification] No changes to report 23:20:50 [Notification] Notifications changes: 0 23:20:50 [MAIN] Process: Wait 23:20:55 [MAIN] waiting to start next loop 23:21:00 [MAIN] waiting to start next loop 23:21:05 [MAIN] waiting to start next loop 23:21:10 [MAIN] waiting to start next loop 23:21:15 [MAIN] waiting to start next loop 23:21:20 [MAIN] waiting to start next loop 23:21:26 [MAIN] waiting to start next loop 23:21:31 [MAIN] waiting to start next loop 23:21:36 [MAIN] waiting to start next loop 23:21:41 [MAIN] waiting to start next loop 23:21:46 [API] Updating table_devices.json file in /front/api 23:21:46 [MAIN] waiting to start next loop 23:21:51 [Plugins] --------------------------------------------- 23:21:51 [Plugins] display_name: Arp-Scan (Network scan) 23:21:51 [Plugins] Executing: python3 /home/pi/pialert/front/plugins/arp_scan/script.py userSubnets={subnets} 23:22:44 [Plugins] SUCCESS, received 49 entries 23:22:45 [API] Updating table_plugins_events.json file in /front/api 23:22:46 [API] Updating table_plugins_history.json file in /front/api 23:22:46 [API] Updating table_plugins_objects.json file in /front/api 23:22:46 [Plugins] --------------------------------------------- 23:22:46 [Plugins] display_name: DHCP Leases (Device import) 23:22:46 [Plugins] Executing: python3 /home/pi/pialert/front/plugins/dhcp_leases/script.py paths={paths} 23:22:46 [Plugins] SUCCESS, received 41 entries 23:22:47 [API] Updating table_plugins_history.json file in /front/api 23:22:47 [Plugins] --------------------------------------------- 23:22:47 [Plugins] display_name: Internet check 23:22:47 [Plugins] Executing: python3 /home/pi/pialert/front/plugins/internet_ip/script.py prev_ip={prev_ip} DIG_GET_IP_ARG={DIG_GET_IP_ARG} 23:22:47 [Plugins] SUCCESS, received 1 entries 23:22:47 [API] Updating table_plugins_history.json file in /front/api 23:22:47 [Process Scan] Processing scan results 23:22:47 [Process Scan] Print Stats 23:22:47 [Scan Stats] Devices Detected.......: 92 23:22:47 [Scan Stats] New Devices............: 0 23:22:47 [Scan Stats] Down Alerts............: 0 23:22:47 [Scan Stats] New Down Alerts........: 0 23:22:47 [Scan Stats] New Connections........: 2 23:22:47 [Scan Stats] Disconnections.........: 0 23:22:47 [Scan Stats] IP Changes.............: 1 23:22:47 [Scan Stats] Scan Method Statistics: 23:22:47 DHCPLSS: 41 23:22:47 INTRNT: 1 23:22:47 arp-scan: 49 23:22:47 local_MAC: 1 23:22:47 [Process Scan] Stats end 23:22:47 [Process Scan] Sessions Events (connect / discconnect) 23:22:47 [Process Scan] Creating new devices 23:22:47 [Process Scan] Updating Devices Info

jokob-sk commented 1 year ago

Hi there,

Can you please enable LOG_LEVEL=debug and post the pialert.log file after a crash?

The SQL query to execute explicitly checks for nulls (notice AND cur_Name IS NOT NULL below), so unsure why this would be happening without additional logs:

UPDATE Devices
                    SET dev_NAME = (SELECT cur_Name FROM CurrentScan
                                    WHERE cur_MAC = dev_MAC)
                    WHERE (dev_Name in ("(unknown)", "(name not found)", "" )
                           OR dev_Name IS NULL)
                      AND EXISTS (SELECT 1 FROM CurrentScan
                                  WHERE cur_MAC = dev_MAC
                                    AND cur_Name IS NOT NULL
                                    AND cur_Name IS NOT 'null'
                                    AND cur_Name <> '') 

Thanks in advance, j

ftrueck commented 1 year ago

Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/pi/pialert/pialert/main.py", line 179, in sys.exit(main())
File "/home/pi/pialert/pialert/main.py", line 136, in main process_scan(db) File "/home/pi/pialert/pialert/networkscan.py", line 41, in process_scan update_devices_data_from_scan (db) File "/home/pi/pialert/pialert/device.py", line 193, in update_devices_data_from_scan sql.execute ("""UPDATE Devices sqlite3.IntegrityError: NOT NULL constraint failed: Devices.dev_Name 13:01:00 [Scheduler] - Scheduler run for DHCPSRVS: NO 13:01:00 [Scheduler] - Scheduler run for INTRNT: YES 13:01:00 [Plugins] --------------------------------------------- 13:01:00 [Plugins] display_name: Internet check 13:01:00 [Plugins] CMD: python3 /home/pi/pialert/front/plugins/internet_ip/script.py prev_ip={prev_ip} DIG_GET_IP_ARG={DIG_GET_IP_ARG} 13:01:00 [Plugins] Flattening the below array 13:01:00 ['91.62.96.44'] 13:01:00 [Plugins] Flattened array: 91.62.96.44 13:01:00 [Plugins] Resolved value: 91.62.96.44 13:01:00 [Plugins] Convert to Base64: False 13:01:00 [Plugins] setTyp: text 13:01:00 [Plugins] Resolved value: -4 myip.opendns.com @resolver1.opendns.com 13:01:00 [Plugins] Convert to Base64: True 13:01:00 [Plugins] base64 value: b'LTQgbXlpcC5vcGVuZG5zLmNvbSBAcmVzb2x2ZXIxLm9wZW5kbnMuY29t' 13:01:00 [Plugins] Timeout: 30 13:01:00 [Plugins] Pre-Resolved CMD: python3/home/pi/pialert/front/plugins/internet_ip/script.pyprev_ip={prev_ip}DIG_GET_IP_ARG={DIG_GET_IP_ARG} 13:01:00 [Plugins] Executing: python3 /home/pi/pialert/front/plugins/internet_ip/script.py prev_ip={prev_ip} DIG_GET_IP_ARG={DIG_GET_IP_ARG} 13:01:00 [Plugins] Resolved : ['python3', '/home/pi/pialert/front/plugins/internet_ip/script.py', 'prev_ip=91.62.96.44', "DIG_GET_IP_ARG=b'LTQgbXlpcC5vcGVuZG5zLmNvbSBAcmVzb2x2ZXIxLm9wZW5kbnMuY29t'"] 13:01:01 [Plugins] SUCCESS, received 1 entries 13:01:01 [Plugins] sqlParam entries: [(0, 'INTRNT', 'Internet', '91.62.96.44', 'null', '2023-10-14 13:01:01', 'Previous IP: 91.62.96.44', '', '', '', 'not-processed', 'Previous IP: 91.62.96.44', 'null', 'Internet')] 13:01:01 [Plugins] Processing : INTRNT 13:01:01 [Plugins] Existing objects from Plugins_Objects: 1 13:01:01 [Plugins] Logged events from the plugin run : 1 13:01:01 [Plugins] statuses_to_report_on: ['new', 'watched-changed'] 13:01:01 [Plugins] pluginEvents count: 1 13:01:01 [Plugins] pluginObjects count: 1 13:01:01 [Plugins] events_to_insert count: 0 13:01:01 [Plugins] history_to_insert count: 1 13:01:01 [Plugins] objects_to_insert count: 0 13:01:01 [Plugins] objects_to_update count: 1 13:01:01 [Plugins] In pluginEvents there are 1 events with the status "watched-not-changed" 13:01:01 [Plugins] In pluginObjects there are 1 events with the status "watched-not-changed" 13:01:01 [Plugins] Mapping objects to database table: CurrentScan 13:01:01 [Plugins] SQL query for mapping: INSERT into CurrentScan ( "cur_MAC", "cur_IP", "cur_ScanMethod", "cur_DateTime") VALUES ( ?, ?, ?, ?) 13:01:01 [Plugins] SQL sqlParams for mapping: [('Internet', '91.62.96.44', 'INTRNT', '2023-10-12 16:35:31')] 13:01:01 [API] Update API starting 13:01:01 [API] Updating table_plugins_history.json file in /front/api 13:01:01 [Plugins] Setting with "function":"RUN" is missing in plugin: New Devices 13:01:01 [Scheduler] - Scheduler run for NMAP: NO 13:01:01 [Scheduler] - Scheduler run for VNDRPDT: NO 13:01:01 [Plugins] Check if any plugins need to be executed on run type: always_after_scan 13:01:01 [Plugins] Setting with "function":"RUN" is missing in plugin: New Devices 13:01:01 [MAIN] processScan: True 13:01:01 [MAIN] start processig scan results 13:01:01 [Process Scan] Processing scan results 13:01:01 [Save Devices] Saving this IP into the CurrentScan table:192.168.9.6 13:01:01 [Process Scan] Print Stats 13:01:01 [Scan Stats] Devices Detected.......: 86 13:01:01 [Scan Stats] New Devices............: 1 13:01:01 [Scan Stats] Down Alerts............: 0 13:01:01 [Scan Stats] New Down Alerts........: 0 13:01:01 [Scan Stats] New Connections........: 11 13:01:01 [Scan Stats] Disconnections.........: 0 13:01:01 [Scan Stats] IP Changes.............: 1 13:01:01 [Scan Stats] Scan Method Statistics: 13:01:01 DHCPLSS: 43 13:01:01 INTRNT: 1 13:01:01 arp-scan: 41 13:01:01 local_MAC: 1 13:01:01 [Process Scan] Stats end 13:01:01 [Process Scan] Sessions Events (connect / discconnect) 13:01:01 [Events] - 1 - Devices down 13:01:01 [Events] - 2 - New Connections 13:01:01 [Events] - 3 - Disconnections 13:01:01 [Events] - 4 - IP Changes 13:01:01 [Events] - Events end 13:01:01 [Process Scan] Creating new devices 13:01:01 [New Devices] New devices - 1 Events 13:01:01 [New Devices] Insert Connection into session table 13:01:01 [New Devices] 2 Create devices 13:01:01 [New Devices] New Devices end 13:01:01 [Process Scan] Updating Devices Info 13:01:01 [Update Devices] 1 Last Connection 13:01:01 [Update Devices] 2 Clean no active devices 13:01:01 [Update Devices] - 3 LastIP & Vendor 13:01:01 [Update Devices] - 4 Unknown Name

ftrueck commented 1 year ago

pialert.conf.txt

Here the config file from the time of the crash

ftrueck commented 1 year ago

What I noticed is: It crashes when DHCP.leases is scheduled for scan. The only remarkable things in my leases file are empty entries like this one:

1697357042 a4:7e:fa:11:1c:ee 192.168.11.176

This device gets created as an empty device with the name "*"

if i try in the details page to traceroute it I get an error that the IP address is wrong. Same for nmap scan. Seems it uses the wrong data field for the IP Address, since there clearly is an address in the data.

jokob-sk commented 1 year ago

Hi,

Thanks for the details.

I have a sample pihole.leases file with the following content and this seems to work fine:

1695112936 aa:87:ba:b1:51:aa 192.168.1.247 Tapo_SmartPlug *
1695105003 aa:87:ba:b1:51:ab 192.168.1.209 * *
1695101756 aa:87:ba:b1:51:ac 192.168.1.201 * *

image

Can you please check the Plugins > Any enabled plugin section if some unusual data is stored for the Device name?

I also implemented some debugging that will print the whole CurrentScan table if a new device is discovered. You can try deploying the pi.alert_dev image to get a better debug output.

Another thing you could try is to disable PLugins one-by-one to narrow down the plugin causing the issue.

I'll try reproducing the issue on my end as well.

ftrueck commented 1 year ago

As I wrote before: The container seems to crash only when I have DHCP leases scan enabled. Maybe also pihole, but that I did not enable when the crash happened.

jokob-sk commented 1 year ago

Thanks for confirming! Would it be possible to share the dhcp.leases file causing the crash? You can email it to jokob@duck.com if you prefer that.

ftrueck commented 1 year ago

Email sent.

jokob-sk commented 1 year ago

@ftrueck Thanks for that and for your patience!

I tested the provided file and it imported fine in my pi.pialert_dev instance. I think it should also work on the current production image.

image

What could have happened is that the Devices table got corrupted causing the subsequent import to fail.

You can try:

Option 1: DB cleanup

Detailed guide on the below: https://github.com/jokob-sk/Pi.Alert/blob/main/docs/DEVICES_BULK_EDITING.md

  1. Exporting the Devices via the CSV Export functionality
  2. Having a look at the CSV/Cleaning it up
  3. Importing the cleaned-up CSV file

Option 2: Start from scratch with pialert.conf and devices.csv on the production image (pi.alert)

Another alternative is to try setting up the container from scratch, with the CSV file and pialert.conf as a starting point (these 2 files contain most of your configuration, including the network tree setup).

Option 3: Start from scratch with pialert.conf and devices.csv on the dev image (pi.alert_dev)

The same applies as with option 1,2,3 - you only use the latest code base. Please note, that there might be other bugs as the next version is still under heavy development (e.g. there is a known permission issue saving the Settings).

Hope this helps, j

jokob-sk commented 1 year ago

Closing as no response > will reopen if needed