search

pi-hole / FTL

The Pi-hole FTL engine
https://pi-hole.net
Other
1.38k stars 196 forks source link

Strange issue #17

Closed DL6ER closed 7 years ago

DL6ER commented 7 years ago

@strrrn wrote:

mine is crashing with the following:

[2017-04-10 00:00:09.177] Notice: Increasing queries struct size from 0 to 10000 (281020.00 B)
[2017-04-10 00:00:09.178] Notice: Increasing overTime struct size from 0 to 100 (283420.00 B)
[2017-04-10 00:00:09.178] Notice: Increasing domains struct size from 0 to 1000 (299428.00 B)
[2017-04-10 00:00:11.968] Notice: Increasing domains struct size from 1000 to 2000 (338743.00 B)
[2017-04-10 00:00:11.994] Notice: Increasing queries struct size from 10000 to 20000 (619161.00 B)
[2017-04-10 00:00:15.548] Notice: Increasing overTime struct size from 100 to 200 (637702.00 B)
[2017-04-10 00:00:15.658] Notice: Increasing queries struct size from 20000 to 30000 (917877.00 B)
[2017-04-10 00:00:17.633] Notice: Increasing domains struct size from 2000 to 3000 (939924.00 B)
[2017-04-10 00:00:19.814] Notice: Increasing queries struct size from 30000 to 40000 (1225976.00 B)
[2017-04-10 00:00:21.281] Reading from /var/log/pihole.log
[2017-04-10 00:36:26.555] Notice: Increasing clients struct size from 0 to 20 (1239221.00 B)
[2017-04-10 00:36:26.759] Added new client: 2001:470:HIDDEN
[2017-04-10 00:36:26.760] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2017-04-10 00:36:26.760] ----------------------------> FTL crashed! <----------------------------
[2017-04-10 00:36:26.761] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2017-04-10 00:36:26.761] > Please report a bug at https://github.com/pi-hole/FTL/issues
[2017-04-10 00:36:26.761] > and include in your report already the following details:
[2017-04-10 00:36:26.761] >
[2017-04-10 00:36:26.762] > Received signal: Segmentation fault
[2017-04-10 00:36:26.762] at address: 513
[2017-04-10 00:36:26.762] with code: SEGV_MAPERR (Address not mapped to object)
[2017-04-10 00:36:26.763] > Memory usage (structs): 1173176
[2017-04-10 00:36:26.763] > Memory usage (dynamic): 66084
[2017-04-10 00:36:26.763] >
[2017-04-10 00:36:26.764] > Thank you for helping us to improve our FTL engine!
[2017-04-10 00:36:26.764] FTL terminated!

any updates on this?

DL6ER commented 7 years ago

This log is quite odd, e.g. (1225976.00 B) which should read (1.22 MB).

Can you run pihole-FTL inside the debugger like suggested here? Once it crashes, please type where and provide the output. Further steps in the debugger might be necessary when I know where in the code the crash happened.

nin9s commented 7 years ago

unfortunately the following didnt work at all:

FTL has been designed such that a debugger can be attached to an already running process to ease debugging. Use sudo gdb -p $(cat /var/run/pihole-FTL.pid) to attach to the already running pihole-FTL process. You can leave off sudo if you are running pihole-FTL with the current user. Once loading of the symbols has finished (the (gdb) input prompt is shown), run continue to continue operation of pihole-FTL inside the debugger. All debugger features are now available.

When you want to detach the debugger from FTL without terminating the process, you can hit Ctrl+C and enter detach followed by quit

what is working is the following: pihole-FTL debug test

[2017-04-11 00:07:26.199] ########## FTL started! ########## [2017-04-11 00:07:26.202] FTL branch: (no branch) [2017-04-11 00:07:26.204] FTL hash: v2.0 [2017-04-11 00:07:26.206] FTL date: 2017-04-10 00:46:52 +0200 [2017-04-11 00:07:26.208] FTL user: root [2017-04-11 00:07:26.211] Notice: Found no readable FTL config file [2017-04-11 00:07:26.212] Using default settings [2017-04-11 00:07:26.214] Starting config file parsing [2017-04-11 00:07:26.216] SOCKET_LISTENING: only local [2017-04-11 00:07:26.217] TIMEFRAME: Rolling 24h [2017-04-11 00:07:26.219] QUERY_DISPLAY: Show queries [2017-04-11 00:07:26.220] Finished config file parsing [2017-04-11 00:07:26.222] PID of FTL process: 21610 [2017-04-11 00:07:28.312] Gravity list entries: 554859 [2017-04-11 00:07:28.315] Blacklist entries: 1 [2017-04-11 00:07:28.317] Wildcard blocking list entries: 11 [2017-04-11 00:07:28.320] Starting initial log file parsing [2017-04-11 00:07:28.322] Reading from /var/log/pihole.log.1 (rw-r--r--) [2017-04-11 00:07:28.324] Notice: Increasing queries struct size from 0 to 10000 (280278.00 B) [2017-04-11 00:07:28.327] at query time: [2017-04-11 00:07:28.339] Notice: Increasing overTime struct size from 0 to 100 (282678.00 B) [2017-04-11 00:07:28.341] at query time: Apr 10 00:00:24 [2017-04-11 00:07:28.343] Notice: Increasing forwarded struct size from 0 to 4 (282734.00 B) [2017-04-11 00:07:28.346] at query time: Apr 10 00:00:24 [2017-04-11 00:07:28.349] Added new forward server: 127.10.10.1 (OpenDNSCrypt) [2017-04-11 00:07:28.351] Notice: Increasing domains struct size from 0 to 1000 (298759.00 B) [2017-04-11 00:07:28.353] at query time: Apr 10 00:00:24 [2017-04-11 00:07:28.356] Notice: Increasing clients struct size from 0 to 20 (299027.00 B) [2017-04-11 00:07:28.358] at query time: Apr 10 00:00:24 [2017-04-11 00:07:28.559] Added new client: 2001:470:1f0b:ad7:d1:7a46:aa74:c3ce [2017-04-11 00:07:28.742] Added new client: 2001:470:1f0b:ad7:1de6:a19d:1174:35db [2017-04-11 00:07:28.801] Added new client: 192.168.254.103 [2017-04-11 00:07:28.860] Added new client: 192.168.179.251 [2017-04-11 00:07:29.043] Added new client: 2001:470:1f0b:ad7:6e3b:6bff:fe77:8b6c [2017-04-11 00:07:29.045] Added new client: 192.168.254.6 (nas) [2017-04-11 00:07:29.104] Added new client: 192.168.254.106 [2017-04-11 00:07:29.174] Added new client: 192.168.254.139 [2017-04-11 00:07:29.233] Added new client: 192.168.254.129 [2017-04-11 00:07:29.291] Added new client: 192.168.254.100 [2017-04-11 00:07:29.353] Added new client: 192.168.254.108 [2017-04-11 00:07:29.630] Added new client: 2001:470:1f0b:ad7:b10d:4d22:a0b5:3828 [2017-04-11 00:07:29.688] Added new client: 192.168.254.116 [2017-04-11 00:07:29.911] Added new client: 2001:470:1f0b:ad7:18e0:38bb:2321:3bc5 [2017-04-11 00:07:30.198] Added new client: 2001:470:1f0b:ad7:5579:3466:37aa:ec0a [2017-04-11 00:07:31.142] Added new client: 192.168.254.102 [2017-04-11 00:07:31.660] Added new client: 2001:470:1f0b:ad7:18b1:9a0:ed0d:9f68 [2017-04-11 00:07:32.016] Added new client: 192.168.254.105 [2017-04-11 00:07:32.554] Added new client: 10.0.0.121 [2017-04-11 00:07:32.630] Added new client: 127.0.0.1 (localhost) [2017-04-11 00:07:32.949] Notice: Increasing overTime struct size from 100 to 200 (323740.00 B) [2017-04-11 00:07:32.951] at query time: Apr 10 16:40:02 [2017-04-11 00:07:32.984] Notice: Increasing queries struct size from 10000 to 20000 (604011.00 B) [2017-04-11 00:07:32.986] at query time: Apr 10 16:48:00 [2017-04-11 00:07:33.222] Notice: Increasing clients struct size from 20 to 40 (605062.00 B) [2017-04-11 00:07:33.224] at query time: Apr 10 18:26:15 [2017-04-11 00:07:33.283] Added new client: 192.168.254.101 [2017-04-11 00:07:33.341] Added new client: 192.168.254.200 [2017-04-11 00:07:33.475] Added new client: 192.168.254.104 [2017-04-11 00:07:33.518] Notice: Increasing domains struct size from 1000 to 2000 (622489.00 B) [2017-04-11 00:07:33.530] at query time: Apr 10 18:27:06 [2017-04-11 00:07:35.160] Added new client: 2001:470:1f0b:ad7:d083:6ac2:9db7:2050 [2017-04-11 00:07:35.894] Added new client: 2001:470:1f0b:ad7:8df6:64a8:2cf2:c05a [2017-04-11 00:07:36.297] Added new client: 2001:470:1f0b:ad7:543c:27ac:7e7e:c53e [2017-04-11 00:07:37.267] Notice: Increasing queries struct size from 20000 to 30000 (910082.00 B) [2017-04-11 00:07:37.269] at query time: Apr 10 23:28:14 [2017-04-11 00:07:37.641] Reading from /var/log/pihole.log (rw-r--r--) [2017-04-11 00:07:37.780] Finished initial log file parsing [2017-04-11 00:07:37.782] -> Total DNS queries: 21435 [2017-04-11 00:07:37.784] -> Cached DNS queries: 6663 [2017-04-11 00:07:37.786] -> Blocked DNS queries: 1507 [2017-04-11 00:07:37.789] -> Unknown DNS queries: 0 [2017-04-11 00:07:37.791] -> Unique domains: 1347 [2017-04-11 00:07:37.793] -> Unique clients: 26 [2017-04-11 00:07:37.795] Successfully accessed setupVars.conf [2017-04-11 00:07:37.798] Listening on port 4711 [2017-04-11 00:07:37.801] Shutting down... [2017-04-11 00:07:37.804] ########## FTL terminated! ##########

I'm confused as the FTL service isnt starting anymore...

sudo service pihole-FTL status ● pihole-FTL.service - LSB: pihole-FTL daemon Loaded: loaded (/etc/init.d/pihole-FTL) Active: active (exited) since Tue 2017-04-11 00:05:12 CEST; 45s ago

and syslog is repeately complaining about:

tail -f /var/log/syslog Apr 11 00:05:47 lighttpd[19767]: for additional help text. Apr 11 00:05:47 lighttpd[19767]: For more details see ps(1). Apr 11 00:05:56 systemd[1]: Started LSB: pihole-FTL daemon. Apr 11 00:05:57 lighttpd[19767]: error: process ID list syntax error Apr 11 00:05:57 lighttpd[19767]: Usage: Apr 11 00:05:57 lighttpd[19767]: ps [options] Apr 11 00:05:57 lighttpd[19767]: Try 'ps --help <simple|list|output|threads|misc|all>' Apr 11 00:05:57 lighttpd[19767]: or 'ps --help <s|l|o|t|m|a>' Apr 11 00:05:57 lighttpd[19767]: for additional help text. Apr 11 00:05:57 lighttpd[19767]: For more details see ps(1). Apr 11 00:06:07 lighttpd[19767]: error: process ID list syntax error Apr 11 00:06:07 lighttpd[19767]: Usage: Apr 11 00:06:07 lighttpd[19767]: ps [options] Apr 11 00:06:07 lighttpd[19767]: Try 'ps --help <simple|list|output|threads|misc|all>' Apr 11 00:06:07 lighttpd[19767]: or 'ps --help <s|l|o|t|m|a>' Apr 11 00:06:07 lighttpd[19767]: for additional help text. Apr 11 00:06:07 lighttpd[19767]: For more details see ps(1).

any hints?

//edit: reboot solved FTL start but how should I continue as the debug does not seem to work?

Thanks!

AzureMarker commented 7 years ago

What was causing the normal debug steps to not work?

technicalpyro commented 7 years ago

these steps should work to get you to the correct debug

  1. Install screen and gdb (probably sudo apt-get install screen gdb)
  2. Start a screen session (it will allow you to come back even if the SSH connection died) 2.1. If you don't know about screen, then read about it (you will love it!) 2.2. Continue afterwards :-)
  3. Start a screen session using screen
  4. Ensure that pihole-FTL is terminated (e.g. sudo killall pihole-FTL)
  5. Arrange file permissions to be able to start FTL as your current user (the following assumes you are logged in as user pi). 5.1. sudo touch /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port 5.2. sudo chown pi:pi /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port 5.3. sudo chmod 0644 /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port
  6. Start pihole-FTL in the debugger: gdb pihole-FTL
  7. Type run debug to start FTL (you should see some lines of text and FTL should start successfully).
  8. You can now close the terminal (Ctrl+A and then D to detach) and come back later using (screen -r) when it has crashed
  9. If it has crashed, copy&past the terminal output, and type also backtrace and post the output
DL6ER commented 7 years ago

Oh yes... The simplified debugging does not work when FTL crashes during or shortly after startup. Use the instructions @technicalpyro posted instead.

There is a catch in service when a process crashes unexpectedly that prevents sudo service ... start from working at all. Try restart as option, instead.

They are now also included in the official README: https://github.com/pi-hole/FTL#debugging-instructions-when-ftl-is-not-starting-reliably

nin9s commented 7 years ago

/edited

@DL6ER @technicalpyro thanks for sharing some pretty useful knowledge :) I'm running the screen debug session right now and will post back if it has crashed.

@DL6ER unfortunately restart didnt change a thing :/

technicalpyro commented 7 years ago

@DL6ER taught it to me so i cannot take any credit

nin9s commented 7 years ago

finally it happened again:

`[New Thread 0xb5b7a460 (LWP 28133)] [Thread 0xb5b7a460 (LWP 28133) exited] [2017-04-13 09:20:02.258] Notice: Increasing overTime struct size from 200 to 300 (920052.00 B) [2017-04-13 09:20:02.258] at query time: Apr 13 09:20:02

a^[[2017-04-13 09:59:00.076] Running GC on data structure [New Thread 0xb5b7a460 (LWP 7905)] [Thread 0xb5b7a460 (LWP 7905) exited] [2017-04-13 10:59:00.020] Running GC on data structure [New Thread 0xb5b7a460 (LWP 21481)] [Thread 0xb5b7a460 (LWP 21481) exited]

[2017-04-13 11:59:00.009] Running GC on data structure [New Thread 0xb5b7a460 (LWP 2759)] [Thread 0xb5b7a460 (LWP 2759) exited] [2017-04-13 12:59:00.003] Running GC on data structure [New Thread 0xb5b7a460 (LWP 16334)] [Thread 0xb5b7a460 (LWP 16334) exited] [2017-04-13 13:46:23.590] Notice: Increasing queries struct size from 30000 to 40000 (1204775.00 B) [2017-04-13 13:46:23.591] at query time: Apr 13 13:46:22 [2017-04-13 13:59:00.032] Running GC on data structure [New Thread 0xb5b7a460 (LWP 29946)] [Thread 0xb5b7a460 (LWP 29946) exited]

[2017-04-13 14:59:00.085] Running GC on data structure [New Thread 0xb5b7a460 (LWP 11139)] [Thread 0xb5b7a460 (LWP 11139) exited] [2017-04-13 15:59:00.034] Running GC on data structure [New Thread 0xb5b7a460 (LWP 18155)] [Thread 0xb5b7a460 (LWP 18155) exited] [2017-04-13 16:59:00.002] Running GC on data structure [New Thread 0xb5b7a460 (LWP 26243)] [Thread 0xb5b7a460 (LWP 26243) exited] [2017-04-13 17:59:00.090] Running GC on data structure [New Thread 0xb5b7a460 (LWP 10073)] [Thread 0xb5b7a460 (LWP 10073) exited] [2017-04-13 18:59:00.029] Running GC on data structure [New Thread 0xb5b7a460 (LWP 26331)] [Thread 0xb5b7a460 (LWP 26331) exited] [2017-04-13 19:02:15.613] Notice: Increasing clients struct size from 0 to 20 (1207933.00 B) [2017-04-13 19:02:15.614] at query time: Apr 13 19:02:15 [2017-04-13 19:02:15.879] Added new client: 2001:470:1f0b:ad7:98a2:aa49:9e36:2d25

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb6b7a460 (LWP 7647)] 0x7f559868 in findClient (client=0x7f57b5a4 "\001") at parser.c:729 729 parser.c: No such file or directory.`

`(gdb) backtrace

0 0x7f559868 in findClient (client=0x7f57b5a4 "\001") at parser.c:729

1 process_pihole_log (file=42) at parser.c:373

2 0x7f55a458 in pihole_log_thread (val=) at parser.c:103

3 0xb6f1de90 in start_thread (arg=0xb6b7a460) at pthread_create.c:311

4 0xb6ea7598 in ?? () at ../ports/sysdeps/unix/sysv/linux/arm/nptl/../clone.S:92 from /lib/arm-linux-gnueabihf/libc.so.6

Backtrace stopped: previous frame identical to this frame (corrupt stack?) `

DL6ER commented 7 years ago

Okay, so this is still strange. Please try again with FTL v2.2 where I have implemented more sophisticated checks whether arrays are allowed to be accessed (there will be more helpful output, too).

What operating system on which device are you running?

nin9s commented 7 years ago

ok, I'll update FTL and update this thread if it happens again device is raspberry model a or b, I'm not 100% sure

Linux 4.4.50+ #970 Mon Feb 20 19:12:50 GMT 2017 armv6l GNU/Linux

Could it have something to do that I've disabled swapping completely?

//Edit: while upgradeing I get this at the end

::: DNS service is running
::: Pi-hole blocking is Enabled
:::
::: Starting pihole-FTL service...Unable to complete update, contact Pi-hole

systemctl status pihole-FTL.service ● pihole-FTL.service - LSB: pihole-FTL daemon Loaded: loaded (/etc/init.d/pihole-FTL) Active: failed (Result: exit-code) since Thu 2017-04-13 20:23:04 CEST; 4s ago Process: 6596 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=1/FAILURE)

technicalpyro commented 7 years ago

you must exit the FTL process that is running in the debug in order to have the new one being used

if you have then im not sure on that error

nin9s commented 7 years ago

sorry, my bad

● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL)
   Active: active (running) since Thu 2017-04-13 20:54:01 CEST; 3s ago
  Process: 10608 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/S                                                                         UCCESS)
   CGroup: /system.slice/pihole-FTL.service
           └─10633 /usr/bin/pihole-FTL
technicalpyro commented 7 years ago

ok f it still isnt updating use sudo killall pihole-FTL

then run the update

also can use pihole-FTL version to determine what version you are currently running

nin9s commented 7 years ago

its already 2.2 - thanks

DL6ER commented 7 years ago

Yeah, according to your output the update went fine, but only the restarting hasn't worked, because, as @technicalpyro said, there was still a running process inside the debugger (which cannot be terminated from the outside of the debugger).

Hmm, still it is strange that your system shows things like Notice: Increasing queries struct size from 30000 to 40000 (1204775.00 B) because that last number should really be shown in MB (and that seems to work on every other system I've seen).

Could it have something to do that I've disabled swapping completely?

? Is there swapping enabled by default? I'd guess no because that would almost certainly kill any SD card in a short time frame

nin9s commented 7 years ago

? Is there swapping enabled by default? I'd guess no because that would almost certainly kill any SD card in a short time frame

thats the reason I have disabled it

indeed it is definetely enabled by default ! I had to manually disable it and additionally mounted some folders including /var/log/ into ramfs

right now the debug is spamming me with this

[Thread 0xb5b3f460 (LWP 24147) exited]
[New Thread 0xb5b3f460 (LWP 24151)]
[Thread 0xb5b3f460 (LWP 24151) exited]
[New Thread 0xb5b3f460 (LWP 24155)]
[Thread 0xb5b3f460 (LWP 24155) exited]
[New Thread 0xb5b3f460 (LWP 24171)]
[Thread 0xb5b3f460 (LWP 24171) exited]
[New Thread 0xb5b3f460 (LWP 24175)]
[Thread 0xb5b3f460 (LWP 24175) exited]
[New Thread 0xb5b3f460 (LWP 24191)]
[Thread 0xb5b3f460 (LWP 24191) exited]
[New Thread 0xb5b3f460 (LWP 24195)]
[Thread 0xb5b3f460 (LWP 24195) exited]
[New Thread 0xb5b3f460 (LWP 24211)]
[Thread 0xb5b3f460 (LWP 24211) exited]
[New Thread 0xb5b3f460 (LWP 24215)]
[Thread 0xb5b3f460 (LWP 24215) exited]
[New Thread 0xb5b3f460 (LWP 24219)]
[Thread 0xb5b3f460 (LWP 24219) exited]
[New Thread 0xb5b3f460 (LWP 24223)]
[Thread 0xb5b3f460 (LWP 24223) exited]
[New Thread 0xb5b3f460 (LWP 24227)]
[Thread 0xb5b3f460 (LWP 24227) exited]
[New Thread 0xb5b3f460 (LWP 24231)]
[Thread 0xb5b3f460 (LWP 24231) exited]
[New Thread 0xb5b3f460 (LWP 24247)]
[Thread 0xb5b3f460 (LWP 24247) exited]
[New Thread 0xb5b3f460 (LWP 24251)]
[Thread 0xb5b3f460 (LWP 24251) exited]
[New Thread 0xb5b3f460 (LWP 24267)]
[Thread 0xb5b3f460 (LWP 24267) exited]
[New Thread 0xb5b3f460 (LWP 24271)]
[Thread 0xb5b3f460 (LWP 24271) exited]
[New Thread 0xb5b3f460 (LWP 24287)]
[Thread 0xb5b3f460 (LWP 24287) exited]
[New Thread 0xb5b3f460 (LWP 24291)]
[Thread 0xb5b3f460 (LWP 24291) exited]
[New Thread 0xb5b3f460 (LWP 24295)]
[Thread 0xb5b3f460 (LWP 24295) exited]
[New Thread 0xb5b3f460 (LWP 24299)]
[Thread 0xb5b3f460 (LWP 24299) exited]
[New Thread 0xb5b3f460 (LWP 24303)]
[Thread 0xb5b3f460 (LWP 24303) exited]
[New Thread 0xb5b3f460 (LWP 24307)]
[Thread 0xb5b3f460 (LWP 24307) exited]
[New Thread 0xb5b3f460 (LWP 24323)]
[Thread 0xb5b3f460 (LWP 24323) exited]
[New Thread 0xb5b3f460 (LWP 24327)]
[Thread 0xb5b3f460 (LWP 24327) exited]
[New Thread 0xb5b3f460 (LWP 24343)]
[Thread 0xb5b3f460 (LWP 24343) exited]
[New Thread 0xb5b3f460 (LWP 24347)]
[Thread 0xb5b3f460 (LWP 24347) exited]
[New Thread 0xb5b3f460 (LWP 24363)]
[Thread 0xb5b3f460 (LWP 24363) exited]
[New Thread 0xb5b3f460 (LWP 24367)]
[Thread 0xb5b3f460 (LWP 24367) exited]
[New Thread 0xb5b3f460 (LWP 24372)]
[Thread 0xb5b3f460 (LWP 24372) exited]
[New Thread 0xb5b3f460 (LWP 24376)]
[Thread 0xb5b3f460 (LWP 24376) exited]
[New Thread 0xb5b3f460 (LWP 24380)]
[Thread 0xb5b3f460 (LWP 24380) exited]
[New Thread 0xb5b3f460 (LWP 24384)]
[Thread 0xb5b3f460 (LWP 24384) exited]
[New Thread 0xb5b3f460 (LWP 24388)]
[Thread 0xb5b3f460 (LWP 24388) exited]
[New Thread 0xb5b3f460 (LWP 24404)]
[Thread 0xb5b3f460 (LWP 24404) exited]
[New Thread 0xb5b3f460 (LWP 24408)]
[Thread 0xb5b3f460 (LWP 24408) exited]
[New Thread 0xb5b3f460 (LWP 24424)]
[Thread 0xb5b3f460 (LWP 24424) exited]
[New Thread 0xb5b3f460 (LWP 24428)]
[Thread 0xb5b3f460 (LWP 24428) exited]

but ftl is running fine atm

DL6ER commented 7 years ago

Interesting, my Raspberry Pi model B is now 5 years old and still running with the same SD card from the first day on in almost uninterrupted 24/7 operation. It has also been used as the daily Pi-hole device for the internal network for about one year. Still, I have never seen a problem with either the device itself or the SD card. It is now replaced by the (much better) NanoPi NEO which seems to be really the ideal device for Pi-hole. I disabled swap now as well, but according to free it wasn't really used before (only a few KB).

The "spamming" is fine - it shows you that there is activity going on. In your case, it is most likely that this corresponds to requests from the web interface. Try to close all web interface instances and they should go away (only new log events, but no periodic messages).

nin9s commented 7 years ago

Ive disabled it just to be save. I dont know how long it takes to kill a SD with standard OS swapping never heard about the nanoPI neo. what makes it so special?

DL6ER commented 7 years ago

It is very small and cheap (less than $10) while having about the same performance data as the current Raspberry Pi version 3. Also, it comes w/o a graphics card which is something I have never needed once on my Raspberry Pi devices (I don't even have a HDMI capable device at home).

nin9s commented 7 years ago

wow thats neat - Im thinking of getting one as I dont need a gpu as well

//edit: its still running but the debug shows

[2017-04-14 00:09:26.165] FATAL ERROR: Trying to access clients[23], but maximum is 0
[2017-04-14 00:09:26.166]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.167] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.167]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.168] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.169]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.170] FATAL ERROR: Trying to access clients[23], but maximum is 0
[2017-04-14 00:09:26.171]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.172] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.173]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.174] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.174]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.175] FATAL ERROR: Trying to access clients[23], but maximum is 0
[2017-04-14 00:09:26.176]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.177] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.178]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.179] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.180]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.180] FATAL ERROR: Trying to access clients[23], but maximum is 0
[2017-04-14 00:09:26.181]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.182] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.183]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.184] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.185]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.186] FATAL ERROR: Trying to access clients[23], but maximum is 0
[2017-04-14 00:09:26.187]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.187] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.188]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.189] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.190]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.191] FATAL ERROR: Trying to access clients[23], but maximum is 0
[2017-04-14 00:09:26.192]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.193] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.193]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.194] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.195]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.196] FATAL ERROR: Trying to access clients[19], but maximum is 0
[2017-04-14 00:09:26.197]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.198] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.199]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.199] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.200]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.201] FATAL ERROR: Trying to access clients[19], but maximum is 0
[2017-04-14 00:09:26.202]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.203] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.204]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.205] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.205]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.206] FATAL ERROR: Trying to access clients[19], but maximum is 0
[2017-04-14 00:09:26.207]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.208] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.209]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.210] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.211]              found in GC_thread() (line 103) in gc.c
[2017-04-14 00:09:26.212] FATAL ERROR: Trying to access clients[19], but maximum is 0
[2017-04-14 00:09:26.212]              found in GC_thread() (line 46) in gc.c
[2017-04-14 00:09:26.213] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.214]              found in GC_thread() (line 66) in gc.c
[2017-04-14 00:09:26.215] FATAL ERROR: Trying to access forwarded[0], but maximum is 0
[2017-04-14 00:09:26.216]              found in GC_thread() (line 103) in gc.c
DL6ER commented 7 years ago

Yeah, I have just been able to confirm this locally after flushing took place. I think that all issues that have been reported are due to illegal struct resizing after flushing. At least this cures the error flood you have seen. FTL ended up in undefined behavior when adding the first new client or forward server after a flushing event has happened. That was a scenario which I did never test intentionally.

If there are still more issues with the data structure, my sophisticated memory access validator (whose output you have seen above) should be able to identify them and warn in a similar manner.

Anyhow, it should be fixed in FTL v2.3

nin9s commented 7 years ago

looks like its fixed - never had an issue until now

thanks for your quick help @DL6ER & @technicalpyro

DL6ER commented 7 years ago

Closing since the issue has been resolved. @strrrn Feel free to re-open if you experience the same issue again. Please file a new issue report for other issues.

thegeekoftheworld commented 7 years ago

I'm having the same issue running Pi-hole Version v3.0.1 Web Interface Version v3.0.1 FTL Version v2.6.2 on a VM

[2017-05-07 00:00:01.745] Reading from /var/log/pihole.log (rw-r--r--) [2017-05-07 00:09:57.732] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [2017-05-07 00:09:57.732] ----------------------------> FTL crashed! <---------------------------- [2017-05-07 00:09:57.732] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [2017-05-07 00:09:57.732] > Please report a bug at https://github.com/pi-hole/FTL/issues [2017-05-07 00:09:57.732] > and include in your report already the following details: [2017-05-07 00:09:57.733] > [2017-05-07 00:09:57.733] > Received signal: Segmentation fault [2017-05-07 00:09:57.733] at address: 0 [2017-05-07 00:09:57.733] with code: SEGV_MAPERR (Address not mapped to object) [2017-05-07 00:09:57.733] > Memory usage (structs): 360516 [2017-05-07 00:09:57.733] > Memory usage (dynamic): 16612 [2017-05-07 00:09:57.733] > [2017-05-07 00:09:57.733] > Thank you for helping us to improve our FTL engine! [2017-05-07 00:09:57.733] FTL terminated! root@Master-VM:~# cat /var/log/pihole-FTL.log nc tricorder.pi-hole.net 9999 tx49qi0wib root@Master-VM:~# pihole -up ::: Checking for updates... ::: Pi-hole Core: up to date ::: FTL: up to date ::: Web Interface: up to date

::: Everything is up to date!

DL6ER commented 7 years ago

Thanks for reporting this!

However, as this is almost certainly another issue, could I ask you to file a new bug report? You can then also already include some more details which you can obtain by going into debug mode (see here for detailed instructions)