pi-hole / FTL

The Pi-hole FTL engine
https://pi-hole.net
Other
1.34k stars 187 forks source link

[Pi-hole V6] Ftl crash because of database? #1898

Closed Gontier-Julien closed 3 months ago

Gontier-Julien commented 4 months ago

Versions

Platform

Expected behavior

Shouldn't crash

Actual behavior / bug

pi-hole randomly crash

Steps to reproduce

Steps to reproduce the behavior:

Let pi-hole run and randomly crash

Debug Token

Screenshots

If applicable, add screenshots to help explain your problem.

2024-03-02 04:56:17.026 [142/T143] INFO: Gravity database has been updated, reloading now
2024-03-02 04:56:17.039 [142/T143] INFO: Compiled 7 allow and 4 deny regex for 91 clients in 10.6 msec
2024-03-02 05:00:00.075 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 05:10:00.067 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 05:20:00.056 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 14 rows
2024-03-02 05:30:00.034 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 05:40:00.021 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 05:50:00.039 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 14 rows
2024-03-02 06:00:00.076 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 15 rows
2024-03-02 06:10:00.049 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 24 rows
2024-03-02 06:19:00.075 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 9 rows
2024-03-02 06:30:00.075 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 06:40:00.046 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 13 rows
2024-03-02 06:49:00.070 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 14 rows
2024-03-02 07:00:00.117 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 13 rows
2024-03-02 07:10:00.088 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 07:19:00.111 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 13 rows
2024-03-02 07:30:00.061 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 07:40:00.037 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 11 rows
2024-03-02 07:50:00.105 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 14 rows
2024-03-02 08:00:00.035 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 11 rows
2024-03-02 08:10:00.109 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 9 rows
2024-03-02 08:20:00.082 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 08:30:00.053 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 9 rows
2024-03-02 08:40:00.071 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 9 rows
2024-03-02 08:50:00.059 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 11 rows
2024-03-02 09:00:00.090 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 17 rows
2024-03-02 09:10:00.064 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 11 rows
2024-03-02 09:20:00.050 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 09:30:00.028 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 12 rows
2024-03-02 09:40:00.112 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 09:50:00.106 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 10:00:00.102 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 18 rows
2024-03-02 10:10:00.084 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 8 rows
2024-03-02 10:20:00.044 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 10:30:00.130 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 8 rows
2024-03-02 10:40:00.087 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 10:50:00.032 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 11:00:00.092 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 11:10:00.052 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 11:20:00.110 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 13 rows
2024-03-02 11:30:00.058 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 11:40:00.112 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 10 rows
2024-03-02 11:50:00.113 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 15 rows
2024-03-02 12:00:00.067 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 14 rows
2024-03-02 12:10:00.088 [142/T143] INFO: Size of /etc/pihole/pihole-FTL.db is 46.56 MB, deleted 20 rows
2024-03-02 12:19:00.090 [142/T143] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-03-02 12:19:00.090 [142/T143] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-03-02 12:19:00.090 [142/T143] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-03-02 12:19:00.090 [142/T143] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-03-02 12:19:00.090 [142/T143] INFO: and include in your report already the following details:
2024-03-02 12:19:00.090 [142/T143] INFO: FTL has been running for 73297 seconds
2024-03-02 12:19:00.090 [142/T143] INFO: FTL branch: development-v6
2024-03-02 12:19:00.090 [142/T143] INFO: FTL version: vDev-bffd2bd
2024-03-02 12:19:00.090 [142/T143] INFO: FTL commit: bffd2bd4
2024-03-02 12:19:00.090 [142/T143] INFO: FTL date: 2024-02-20 05:27:37 +0100
2024-03-02 12:19:00.090 [142/T143] INFO: FTL user: started as pihole, ended as pihole
2024-03-02 12:19:00.090 [142/T143] INFO: Compiled for linux/amd64 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-03-02 12:19:00.090 [142/T143] INFO: Process details: MID: 142
2024-03-02 12:19:00.090 [142/T143] INFO:                  PID: 142
2024-03-02 12:19:00.090 [142/T143] INFO:                  TID: 143
2024-03-02 12:19:00.090 [142/T143] INFO:                  Name: database
2024-03-02 12:19:00.090 [142/T143] INFO: Received signal: Segmentation fault
2024-03-02 12:19:00.090 [142/T143] INFO:      at address: 0x28
2024-03-02 12:19:00.090 [142/T143] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2024-03-02 12:19:00.090 [142/T143] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-03-02 12:19:00.090 [142/T143] INFO: ------ Listing content of directory /dev/shm ------
2024-03-02 12:19:00.090 [142/T143] INFO: File Mode User:Group      Size  Filename
2024-03-02 12:19:00.090 [142/T143] INFO: rwxrwxrwx root:root       280  .
2024-03-02 12:19:00.090 [142/T143] INFO: rwxr-xr-x root:root       340  ..
2024-03-02 12:19:00.090 [142/T143] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-03-02 12:19:00.090 [142/T143] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-03-02 12:19:00.090 [142/T143] INFO: rw------- pihole:pihole   111K  FTL-dns-cache
2024-03-02 12:19:00.090 [142/T143] INFO: rw------- pihole:pihole     8K  FTL-overTime
2024-03-02 12:19:00.090 [142/T143] INFO: rw------- pihole:pihole   885K  FTL-queries
2024-03-02 12:19:00.090 [142/T143] INFO: rw------- pihole:pihole    29K  FTL-upstreams
2024-03-02 12:19:00.091 [142/T143] INFO: rw------- pihole:pihole    86K  FTL-clients
2024-03-02 12:19:00.091 [142/T143] INFO: rw------- pihole:pihole   135K  FTL-domains
2024-03-02 12:19:00.091 [142/T143] INFO: rw------- pihole:pihole    82K  FTL-strings
2024-03-02 12:19:00.091 [142/T143] INFO: rw------- pihole:pihole    16  FTL-settings
2024-03-02 12:19:00.091 [142/T143] INFO: rw------- pihole:pihole   292  FTL-counters
2024-03-02 12:19:00.091 [142/T143] INFO: rw------- pihole:pihole    88  FTL-lock
2024-03-02 12:19:00.091 [142/T143] INFO: ---------------------------------------------------
2024-03-02 12:19:00.091 [142/T143] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-03-02 12:19:00.091 [142/T143] INFO: Thank you for helping us to improve our FTL engine!
2024-03-02 12:19:00.091 [142/T143] INFO: Waiting for threads to join
2024-03-02 12:19:00.801 [142/T145] INFO: Terminating resolver thread
2024-03-02 12:19:01.105 [142/T144] INFO: Terminating GC thread
2024-03-02 12:19:02.091 [142/T143] INFO: Thread database (0) is still busy, cancelling it.
2024-03-02 12:19:02.091 [142/T143] INFO: All threads joined
2024-03-02 12:19:02.091 [142/T143] ERROR: Error when obtaining outer SHM lock: Resource deadlock would occur
2024-03-02 12:19:02.091 [142/T143] ERROR: Error when obtaining inner SHM lock: Resource deadlock would occur
2024-03-02 12:19:10.414 [142M] ERROR: Error when obtaining outer SHM lock: Previous owner died
2024-03-02 12:19:10.414 [142M] ERROR: Error when obtaining inner SHM lock: Previous owner died
2024-03-02 12:24:42.513 [142M] ERROR: SQLite3: API call with invalid database connection pointer (21)
2024-03-02 12:24:42.513 [142M] ERROR: SQLite3: misuse at line 141504 of [e876e51a0e] (21)
2024-03-02 12:24:42.513 [142M] ERROR: get_client_groupids("::") - SQL error prepare: bad parameter or other API misuse
2024-03-02 12:24:42.513 [142M] ERROR: SQLite3: API call with invalid database connection pointer (21)
2024-03-02 12:24:42.513 [142M] ERROR: SQLite3: misuse at line 141504 of [e876e51a0e] (21)
2024-03-02 12:24:42.513 [142M] ERROR: get_client_groupids("::") - SQL error prepare: bad parameter or other API misuse

Additional context

Restarting the container fix it

Gontier-Julien commented 4 months ago

Also the web interface is still up when it happen, so it need to be manually restarted when it happens

DL6ER commented 4 months ago

Can this be triggered reliably somehow? As in: Does the crash repeatedly happen after the container restart?

If so, it'd be tremendously helpful is you could follow https://docs.pi-hole.net/ftldns/debugging/ and attach the gdb debugger to pihole-FTL so a backtrace can reveal where the crash is happening. The fact that you are running Pi-hole inside a container may make this more tricky but should still be straightforward enough to not cause too much additional trouble.

Gontier-Julien commented 4 months ago

Can this be triggered reliably somehow? As in: Does the crash repeatedly happen after the container restart?

If so, it'd be tremendously helpful is you could follow https://docs.pi-hole.net/ftldns/debugging/ and attach the gdb debugger to pihole-FTL so a backtrace can reveal where the crash is happening. The fact that you are running Pi-hole inside a container may make this more tricky but should still be straightforward enough to not cause too much additional trouble.

And it doesn't happens only when the contain restart, it happens even if it was the first time spinning it up. I need to restart the container manually since it doesn't stop by itself to restart.

I've already tried to debug it a little, and try to crash it reliably but with no success unfortunately :( It really happens at random, i tried to see if a domain or multiple domain crash it nothing, see if it a specific client maybe nothing.

The only thing i could thing of maybe it could be related to the absurdly high client number? Screenshot from 2024-03-05 00-53-40

But i really doubt it? From what i've understand in #1884 ? It show the same as in the PR from which client it is.

If you have any idea that i could test or anything tell me i'll try it

And i'll see for the debugging to trace it down tomorrow as i'm going to sleep now ^^

Gontier-Julien commented 4 months ago

Should I try to delete all database to see if it fix it?

rdwebdesign commented 4 months ago

Can you please generate a debug log and post here only the Token?

Using the web interface:

Gontier-Julien commented 4 months ago

Screenshot from 2024-03-05 12-20-59

I don't see the option on pi-hole V6? But i've generated a pi-hole debug log that i can upload

I'll think to do on again if it crash again.

rdwebdesign commented 4 months ago

Sorry... It was late and I used a saved reply with v5 text.

You can generate the log running pihole -d inside the container:

docker exec -it <container_name> pihole -d

When asked, answer "Y" to upload the log and then copy and paste only the Token.

Gontier-Julien commented 4 months ago

Sorry... It was late and I used a saved reply with v5 text.

You can generate the log running pihole -d inside the container:

docker exec -it <container_name> pihole -d

When asked, answer "Y" to upload the log and then copy and paste only the Token.

No worry's ^^

Here the token then ^^

B6ouWIHF

Gontier-Julien commented 3 months ago

If i see no crash until the end of the month i'll close that issue then, but so far i didn't encounter it again

Gontier-Julien commented 3 months ago

Closing this as it never happen since then 🤷