FTL Crash issues? Read this thread first!

WiredLife commented 4 years ago

In raising this issue, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your issue:

[X] I have read and understood the contributors guide.
[X] The issue I am reporting can be replicated
[X] The issue I am reporting isn't a duplicate

How familiar are you with the codebase?:

1

[BUG | ISSUE] Expected Behaviour:

[BUG | ISSUE] Actual Behaviour: pi-hole on my 2 completly different systems crashed @ nearly the same time.

[BUG | ISSUE] Steps to reproduce:

-

Log file output [if available] RPi4 Log:

[2020-03-04 00:18:39.277 4096] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-04 00:18:39.277 4096] ---------------------------->  FTL crashed!  <----------------------------
[2020-03-04 00:18:39.277 4096] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-04 00:18:39.277 4096] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-03-04 00:18:39.278 4096] and include in your report already the following details:
[2020-03-04 00:18:39.278 4096] FTL has been running for 106087 seconds
[2020-03-04 00:18:39.278 4096] FTL branch: master
[2020-03-04 00:18:39.278 4096] FTL version: v4.3.1
[2020-03-04 00:18:39.278 4096] FTL commit: b60d63f
[2020-03-04 00:18:39.278 4096] FTL date: 2019-05-25 21:37:26 +0200
[2020-03-04 00:18:39.278 4096] FTL user: started as pihole, ended as pihole
[2020-03-04 00:18:39.278 4096] Received signal: Segmentation fault
[2020-03-04 00:18:39.278 4096]      at address: 0
[2020-03-04 00:18:39.278 4096]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-04 00:18:39.279 4096] Backtrace:
[2020-03-04 00:18:39.279 4096] B[0000]: /usr/bin/pihole-FTL(+0x1a25c) [0x47125c]
[2020-03-04 00:18:39.279 4096] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d4c130]
[2020-03-04 00:18:39.279 4096] B[0002]: /usr/bin/pihole-FTL(+0x32798) [0x489798]
[2020-03-04 00:18:39.279 4096] B[0003]: /usr/bin/pihole-FTL(receive_query+0x5d1) [0x48a4ce]
[2020-03-04 00:18:39.279 4096] B[0004]: /usr/bin/pihole-FTL(+0x40ed6) [0x497ed6]
[2020-03-04 00:18:39.279 4096] B[0005]: /usr/bin/pihole-FTL(main_dnsmasq+0xa3f) [0x49913c]
[2020-03-04 00:18:39.279 4096] B[0006]: /usr/bin/pihole-FTL(main+0x87) [0x46fe18]
[2020-03-04 00:18:39.279 4096] B[0007]: /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x10c) [0xb6d36718]
[2020-03-04 00:18:39.279 4096] Thank you for helping us to improve our FTL engine!
[2020-03-04 00:18:39.279 4096] FTL terminated!`

PC Log:

[2020-03-04 00:29:13.585 22510] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-04 00:29:13.585 22510] ---------------------------->  FTL crashed!  <----------------------------
[2020-03-04 00:29:13.585 22510] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-04 00:29:13.585 22510] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-03-04 00:29:13.585 22510] and include in your report already the following details:
[2020-03-04 00:29:13.585 22510] FTL has been running for 104188 seconds
[2020-03-04 00:29:13.585 22510] FTL branch: master
[2020-03-04 00:29:13.585 22510] FTL version: v4.3.1
[2020-03-04 00:29:13.585 22510] FTL commit: b60d63f
[2020-03-04 00:29:13.585 22510] FTL date: 2019-05-25 21:37:26 +0200
[2020-03-04 00:29:13.585 22510] FTL user: started as pihole, ended as pihole
[2020-03-04 00:29:13.585 22510] Received signal: Segmentation fault
[2020-03-04 00:29:13.585 22510]      at address: 0
[2020-03-04 00:29:13.585 22510]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-04 00:29:13.586 22510] Backtrace:
[2020-03-04 00:29:13.586 22510] B[0000]: /usr/bin/pihole-FTL(+0x255e5) [0x55d1fce755e5]
[2020-03-04 00:29:13.586 22510] B[0001]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fd2fa55c890]
[2020-03-04 00:29:13.586 22510] B[0002]: /usr/bin/pihole-FTL(+0x47a9a) [0x55d1fce97a9a]
[2020-03-04 00:29:13.586 22510] B[0003]: /usr/bin/pihole-FTL(receive_query+0x905) [0x55d1fce98e05]
[2020-03-04 00:29:13.586 22510] B[0004]: /usr/bin/pihole-FTL(+0x5db5b) [0x55d1fceadb5b]
[2020-03-04 00:29:13.586 22510] B[0005]: /usr/bin/pihole-FTL(main_dnsmasq+0xfdc) [0x55d1fceaf67c]
[2020-03-04 00:29:13.586 22510] B[0006]: /usr/bin/pihole-FTL(main+0xbc) [0x55d1fce73acc]
[2020-03-04 00:29:13.586 22510] B[0007]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fd2fa17ab97]
[2020-03-04 00:29:13.586 22510] B[0008]: /usr/bin/pihole-FTL(_start+0x2a) [0x55d1fce73bfa]
[2020-03-04 00:29:13.586 22510] Thank you for helping us to improve our FTL engine!
[2020-03-04 00:29:13.586 22510] FTL terminated!`

Device specifics

Hardware Type: RPi4 4GB and a PC OS: newest Raspbian on RPi4 and Ubuntu Server on PC

This template was created based on the work of udemy-dl.

rtgibbons commented 4 years ago

Just had my crash at 2020-03-03 16:40:31.791 CST

Working to get GDB but nothing is cooperating right now

WiredLife commented 4 years ago


GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 666
[New LWP 667]
[New LWP 668]
[New LWP 669]
[New LWP 670]
[New LWP 671]
[New LWP 672]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
__GI___poll (timeout=-1, nfds=6, fds=0xb1fab0) at ../sysdeps/unix/sysv/linux/poll.c:29
29      ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
(gdb) handle SIGHUP nostop SIGPIPE nostop
Signal        Stop      Print   Pass to program Description
SIGHUP        No        Yes     Yes             Hangup
SIGPIPE       No        Yes     Yes             Broken pipe
(gdb) continue
Continuing.

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
0x004ab798 in forward_query (udpfd=4, udpaddr=0x2, udpaddr@entry=0xbedf99d8, dst_addr=0xbedf99d8, dst_addr@entry=0xbedf9a08, dst_iface=3202324856, dst_iface@entry=2, header=<optimized out>, header@entry=0xb1be48, plen=47,
    plen@entry=30, now=<optimized out>, now@entry=4096, forward=0x1b854d0, ad_reqd=ad_reqd@entry=0, do_bit=0, do_bit@entry=11648584) at dnsmasq/forward.c:313
313     dnsmasq/forward.c: No such file or directory.
(gdb) backtrace
#0  0x004ab798 in forward_query (udpfd=4, udpaddr=0x2, udpaddr@entry=0xbedf99d8, dst_addr=0xbedf99d8, dst_addr@entry=0xbedf9a08, dst_iface=3202324856, dst_iface@entry=2, header=<optimized out>, header@entry=0xb1be48, plen=47,
    plen@entry=30, now=<optimized out>, now@entry=4096, forward=0x1b854d0, ad_reqd=ad_reqd@entry=0, do_bit=0, do_bit@entry=11648584) at dnsmasq/forward.c:313
#1  0x004ac4ce in receive_query (listen=listen@entry=0xb532c0, now=4096, now@entry=1583279886) at dnsmasq/forward.c:1641
#2  0x004b9ed6 in check_dns_listeners (now=now@entry=1583279886) at dnsmasq/dnsmasq.c:1657
#3  0x004bb13c in main_dnsmasq (argc=<optimized out>, argv=<optimized out>) at dnsmasq/dnsmasq.c:1108
#4  0x00491e18 in main (argc=<optimized out>, argv=<optimized out>) at main.c:71
(gdb) continue
Continuing.

Thread 1 "pihole-FTL" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0xb6daf230 in __GI_abort () at abort.c:79
#2  0x004932ba in SIGSEGV_handler (sig=<optimized out>, si=<optimized out>, unused=<optimized out>) at signals.c:66
#3  <signal handler called>
#4  0x004ab798 in forward_query (udpfd=4, udpaddr=0x2, udpaddr@entry=0xbedf99d8, dst_addr=0xbedf99d8, dst_addr@entry=0xbedf9a08, dst_iface=3202324856, dst_iface@entry=2, header=<optimized out>, header@entry=0xb1be48, plen=47,
    plen@entry=30, now=<optimized out>, now@entry=4096, forward=0x1b854d0, ad_reqd=ad_reqd@entry=0, do_bit=0, do_bit@entry=11648584) at dnsmasq/forward.c:313
#5  0x004ac4ce in receive_query (listen=listen@entry=0xb532c0, now=4096, now@entry=1583279886) at dnsmasq/forward.c:1641
#6  0x004b9ed6 in check_dns_listeners (now=now@entry=1583279886) at dnsmasq/dnsmasq.c:1657
#7  0x004bb13c in main_dnsmasq (argc=<optimized out>, argv=<optimized out>) at dnsmasq/dnsmasq.c:1108
#8  0x00491e18 in main (argc=<optimized out>, argv=<optimized out>) at main.c:71`

dschaper commented 4 years ago

Just had my crash at FTL date: 2019-05-25 21:37:26 +0200

That's the date the binary was compiled.

WiredLife commented 4 years ago


Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 1000
[New LWP 1001]
[New LWP 1002]
[New LWP 1003]
[New LWP 1004]
[New LWP 1005]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa1aa8a8bf9 in __GI___poll (fds=0x55844a51a500, nfds=4, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29      ../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht gefunden.
(gdb) handle SIGHUP nostop SIGPIPE nostop
Signal        Stop      Print   Pass to program Description
SIGHUP        No        Yes     Yes             Hangup
SIGPIPE       No        Yes     Yes             Broken pipe
(gdb) continue
Continuing.

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
0x000055844839fa9a in forward_query (udpfd=4, udpaddr=udpaddr@entry=0x7ffd19b78510, dst_addr=dst_addr@entry=0x7ffd19b784f0, dst_iface=dst_iface@entry=2, header=header@entry=0x55844a514e40, plen=44, plen@entry=43, now=1583280653,
    forward=0x55844a51c310, ad_reqd=0, do_bit=0) at dnsmasq/forward.c:313
313     dnsmasq/forward.c: Datei oder Verzeichnis nicht gefunden.
(gdb) backtrace
#0  0x000055844839fa9a in forward_query (udpfd=4, udpaddr=udpaddr@entry=0x7ffd19b78510, dst_addr=dst_addr@entry=0x7ffd19b784f0, dst_iface=dst_iface@entry=2, header=header@entry=0x55844a514e40, plen=44, plen@entry=43, now=1583280653,
    forward=0x55844a51c310, ad_reqd=0, do_bit=0) at dnsmasq/forward.c:313
#1  0x00005584483a0e05 in receive_query (listen=listen@entry=0x55844a52a720, now=now@entry=1583280653) at dnsmasq/forward.c:1641
#2  0x00005584483b5b5b in check_dns_listeners (now=now@entry=1583280653) at dnsmasq/dnsmasq.c:1657
#3  0x00005584483b767c in main_dnsmasq (argc=<optimized out>, argv=<optimized out>) at dnsmasq/dnsmasq.c:1108
#4  0x000055844837bacc in main (argc=1, argv=0x7ffd19b78908) at main.c:71
(gdb) continue
Continuing.

Thread 1 "pihole-FTL" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: Datei oder Verzeichnis nicht gefunden.
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fa1aa7d4801 in __GI_abort () at abort.c:79
#2  0x000055844837d675 in SIGSEGV_handler (sig=<optimized out>, si=<optimized out>, unused=<optimized out>) at signals.c:66
#3  <signal handler called>
#4  0x000055844839fa9a in forward_query (udpfd=4, udpaddr=udpaddr@entry=0x7ffd19b78510, dst_addr=dst_addr@entry=0x7ffd19b784f0, dst_iface=dst_iface@entry=2, header=header@entry=0x55844a514e40, plen=44, plen@entry=43, now=1583280653,
    forward=0x55844a51c310, ad_reqd=0, do_bit=0) at dnsmasq/forward.c:313
#5  0x00005584483a0e05 in receive_query (listen=listen@entry=0x55844a52a720, now=now@entry=1583280653) at dnsmasq/forward.c:1641
#6  0x00005584483b5b5b in check_dns_listeners (now=now@entry=1583280653) at dnsmasq/dnsmasq.c:1657
#7  0x00005584483b767c in main_dnsmasq (argc=<optimized out>, argv=<optimized out>) at dnsmasq/dnsmasq.c:1108
#8  0x000055844837bacc in main (argc=1, argv=0x7ffd19b78908) at main.c:71
(gdb) continue
Continuing.
Register können nicht ausgelesen werden: Kein passender Prozess gefunden.
Register können nicht ausgelesen werden: Kein passender Prozess gefunden.
(gdb) [Thread 0x7fa1a7ed7700 (LWP 1005) exited]
[Thread 0x7fa1a86d8700 (LWP 1004) exited]
[Thread 0x7fa1a8ed9700 (LWP 1003) exited]
[Thread 0x7fa1a96da700 (LWP 1002) exited]
[Thread 0x7fa1a9edb700 (LWP 1001) exited]

Program terminated with signal SIGABRT, Aborted.
The program no longer exists.`

antila commented 4 years ago

I also have the same issue. dnsmasq/forward.c:313 It started an hour ago or so, and it crashes FTL after the first lookup completes.

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
0x004ef798 in forward_query (udpfd=4, udpaddr=0x2, udpaddr@entry=0x7e8219f0, dst_addr=0x7e8219f0, dst_addr@entry=0x7e821a20, dst_iface=2122455440, dst_iface@entry=2,
    header=<optimized out>, header@entry=0xa86470, plen=42, plen@entry=31, now=<optimized out>, now@entry=4096, forward=0xa8d1a0, ad_reqd=ad_reqd@entry=0, do_bit=0,
    do_bit@entry=11035760) at dnsmasq/forward.c:313
313     dnsmasq/forward.c: No such file or directory.
(gdb) p forward
$1 = (struct frec *) 0xa8d1a0
(gdb) p forward[0]
$2 = {source = {sa = {sa_family = 2, sa_data = "\nw\300\250\001\001\000\000\000\000\000\000\000"}, in = {sin_family = 2, sin_port = 30474, sin_addr = {s_addr = 16885952},
      sin_zero = "\000\000\000\000\000\000\000"}, in6 = {sin6_family = 2, sin6_port = 30474, sin6_flowinfo = 16885952, sin6_addr = {__in6_u = {
          __u6_addr8 = "\000\000\000\000\000\000\000\000\060Ө\000\000\000\000", __u6_addr16 = {0, 0, 0, 0, 54064, 168, 0, 0}, __u6_addr32 = {0, 0, 11064112, 0}}},
      sin6_scope_id = 0}}, dest = {addr = {addr4 = {s_addr = 2265032896}, addr6 = {__in6_u = {__u6_addr8 = "\300\250\001\207LӨ\000\002\000\000\000\000\000\000", __u6_addr16 = {
            43200, 34561, 54092, 168, 2, 0, 0, 0}, __u6_addr32 = {2265032896, 11064140, 2, 0}}}, log = {keytag = 43200, algo = 34561, digest = 54092}, rcode = {rcode = 2265032896},
      dnssec = {class = 43200, type = 34561}}}, sentto = 0x0, rfd4 = 0x0, rfd6 = 0x0, iface = 2, orig_id = 9638, new_id = 49468, log_id = 144, fd = 4, forwardall = 1, flags = 256,
  time = 1583280319, hash = {0x23ff40cb <error: Cannot access memory at address 0x23ff40cb>, 0x6e77a7d <error: Cannot access memory at address 0x6e77a7d>,
    0x605b5f4b <error: Cannot access memory at address 0x605b5f4b>, 0xede55c47 <error: Cannot access memory at address 0xede55c47>,
    0x5b3a2cd1 <error: Cannot access memory at address 0x5b3a2cd1>, 0x0 <repeats 15 times>}, class = 1, work_counter = 49, stash = 0x0, stash_len = 42, dependent = 0x0,
  blocking_query = 0x0, next = 0xa8d0d8}
(gdb) p forward->sentto
$3 = (struct server *) 0x0
(gdb) p forward->sentto[0]
Cannot access memory at address 0x0
(gdb) p forward->sentto->addr
Cannot access memory at address 0x0
(gdb) p forward->sentto->addr.sa
Cannot access memory at address 0x0
(gdb) p forward->sentto->addr.sa.sa_family
Cannot access memory at address 0x0

[2020-03-04 00:09:29.869 5079] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-04 00:09:29.869 5079] ---------------------------->  FTL crashed!  <----------------------------
[2020-03-04 00:09:29.869 5079] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-03-04 00:09:29.870 5079] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-03-04 00:09:29.870 5079] and include in your report already the following details:
[2020-03-04 00:09:29.870 5079] FTL has been running for 275 seconds
[2020-03-04 00:09:29.870 5079] FTL branch: master
[2020-03-04 00:09:29.870 5079] FTL version: v4.3.1
[2020-03-04 00:09:29.870 5079] FTL commit: b60d63f
[2020-03-04 00:09:29.870 5079] FTL date: 2019-05-25 21:37:26 +0200
[2020-03-04 00:09:29.871 5079] FTL user: started as pihole, ended as pihole
[2020-03-04 00:09:29.871 5079] Received signal: Segmentation fault
[2020-03-04 00:09:29.873 5079]      at address: 0
[2020-03-04 00:09:29.874 5079]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-03-04 00:09:29.917 5079] Backtrace:
[2020-03-04 00:09:29.917 5079] B[0000]: /usr/bin/pihole-FTL(+0x1a25c) [0x4d725c]
[2020-03-04 00:09:29.918 5079] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0x76dec6c0]
[2020-03-04 00:09:29.918 5079] B[0002]: /usr/bin/pihole-FTL(+0x32798) [0x4ef798]
[2020-03-04 00:09:29.918 5079] B[0003]: /usr/bin/pihole-FTL(receive_query+0x5d1) [0x4f04ce]
[2020-03-04 00:09:29.918 5079] B[0004]: /usr/bin/pihole-FTL(+0x40ed6) [0x4fded6]
[2020-03-04 00:09:29.918 5079] B[0005]: /usr/bin/pihole-FTL(main_dnsmasq+0xa3f) [0x4ff13c]
[2020-03-04 00:09:29.918 5079] B[0006]: /usr/bin/pihole-FTL(main+0x87) [0x4d5e18]
[2020-03-04 00:09:29.919 5079] B[0007]: /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x114) [0x76dd6678]
[2020-03-04 00:09:29.919 5079] Thank you for helping us to improve our FTL engine!
[2020-03-04 00:09:29.920 5079] FTL terminated!

rtgibbons commented 4 years ago

via https://docs.pi-hole.net/ftldns/debugging/


Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
0x000055f95a111a9a in forward_query (udpfd=4, udpaddr=udpaddr@entry=0x7ffe6502a020, dst_addr=dst_addr@entry=0x7ffe6502a000, dst_iface=dst_iface@entry=0, header=header@entry=0x55f95b57d2f0, plen=39, 
    plen@entry=33, now=1583281752, forward=0x55f95b582220, ad_reqd=0, do_bit=0) at dnsmasq/forward.c:313
313     dnsmasq/forward.c: No such file or directory.
(gdb) backtrace
#0  0x000055f95a111a9a in forward_query (udpfd=4, udpaddr=udpaddr@entry=0x7ffe6502a020, dst_addr=dst_addr@entry=0x7ffe6502a000, dst_iface=dst_iface@entry=0, header=header@entry=0x55f95b57d2f0, plen=39, 
    plen@entry=33, now=1583281752, forward=0x55f95b582220, ad_reqd=0, do_bit=0) at dnsmasq/forward.c:313
#1  0x000055f95a112e05 in receive_query (listen=listen@entry=0x55f95b588030, now=now@entry=1583281752) at dnsmasq/forward.c:1641
#2  0x000055f95a127b5b in check_dns_listeners (now=now@entry=1583281752) at dnsmasq/dnsmasq.c:1657
#3  0x000055f95a12967c in main_dnsmasq (argc=<optimized out>, argv=<optimized out>) at dnsmasq/dnsmasq.c:1108
#4  0x000055f95a0edacc in main (argc=1, argv=0x7ffe6502a418) at main.c:71```

nullify005 commented 4 years ago

I was facing this as well this morning & disabling DNSSEC in the options appears to get FTL back to stable running (at least for the last 30mins at least)

sylveon commented 4 years ago

FTL also started crashing here an hour or two ago, after doing no changes to the setup in months.

WiredLife commented 4 years ago

yep i confirm must be something with dnssec if i open http://en.conn.internet.nl/connection/ it crashs constantly when i disable dnssec, it runs

sylveon commented 4 years ago

Disabling DNSSEC also fixed the constant crashing

dschaper commented 4 years ago

Is anyone using CloudFlare as the upstream service? And is DNSSEC enabled or disabled?

BPitts2 commented 4 years ago

Ah ha! Yes, I am using Cloudflare as up stream w/ DNSSEC

WiredLife commented 4 years ago

yep using cloudflare through stubby because of DoT and enabled dnssec

dschaper commented 4 years ago

OP on reddit mentions Cloudflare: https://www.reddit.com/r/pihole/comments/fd44dc/ftl_crashing/

sylveon commented 4 years ago

I'm using stubby to forward queries to Cloudflare using DNS over TLS w/ DNSSEC

kevindelaney commented 4 years ago

Exact same issue for me - using cloudflared. Disabling DNSSEC has me back up for now.

WiredLife commented 4 years ago

i will try to change the server @ stubby and report back if its running with another company :D

jdrch commented 4 years ago

Is anyone using CloudFlare as the upstream service? And is DNSSEC enabled or disabled?

CloudFlare + DNSSEC via dnscrypt-proxy.

WiredLife commented 4 years ago

with the google dns servers it runs fine

jdrch commented 4 years ago

Created a CloudFlare Support ticket: 1842464 (not sure if that's publicly accessible, but feel free to reference it)

jdrch commented 4 years ago

CloudFlare Community thread created.

WiredLife commented 4 years ago

maybe they're trying to exploit some vulnerabilities? 😄

dschaper commented 4 years ago

Meanwhile https://docs.pi-hole.net/guides/unbound/ is something to consider. Unbound as a local upstream and no longer depending on a service for upstreams.

pralor-bot commented 4 years ago

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/lost-connection-to-api/29062/2

jdrch commented 4 years ago

Which other DNS services besides CloudFlare support DNSCrypt and DNSSEC?

Meanwhile https://docs.pi-hole.net/guides/unbound/ is something to consider. Unbound as a local upstream and no longer depending on a service for upstreams.

I'm having the issue using dnscrypt-proxy, which I believe fulfills the same local upstream DNS functionality as unbound.

MattLParker commented 4 years ago

Op Here from https://www.reddit.com/r/pihole/comments/fd44dc/ftl_crashing/ I can finally start doing some real digging on root cause as well.

sylveon commented 4 years ago

@jdrch No. dnscrypt-proxy is a forwarding DNS resolver: it just forwards the request to 1.1.1.1. Meanwhile, unbound is a recursive DNS server: it provides the same service as 1.1.1.1 itself, meaning that it contacts the root DNS nameservers for the TLD nameserver, which it asks for the domain nameserver, and so on until it gets the actual IP.

DNS over TLS and DNS over HTTPS are not used between authoritative and recursive DNS server communication, only DNSSEC is used (if available), so you get the same result as contacting 1.1.1.1 encryption-wise. In fact, the RFC for DoT only mentions authoritative twice.

jdrch commented 4 years ago

@sylveon TIL, thanks. The performance hit does seem to be a significant drawback, though, which is bad for gaming ping times, among other things.

jdrch commented 4 years ago

Help is on the way! (Tweet from Cloudflare person)

sylveon commented 4 years ago

Most if not all games will use raw IPs as the overhead of a DNS lookup for each packet is already significant (if not a socket which binds after resolving the IP)

jdrch commented 4 years ago

Most if not all games will use raw IPs as the overhead of a DNS lookup for each packet is already significant (if not a socket which binds after resolving the IP)

That's interesting. I had ping problems - in Fortnite, specifically - until I changed this dnscrypt-proxy setting to its current value 🤷‍♂️:

## DoH: Disable TLS session tickets - increases privacy but also latency

tls_disable_session_tickets = false

And anyway tbh the Pi-hole official documentation talks about latency >1s. Yikes. Something drastic would have to happen for me to try that.

seamooose commented 4 years ago

Oh many i wish i had checked here about an hour ago. Spent a few hours trying to figure out what was wrong with FTL. Also disabled DNSSEC and it is all working again.

seamooose commented 4 years ago

ALL:

Update cloudflared (assuming you're using that). Its not a recent build, but it does appear to have fixed the issue...

Was: cloudflared version 2020.2.0 (built 2020-02-07-1652 UTC)

Now: cloudflared version 2020.2.1 (built 2020-02-27-1710 UTC)

sylveon commented 4 years ago

I'm not using cloudflared tho, so there's still an issue

seamooose commented 4 years ago

NEVERMIND. I was jumping the gun, just took a little while longer to crash.

ALL:

Update cloudflared (assuming you're using that). Its not a recent build, but it does appear to have fixed the issue...

Was: cloudflared version 2020.2.0 (built 2020-02-07-1652 UTC)

Now: cloudflared version 2020.2.1 (built 2020-02-27-1710 UTC)

vavrusa commented 4 years ago

Just to confirm the dnsmasq is crashing when in validating mode, NOT cloudflared and NOT dnsmasq in insecure mode right?

jdrch commented 4 years ago

Just to confirm the dnsmasq is crashing when in validating mode, NOT cloudflared and NOT dnsmasq in insecure mode right?

The crash occurs only when you're using Cloudflare DNS with DNSSEC enabled.

pralor-bot commented 4 years ago

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/dns-over-https-broken-after-v4-4-update/29066/2

fooflington commented 4 years ago

I am seeing this too (from 23:00 UTC last night). I use cloudflared and, having disabled DNSSEC, am stable again.

madpsy commented 4 years ago

I too started seeing this last night. I'm using 'Stubby' (rather than cloudflared) but with Cloudflare as upstream (DoH) in Stubby. Disabling DNSSEC in pihole settings (thankfully UI is still available!) solves this for me just now but obviously would like to be able to use DNSSEC.

k0nserv commented 4 years ago

So is the ssue with cloudflared itself and will it require an update or is it an upstream issue on Cloudflare's part that they will fix?

juanluispa commented 4 years ago

After disable DNSEC and get DNS relay back I update once. Then I reenabled DNSSEC and having connectivity FTL reports an update available but upgrading, it reports the error:

Error: URL https://github.com/pi-hole/ftl/releases/latest/download/pihole-FTL-arm-linux-gnueabihf not found

LOG:

[✓] Detected ARM-hf architecture (armv7+) [i] Checking for existing FTL binary... [i] Downloading and Installing FTL...curl: (6) Could not resolve host: github.com [✗] Downloading and Installing FTL Error: URL https://github.com/pi-hole/ftl/releases/latest/download/pihole-FTL-arm-linux-gnueabihf not found [✗] FTL Engine not installed

Unable to complete update, please contact Pi-hole Support

jedisct1 commented 4 years ago

If FTL is used behind dnscrypt-proxy, probably the most helpful thing you can do to help the FTL maintainers is to enable query logging in dnscrypt-proxy, and see what the last query was before FTL crashed.

Twanislas commented 4 years ago

It looks like Cloudflare confirmed it was an issue with cloudflared, see https://twitter.com/dinasaur_404/status/1235030960120315905

Quick workaround until it's fixed : sudo nano /etc/dnsmasq.d/01-pihole.conf, comment out dnssec, CTRL+O to save, then sudo pihole restartdns.

Been running steadily for 1h.

MattLParker commented 4 years ago

Tagging on, compiling most of the issues, it appears less to be cloudflared, but more so, Cloudflare DoH/DoT with dnssec. Cloudflared, dnscrypt and Stubby seem to be equally affected.

pralor-bot commented 4 years ago

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/raspbian-install-running-for-a-week-now-getting-ftl-offline-message/29075/2

madpsy commented 4 years ago

As I mentioned previously I use 'Stubby' to query Cloudflare over DoH so this can't just be an issue with cloudflared.

fredprod commented 4 years ago

As I mentioned previously I use 'Stubby' to query Cloudflare over DoH so this can't just be an issue with cloudflared.

I am using Stubby too, I switched to dns.google to not disable dnssec on pi-hole.

MattLParker commented 4 years ago

It sounds like through all the variations of problem, the problem is directly related to the data coming from Cloudflare over DoH and DoT, as mentioned earlier they seem to be aware of a problem and are working to fix it.

dschaper commented 4 years ago

This may be an externally triggered issue but we'd like to get as much info as we can to prevent others mistakes from causing us faults.

jedisct1

If FTL is used behind dnscrypt-proxy, probably the most helpful thing you can do to help the FTL maintainers is to enable query logging in dnscrypt-proxy, and see what the last query was before FTL crashed.

pi-hole / FTL

FTL Crash issues? Read this thread first! #705

-

-