search

pi-hole / FTL

The Pi-hole FTL engine
https://pi-hole.net
Other
1.39k stars 196 forks source link

FTL CPU 98% then stopped working after pihole -r #384

Closed j2l closed 6 years ago

j2l commented 6 years ago

In raising this issue, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your issue:

How familiar are you with the codebase?: 1 (but I read other issues and tried to work it out)

[BUG | ISSUE] Expected Behaviour: FTL online

[BUG | ISSUE] Actual Behaviour: FTL offline sudo service pihole-FTL status -l /etc/init.d/pihole-FTL: ligne69: status : commande introuvable (unknown command) service pihole-FTL start

Not running
dnsmasq: failed to create listening socket for port 53: Permission denied

[BUG | ISSUE] Steps to reproduce: pihole -r (both repair and reset) pihole checkout master

Log file output [if available]

*** [ INITIALIZING ]
[i] 2018-09-19:13:46:44 debug log has been initiated.

*** [ INITIALIZING ] Sourcing setup variables
[i] Sourcing /etc/pihole/setupVars.conf...

*** [ DIAGNOSING ]: Core version
[i] Core: v4.0 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)
[i] Branch: master
[i] Commit: v4.0-0-gddbdb51

*** [ DIAGNOSING ]: Web version
[i] Web: v4.0 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)
[i] Branch: master
[i] Commit: v4.0-0-gaf8c926

*** [ DIAGNOSING ]: FTL version
[✓] FTL: vDev-8493df4 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)

*** [ DIAGNOSING ]: dnsmasq version
[i] 2.62

*** [ DIAGNOSING ]: lighttpd version
[i] 1.4.31

*** [ DIAGNOSING ]: php version
[i] 5.4.45

*** [ DIAGNOSING ]: Operating system
[✓] Raspbian GNU/Linux 7 (wheezy)

*** [ DIAGNOSING ]: SELinux
[i] SELinux not detected

*** [ DIAGNOSING ]: Processor

*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the eth0 interface:
   192.168.1.36/24 matches the IP found in /etc/pihole/setupVars.conf

[✗] No IPv6 address(es) found on the eth0 interace.

[i] Default IPv4 gateway: 192.168.1.1
   * Pinging 192.168.1.1...
[✓] Gateway responded.
[i] Default IPv6 gateway: 192.168.1.1
   * Pinging 192.168.1.1...
[✗] Gateway did not respond. (https://discourse.pi-hole.net/t/why-is-a-default-gateway-important-for-pi-hole/3546)

*** [ DIAGNOSING ]: Ports in use
[10000] is in use by miniserv.
[139] is in use by smbd
[22] is in use by sshd
[445] is in use by smbd
[51413] is in use by transmiss
[53] is in use by dnsmasq
[80] is in use by lighttpd
[9091] is in use by transmiss
[25] is in use by master
[3306] is in use by mysqld

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✓] dugganandcosolicitors.ie is 192.168.1.36 via localhost (127.0.0.1)
[✓] dugganandcosolicitors.ie is 192.168.1.36 via Pi-hole (192.168.1.36)
[✓] doubleclick.com is 216.58.198.206 via a remote, public DNS server (8.8.8.8)

*** [ DIAGNOSING ]: Pi-hole processes
[✗] dnsmasq daemon is 
[✗] lighttpd daemon is 
[✗] pihole-FTL daemon is 

*** [ DIAGNOSING ]: Setup variables
    DHCP_START=192.168.1.37
    DHCP_END=192.168.1.251
    DHCP_ROUTER=192.168.1.1
    DHCP_LEASETIME=24
    PIHOLE_DOMAIN=lan
    DHCP_IPv6=true
    DHCP_ACTIVE=false
    DNSMASQ_LISTENING=single
    DNS_FQDN_REQUIRED=true
    DNS_BOGUS_PRIV=true
    DNSSEC=false
    PIHOLE_INTERFACE=eth0
    IPV4_ADDRESS=192.168.1.36/24
    IPV6_ADDRESS=
    PIHOLE_DNS_1=208.67.222.222
    PIHOLE_DNS_2=208.67.220.220
    QUERY_LOGGING=true
    INSTALL_WEB=true
    LIGHTTPD_ENABLED=0

*** [ DIAGNOSING ]: Dashboard and block page
[✗] X-Header does not match or could not be retrieved.
HTTP/1.1 500 Internal Server Error
X-Powered-By: PHP/5.4.45-0+deb7u12
Content-type: text/html
Date: Wed, 19 Sep 2018 11:47:01 GMT
Server: lighttpd/1.4.31

[✓] X-Pi-hole: The Pi-hole Web interface is working!

*** [ DIAGNOSING ]: Gravity list
-rw-r--r-- 1 root root 5296419 Sep 19 13:05 /etc/pihole/gravity.list
   -----head of gravity.list------
   192.168.1.36 0.0.0.0
   192.168.1.36 0000mps.webpreview.dsl.net
   192.168.1.36 000.0x1f4b0.com
   192.168.1.36 0001.2waky.com

   -----tail of gravity.list------
   192.168.1.36 zzzezeroe.fr
   192.168.1.36 zzzpooeaz-france.com
   192.168.1.36 zzzrtrcm2.com
   192.168.1.36 zzztube.com

*** [ DIAGNOSING ]: contents of /etc/pihole

-rw-r--r-- 1 root root 381 Sep 19 13:45 /etc/pihole/adlists.list
   https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
   https://mirror1.malwaredomains.com/files/justdomains
   http://sysctl.org/cameleon/hosts
   https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
   https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
   https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
   https://hosts-file.net/ad_servers.txt

-rw-r--r-- 1 root root 42 Sep 19 13:05 /etc/pihole/local.list
   192.168.1.36 magenpi
   192.168.1.36 pi.hole

-rw-r--r-- 1 root root 234 Sep 19 13:04 /etc/pihole/logrotate
   /var/log/pihole.log {
    su root root
    daily
    copytruncate
    rotate 5
    compress
    delaycompress
    notifempty
    nomail
   }
   /var/log/pihole-FTL.log {
    su root root
    weekly
    copytruncate
    rotate 3
    compress
    delaycompress
    notifempty
    nomail
   }

-rw-r--r-- 1 root root 117 Jun  7 15:28 /etc/pihole/whitelist.txt
   raw.githubusercontent.com
   mirror1.malwaredomains.com
   sysctl.org
   zeustracker.abuse.ch
   s3.amazonaws.com
   hosts-file.net

*** [ DIAGNOSING ]: contents of /etc/dnsmasq.d

-rw-r--r-- 1 root root 1541 Sep 19 13:04 /etc/dnsmasq.d/01-pihole.conf
   addn-hosts=/etc/pihole/gravity.list
   addn-hosts=/etc/pihole/black.list
   addn-hosts=/etc/pihole/local.list
   localise-queries
   no-resolv
   cache-size=10000
   log-queries
   log-facility=/var/log/pihole.log
   local-ttl=2
   log-async
   server=208.67.222.222
   server=208.67.220.220
   domain-needed
   bogus-priv
   interface=eth0

*** [ DIAGNOSING ]: contents of /etc/lighttpd

-rw-r--r-- 1 root root 3027 Sep 19 13:04 /etc/lighttpd/lighttpd.conf
   server.modules = (
    "mod_access",
    "mod_accesslog",
    "mod_auth",
    "mod_expire",
    "mod_compress",
    "mod_redirect",
    "mod_setenv",
    "mod_rewrite"
   )
   server.document-root        = "/var/www/html"
   server.error-handler-404    = "pihole/index.php"
   server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
   server.errorlog             = "/var/log/lighttpd/error.log"
   server.pid-file             = "/var/run/lighttpd.pid"
   server.username             = "www-data"
   server.groupname            = "www-data"
   server.port                 = 80
   accesslog.filename          = "/var/log/lighttpd/access.log"
   accesslog.format            = "%{%s}t|%V|%r|%s|%b"
   index-file.names            = ( "index.php", "index.html", "index.lighttpd.html" )
   url.access-deny             = ( "~", ".inc", ".md", ".yml", ".ini" )
   static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
   compress.cache-dir          = "/var/cache/lighttpd/compress/"
   compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )
   include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
   include_shell "/usr/share/lighttpd/create-mime.assign.pl"
   include_shell "find /etc/lighttpd/conf-enabled -name '*.conf' -a ! -name 'letsencrypt.conf' -printf 'include \"%p\"
' 2>/dev/null"
   $HTTP["url"] =~ "^/admin/" {

       setenv.add-response-header = (
           "X-Pi-hole" => "The Pi-hole Web interface is working!",
           "X-Frame-Options" => "DENY"
       )
       $HTTP["url"] =~ ".ttf$" {

           setenv.add-response-header = ( "Access-Control-Allow-Origin" => "*" )
       }
   }
   $HTTP["url"] =~ "^/admin/\.(.*)" {
        url.access-deny = ("")
   }
   include_shell "cat external.conf 2>/dev/null"

*** [ DIAGNOSING ]: contents of /etc/cron.d

-rw-r--r-- 1 root root 1629 Sep 19 13:04 /etc/cron.d/pihole
   43 4   * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity
   00 00   * * *   root    PATH="$PATH:/usr/local/bin/" pihole flush once quiet
   @reboot root /usr/sbin/logrotate /etc/pihole/logrotate
   */10 *  * * *   root    PATH="$PATH:/usr/local/bin/" pihole updatechecker local
   9 14  * * *   root    PATH="$PATH:/usr/local/bin/" pihole updatechecker remote
   @reboot root    PATH="$PATH:/usr/local/bin/" pihole updatechecker remote reboot

*** [ DIAGNOSING ]: contents of /var/log/lighttpd

-rw-r--r-- 1 www-data www-data 61575 Sep 19 13:47 /var/log/lighttpd/error.log
   2018-09-16 06:28:51: (server.c.1242) logfiles cycled UID = 0 PID = 3726 
   2018-09-16 07:11:53: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 07:25:32: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 07:25:32: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 07:25:32: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 07:26:01: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 07:26:01: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 07:26:01: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 08:26:06: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 08:26:07: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 08:26:07: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 08:52:01: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

   2018-09-16 08:52:01: (mod_fastcgi.c.2683) FastCGI-stderr: PHP Fatal error:  Can't use function return value in write context in /usr/share/nginx/www/html/pihole/index.php on line 108

*** [ DIAGNOSING ]: contents of /var/log

-rw-r--r-- 1 pihole pihole 32751 Sep 19 13:37 /var/log/pihole-FTL.log
   [2018-09-19 12:34:05.240] Notice: Increasing overTime struct size from 4300 to 4400 (219.25 KB)
   [2018-09-19 12:34:28.316] Notice: Increasing overTime struct size from 4400 to 4500 (224.24 KB)
   [2018-09-19 12:34:51.501] Notice: Increasing overTime struct size from 4500 to 4600 (229.24 KB)
   [2018-09-19 12:35:13.929] Notice: Increasing overTime struct size from 4600 to 4700 (234.24 KB)
   [2018-09-19 12:35:29.706] Notice: Increasing overTime struct size from 4700 to 4800 (239.24 KB)
   [2018-09-19 12:35:57.488] Notice: Increasing overTime struct size from 4800 to 4900 (244.24 KB)
   [2018-09-19 12:36:24.204] Notice: Increasing overTime struct size from 4900 to 5000 (249.24 KB)
   [2018-09-19 12:36:44.865] FATAL: FTL received SIGTERM from PID/UID 4134/0, scheduled to exit gracefully
   [2018-09-19 12:37:19.000] 
   [2018-09-19 12:37:19.002] FTL branch: v2.13.2
   [2018-09-19 12:37:19.002] FTL version: v2.13.2
   [2018-09-19 12:37:19.003] FTL commit: 1de2b99c8a812d4d2dbe6b1e4644e20a4b14bb1e
   [2018-09-19 12:37:19.003] FTL date: 2018-01-07 17:34:24 -0800
   [2018-09-19 12:37:19.003] FTL user: pihole
   [2018-09-19 12:37:19.004] Notice: Found no readable FTL config file
   [2018-09-19 12:37:19.004]         Using default settings
   [2018-09-19 12:37:19.005] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
   [2018-09-19 12:37:19.005]    SOCKET_LISTENING: only local
   [2018-09-19 12:37:19.005]    TIMEFRAME: Rolling 24h
   [2018-09-19 12:37:19.006]    QUERY_DISPLAY: Show queries
   [2018-09-19 12:37:19.006]    AAAA_QUERY_ANALYSIS: Show AAAA queries
   [2018-09-19 12:37:19.006]    MAXDBDAYS: max age for stored queries is 365 days
   [2018-09-19 12:37:19.007]    RESOLVE_IPV6: Resolve IPv6 addresses
   [2018-09-19 12:37:19.007]    RESOLVE_IPV4: Resolve IPv4 addresses
   [2018-09-19 12:37:19.018]    DBINTERVAL: saving to DB file every minute

*** [ DIAGNOSING ]: Pi-hole log
-rw-r--r-- 1 dnsmasq root 532301752 Sep 19 13:46 /var/log/pihole.log
   -----head of pihole.log------
   Mar 28 21:17:38 dnsmasq[2081]: lecture /etc/pihole/gravity.list - 141993 adresses
   Mar 28 21:17:40 dnsmasq[2081]: query[A] 0.debian.pool.ntp.org from 192.168.1.36
   Mar 28 21:17:40 dnsmasq[2081]: forwarded 0.debian.pool.ntp.org to 208.67.222.222
   Mar 28 21:17:40 dnsmasq[2081]: forwarded 0.debian.pool.ntp.org to 208.67.220.220
   Mar 28 21:17:40 dnsmasq[2081]: reply 0.debian.pool.ntp.org is 212.83.179.156
   Mar 28 21:17:40 dnsmasq[2081]: reply 0.debian.pool.ntp.org is 94.23.220.217
   Mar 28 21:17:40 dnsmasq[2081]: reply 0.debian.pool.ntp.org is 80.74.64.2
   Mar 28 21:17:40 dnsmasq[2081]: reply 0.debian.pool.ntp.org is 92.222.82.98
   Mar 28 21:17:40 dnsmasq[2081]: query[A] 1.debian.pool.ntp.org from 192.168.1.36
   Mar 28 21:17:40 dnsmasq[2081]: forwarded 1.debian.pool.ntp.org to 208.67.220.220
   Mar 28 21:17:40 dnsmasq[2081]: reply 1.debian.pool.ntp.org is 211.233.40.78
   Mar 28 21:17:40 dnsmasq[2081]: reply 1.debian.pool.ntp.org is 91.220.110.116
   Mar 28 21:17:40 dnsmasq[2081]: reply 1.debian.pool.ntp.org is 194.190.168.1
   Mar 28 21:17:40 dnsmasq[2081]: reply 1.debian.pool.ntp.org is 62.210.36.58
   Mar 28 21:17:40 dnsmasq[2081]: query[A] 2.debian.pool.ntp.org from 192.168.1.36
   Mar 28 21:17:40 dnsmasq[2081]: forwarded 2.debian.pool.ntp.org to 208.67.220.220
   Mar 28 21:17:40 dnsmasq[2081]: reply 2.debian.pool.ntp.org is 178.33.111.49
   Mar 28 21:17:40 dnsmasq[2081]: reply 2.debian.pool.ntp.org is 129.250.35.250
   Mar 28 21:17:40 dnsmasq[2081]: reply 2.debian.pool.ntp.org is 195.154.189.15
   Mar 28 21:17:40 dnsmasq[2081]: reply 2.debian.pool.ntp.org is 193.70.90.148

********************************************
********************************************
[✓] ** FINISHED DEBUGGING! **

    * The debug log can be uploaded to tricorder.pi-hole.net for sharing with developers only.
    * For more information, see: https://pi-hole.net/2016/11/07/crack-our-medical-tricorder-win-a-raspberry-pi-3/
    * If available, we'll use openssl to upload the log, otherwise it will fall back to netcat.
[i] Debug script running in automated mode
    * Using openssl for transmission.

***********************************
***********************************
[✓] Your debug token is: 032rviz3yf
***********************************
***********************************

   * Provide the token above to the Pi-hole team for assistance at
   * https://discourse.pi-hole.net
   * Your log will self-destruct on our server after 48 hours.
   * A local copy of the debug log can be found at: /var/log/pihole_debug-sanitized.log

Device specifics Hardware Type: rPi B v1 OS: wheezy (ARM)

tail -n 40 /var/log/pihole-FTL.log

[2018-09-19 13:54:07.081] FTL version:
[2018-09-19 13:54:07.082] FTL commit: 8493df4
[2018-09-19 13:54:07.082] FTL date: 2018-08-05 13:40:30 -0700
[2018-09-19 13:54:07.083] FTL user: pihole
[2018-09-19 13:54:07.083] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2018-09-19 13:54:07.084]    SOCKET_LISTENING: all destinations
[2018-09-19 13:54:07.084]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2018-09-19 13:54:07.085]    MAXDBDAYS: max age for stored queries is 1 days
[2018-09-19 13:54:07.085]    RESOLVE_IPV6: Don't resolve IPv6 addresses
[2018-09-19 13:54:07.086]    RESOLVE_IPV4: Resolve IPv4 addresses
[2018-09-19 13:54:07.086]    DBINTERVAL: saving to DB file every minute
[2018-09-19 13:54:07.087]    DBFILE: Using /etc/pihole/pihole-FTL.db
[2018-09-19 13:54:07.087]    MAXLOGAGE: Importing up to 24.0 hours of log data
[2018-09-19 13:54:07.088] Notice: Changing privacy level from 0 to 3
[2018-09-19 13:54:07.088]    PRIVACYLEVEL: Set to 3
[2018-09-19 13:54:07.089]    IGNORE_LOCALHOST: Show queries from localhost
[2018-09-19 13:54:07.089]    BLOCKINGMODE: Null IPs for blocked domains
[2018-09-19 13:54:07.089]    REGEX_DEBUGMODE: Inactive
[2018-09-19 13:54:07.090] Finished config file parsing
[2018-09-19 13:54:07.091] INFO: No Regex file found
[2018-09-19 13:54:07.142] Database successfully initialized
[2018-09-19 13:54:07.162] Notice: Increasing queries struct size from 0 to 10000
[2018-09-19 13:54:07.162] Notice: Increasing domains struct size from 0 to 1000
[2018-09-19 13:54:07.163] Notice: Increasing clients struct size from 0 to 10
[2018-09-19 13:54:07.163] Notice: Increasing overTime struct size from 0 to 100
[2018-09-19 13:54:07.164] New forward server: 208.67.220.220 (0/0)
[2018-09-19 13:54:07.164] Notice: Increasing forwarded struct size from 0 to 4
[2018-09-19 13:54:07.165] New forward server: 208.67.222.222 (1/4)
[2018-09-19 13:54:07.345] Notice: Increasing overTime struct size from 100 to 200
[2018-09-19 13:54:07.728] Notice: Increasing queries struct size from 10000 to 20000
[2018-09-19 13:54:07.757] Imported 10470 queries from the long-term database
[2018-09-19 13:54:07.759]  -> Total DNS queries: 10470
[2018-09-19 13:54:07.759]  -> Cached DNS queries: 2418
[2018-09-19 13:54:07.760]  -> Forwarded DNS queries: 5423
[2018-09-19 13:54:07.760]  -> Exactly blocked DNS queries: 2627
[2018-09-19 13:54:07.760]  -> Unknown DNS queries: 2
[2018-09-19 13:54:07.761]  -> Unique domains: 815
[2018-09-19 13:54:07.761]  -> Unique clients: 8
[2018-09-19 13:54:07.761]  -> Known forward destinations: 2
[2018-09-19 13:54:07.762] Successfully accessed setupVars.conf

I wanted to fix pihole-FTL CPU at 98%, rebooted, but updating pihole flushed out FTL. No number or chart in dashboard, stuck on processing in Query Log.

Located in Europe and network is down at night, so debug token might not work sometimes.

DL6ER commented 6 years ago

/etc/init.d/pihole-FTL: ligne69: status : commande introuvable (unknown command)

What is in line 69 ? In the Pi-hole provided version of this file, there is an echo and I double your system would say unknown command.

[✓] Raspbian GNU/Linux 7 (wheezy)

What kind of device do you try running Pi-hole on? Note that we officially don't support Wheezy as it is really outdated and does not receive any security updates since a long time.

[53] is in use by dnsmasq

dnsmasq needs to be disabled as Pi-hole v4.0+ ships its own DNS resolver (based on dnsmasq) onboard of pihole-FTL which needs access to port 53.

j2l commented 6 years ago

Thank you @DL6ER, here are details to reply: /etc/init.d/pihole-FTL 69 status pihole-FTL Indeed, it's a unknown command on wheezy.

I have this running Raspberry Pi 1 Model B box since 2012 (pre-order), behind the home router. It was refurbished as a Pi-Hole server a few months ago, Until recently, it was running nice on it for all the family 😄

OK, so I'll try to upgrade and disable dnsmasq and let you know.

j2l commented 6 years ago 
root@magenpi:~# sudo service dnsmasq stop
[ ok ] Stopping DNS forwarder and DHCP server: dnsmasq.
root@magenpi:~# sudo service pihole-FTL start
Not running
dnsmasq: failed to create listening socket for port 53: Permission denied

Any idea?

j2l commented 6 years ago

Forget my previous comment, upgrade didn't work, I had to format the SD Card to install Stretch but it works fine now. Up and running. Just need to setup up the rest 😞 Pi-hole Version v4.0 Web Interface Version v4.0 FTL Version v4.0