search

pi-hole / FTL

The Pi-hole FTL engine
https://pi-hole.net
Other
1.37k stars 197 forks source link

lost connection to API occurring randomly #157

Closed technicalpyro closed 6 years ago

technicalpyro commented 6 years ago

In raising this issue, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your issue:

How familiar are you with the codebase?:

{5}

[BUG | ISSUE] Expected Behaviour: pihole FTL log be openable and status returned correctly

[BUG | ISSUE] Actual Behaviour:

pi@raspberrypi:/var/log $ pihole-FTL status
FATAL: Opening of FTL log (/var/log/pihole-FTL.log) failed!
       Make sure it exists and is writeable by user pi

[BUG | ISSUE] Steps to reproduce:

-latest Developement branch -run pihole-FTL status

-

Log file output [if available]

2017-11-03 00:00:03.255] NOTICE: pihole.log has been flushed
[2017-11-03 00:00:03.256]   Resetting internal data structure
[2017-11-03 00:00:03.256]   Queries in memory before flushing: 17694
[2017-11-03 00:00:06.894] Gravity list entries: 1210427
[2017-11-03 00:00:06.895] Blacklist entries: 2
[2017-11-03 00:00:06.895] No wildcard blocking list present
[2017-11-03 00:00:06.895] Reading from /var/log/pihole.log.1 (rw-r--r--)
[2017-11-03 00:00:06.895] Notice: Increasing queries struct size from 0 to 10000 (320.45 KB)
[2017-11-03 00:00:06.895] Notice: Increasing overTime struct size from 0 to 100 (323.65 KB)
[2017-11-03 00:00:06.895] Notice: Increasing forwarded struct size from 0 to 4 (323.72 KB)
[2017-11-03 00:00:06.897] New forward server: 208.67.220.220 resolver2.opendns.com (0/4)
[2017-11-03 00:00:06.897] Notice: Increasing domains struct size from 0 to 1000 (343.76 KB)
[2017-11-03 00:00:06.897] Notice: Increasing clients struct size from 0 to 10 (343.95 KB)
[2017-11-03 00:00:06.897] New client: 172.16.1.100 Sean-Laptop (0/10)
[2017-11-03 00:00:06.898] New forward server: 208.67.222.222 resolver1.opendns.com (1/4)
[2017-11-03 00:00:06.899] New client: 127.0.0.1 localhost (1/10)
[2017-11-03 00:00:06.904] New client: 172.16.1.101 Sean-MBP (2/10)
[2017-11-03 00:00:06.914] New client: 172.16.1.105 Sean-Phone (3/10)
[2017-11-03 00:00:07.141] Notice: Increasing overTime struct size from 100 to 200 (351.59 KB)
[2017-11-03 00:00:07.253] Reading from /var/log/pihole.log (rw-r--r--)
[2017-11-03 09:16:18.638] Notice: Increasing overTime struct size from 200 to 300 (356.80 KB)
[2017-11-03 12:26:47.498] dnsmasq process restarted
-rw-r--r--  1 pihole      pihole     1647 Nov  3 12:26 pihole-FTL.log
-rw-r--r--  1 pihole      pihole     1745 Nov  3 00:00 pihole-FTL.log.1

Device specifics

Hardware Type: rPi, VPS, etc OS:

This template was created based on the work of udemy-dl.

AzureMarker commented 6 years ago

status is not an FTL command line parameter, so it tries to start up FTL instead (which fails because it's run under the pi user). There doesn't appear to be anything wrong here?

technicalpyro commented 6 years ago

i ran status as i have the lost connection to API error on the web interface thought the root of the issue was with the log file FTL showing offline> attempted status > led me to error postred

AzureMarker commented 6 years ago

Did you mean to run sudo service pihole-FTL status?

technicalpyro commented 6 years ago

my intent was to determine why i lost api connection and in doing so found myself unable to even get the proper status return

However running sudo pihole-FTL status returned a working instance so perhaps it was just missing the need for sudo

AzureMarker commented 6 years ago

No, that command does not help you. There is no status command built into FTL, and by running pihole-FTL under root you were only running a new instance of FTL under the root user until you stopped the command (it normally runs under pihole).

The best way to determine why FTL is not working is by checking its log and/or creating a debug log with pihole -d and looking through that.

technicalpyro commented 6 years ago

6jiuq5ef4m debug token it appears again that the API connection is lost however FTL is running if this is not FTL but rather wemin related let me know i will post in the admin interface area as well

AzureMarker commented 6 years ago

What's the contents of /var/log/pihole-FTL.log after you run sudo service pihole-FTL restart

technicalpyro commented 6 years ago 
[2017-11-14 19:45:40.998] ########## FTL started! ##########
[2017-11-14 19:45:40.999] FTL branch: (no branch)
[2017-11-14 19:45:40.999] FTL hash: v2.11.1
[2017-11-14 19:45:40.999] FTL date: 2017-10-08 12:35:40 -0700
[2017-11-14 19:45:40.999] FTL user: pihole
[2017-11-14 19:45:40.999] Notice: Found no readable FTL config file
[2017-11-14 19:45:40.999]         Using default settings
[2017-11-14 19:45:40.999] Starting config file parsing
[2017-11-14 19:45:40.999]    SOCKET_LISTENING: only local
[2017-11-14 19:45:40.999]    TIMEFRAME: Rolling 24h
[2017-11-14 19:45:40.999]    QUERY_DISPLAY: Show queries
[2017-11-14 19:45:41.000]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2017-11-14 19:45:41.000]    MAXDBDAYS: max age for stored queries is 365 days
[2017-11-14 19:45:41.000] Finished config file parsing
[2017-11-14 19:45:41.010] Found no other running pihole-FTL process
[2017-11-14 19:45:41.013] PID of FTL process: 28743
[2017-11-14 19:45:44.833] Gravity list entries: 1217037
[2017-11-14 19:45:44.833] Blacklist entries: 2
[2017-11-14 19:45:44.833] No wildcard blocking list present
[2017-11-14 19:45:44.834] Database initialized
[2017-11-14 19:45:44.834] Starting initial log file parsing
[2017-11-14 19:45:44.835] Reading from /var/log/pihole.log.1 (rw-r--r--)
[2017-11-14 19:45:44.835] Notice: Increasing forwarded struct size from 0 to 4 (160.00 B)
[2017-11-14 19:45:44.870] New forward server: 208.67.220.220 resolver2.opendns.com (0/4)
[2017-11-14 19:45:44.870] Notice: Increasing overTime struct size from 0 to 100 (3.40 KB)
[2017-11-14 19:45:44.909] New forward server: 208.67.222.222 resolver1.opendns.com (1/4)
[2017-11-14 19:45:45.322] Notice: Increasing overTime struct size from 100 to 200 (7.63 KB)
[2017-11-14 19:45:45.393] Notice: Increasing queries struct size from 0 to 10000 (327.77 KB)
[2017-11-14 19:45:45.394] Notice: Increasing domains struct size from 0 to 1000 (347.77 KB)
[2017-11-14 19:45:45.394] Notice: Increasing clients struct size from 0 to 10 (347.95 KB)
[2017-11-14 19:45:45.394] New client: 127.0.0.1 localhost (0/10)
[2017-11-14 19:45:45.394] New client: 172.16.1.100 Sean-Laptop (1/10)
[2017-11-14 19:45:45.445] New client: 172.16.1.105 Sean-Phone (2/10)
[2017-11-14 19:45:45.504] Reading from /var/log/pihole.log (rw-r--r--)
[2017-11-14 19:45:45.725] New client: 172.16.1.106 Sean-iPad (3/10)
[2017-11-14 19:45:46.209] Notice: Increasing overTime struct size from 200 to 300 (360.86 KB)
[2017-11-14 19:45:47.078] Notice: Increasing queries struct size from 10000 to 20000 (690.20 K$
[2017-11-14 19:45:47.354] New client: 172.16.1.102 raspberrypi (4/10)
[2017-11-14 19:45:47.426] Finished initial log file parsing
[2017-11-14 19:45:47.426]  -> Total DNS queries: 11866
[2017-11-14 19:45:47.426]  -> Cached DNS queries: 3577
[2017-11-14 19:45:47.426]  -> Blocked DNS queries: 4455
[2017-11-14 19:45:47.426]  -> Unknown DNS queries: 0
[2017-11-14 19:45:47.426]  -> Unique domains: 850
[2017-11-14 19:45:47.426]  -> Unique clients: 5
[2017-11-14 19:45:47.426] Successfully accessed setupVars.conf
[2017-11-14 19:45:47.427] Listening on port 4711 for incoming connections
AzureMarker commented 6 years ago

Everything looks good there, what's the output of this on your Pi-hole?

echo ">stats" | nc localhost 4711
technicalpyro commented 6 years ago 
pi@raspberrypi:~ $ echo ">stats" | nc localhost 4711
domains_being_blocked 1217039
dns_queries_today 12295
ads_blocked_today 4455
ads_percentage_today 36.234241
unique_domains 1041
queries_forwarded 4188
queries_cached 3652
clients_ever_seen 5
unique_clients 5
---EOM---

which is what i would expect since restarting

i could potentially run it in gcc and see what comes out seems to occur once a week ish

AzureMarker commented 6 years ago

What's the output of curl -i http://pi.hole/admin/api.php

technicalpyro commented 6 years ago 
pi@raspberrypi:~ $ curl http://pi.hole/admin/api.php
{"domains_being_blocked":1217039,"dns_queries_today":12311,"ads_blocked_today":4455,"ads_percentage_today":36.187149,"unique_domains":1041,"queries_forwarded":4197,"queries_cached":3659,"clients_ever_seen":5,"unique_clients":5}
AzureMarker commented 6 years ago

If you are able to get the same kind of output from your client device, then the web interface should be working fine.

technicalpyro commented 6 years ago

I am at this time but until it fails again I won't be able to determine the failure point.

technicalpyro commented 6 years ago

so it died on me again will be running in a debug to attempt to see what is happening when i use curl on the api address i get FTL not running so im not sure what is causing it to quit but running it in a debug should help figure that out

EDIT running FTL in gdb to determine failure point will report back when i know more

Th3M3 commented 6 years ago

@technicalpyro If your're on the development branch, are you using the FTL's development branch too?

Some days ago I run pihole -up and afterwards the FTL was crashing every time I added a domain to black/whitelist or if I changed one of the settings. On the dashboard, instead of the temperature a red circle and FTL offline was shown. Then I found out that I need to switch the FTL's branch manually to development, with command pihole checkout ftl development.

Since that FTL didn't crash anymore for me.

technicalpyro commented 6 years ago

Result of gdb

[2017-11-16 22:00:00.020]         There are queries that have not been saved
[Thread 0x75bff470 (LWP 28336) exited]
[New Thread 0x75bff470 (LWP 28365)]
[Thread 0x75bff470 (LWP 28365) exited]
[New Thread 0x75bff470 (LWP 28368)]
[Thread 0x75bff470 (LWP 28368) exited]
[New Thread 0x75bff470 (LWP 28371)]
[Thread 0x75bff470 (LWP 28371) exited]
[New Thread 0x75bff470 (LWP 28374)]
[Thread 0x75bff470 (LWP 28374) exited]
[New Thread 0x75bff470 (LWP 28377)]
[Thread 0x75bff470 (LWP 28377) exited]
[New Thread 0x75bff470 (LWP 28432)]
[Thread 0x75bff470 (LWP 28432) exited]
[New Thread 0x75bff470 (LWP 28435)]
[Thread 0x75bff470 (LWP 28435) exited]
[New Thread 0x75bff470 (LWP 28438)]
[Thread 0x75bff470 (LWP 28438) exited]
[New Thread 0x75bff470 (LWP 28441)]
[Thread 0x75bff470 (LWP 28441) exited]
[New Thread 0x75bff470 (LWP 28444)]
[Thread 0x75bff470 (LWP 28444) exited]
[New Thread 0x75bff470 (LWP 28447)]
[Thread 0x75bff470 (LWP 28447) exited]
[New Thread 0x75bff470 (LWP 28450)]
[Thread 0x75bff470 (LWP 28450) exited]
[New Thread 0x75bff470 (LWP 28453)]
[Thread 0x75bff470 (LWP 28453) exited]
[New Thread 0x75bff470 (LWP 28456)]
[Thread 0x75bff470 (LWP 28456) exited]
[2017-11-16 22:00:46.295] dnsmasq process restarted

Thread 1 "pihole-FTL" received signal SIGHUP, Hangup.
0x76f042a4 in select () at ../sysdeps/unix/syscall-template.S:84
84      ../sysdeps/unix/syscall-template.S: No such file or directory.

(gdb) backtrace
#0  0x76f042a4 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x0040b59e in sleepms (milliseconds=<optimized out>) at daemon.c:204
#2  0x0040a602 in main (argc=<optimized out>, argv=<optimized out>) at main.c:82
AzureMarker commented 6 years ago

If you are on the development branch of core/web, you are expected to have the development version of FTL. In this case, the development version of core sends a SIGHUP to FTL when dnsmasq is restarted, which the development version of FTL handles correctly: https://github.com/pi-hole/FTL/pull/151. If you are not on that version though, you could have unexpected results such as what you currently see.

technicalpyro commented 6 years ago 
pi@raspberrypi:~ $ pihole checkout ftl development
  Please note that changing branches severely alters your Pi-hole subsystems
  Features that work on the master branch, may not on a development branch
  This feature is NOT supported unless a Pi-hole developer explicitly asks!
  Have you read and understood this? [y/N] y

  [✓] Detected ARM-hf architecture (armv7+)
  [✓] Branch development exists
  [i] Installing FTL...pihole-FTL-arm-linux-gnueabihf: FAILED
sha1sum: WARNING: 1 computed checksum did NOT match
  [✗] Installing FTL

not sure if ive missed a step somewhere but attempting to run the dev branch errors me out

AzureMarker commented 6 years ago

I'm getting this too, we're investigating...

technicalpyro commented 6 years ago

ok i have the day off. So if there is anything i can do let me know

technicalpyro commented 6 years ago 
  [i] Checking for existing FTL binary...
  [i] Downloading and Installing FTL...curl: (7) Failed to connect to github-production-release-asset-2e65be.s3.amazonaws.com port 443: Connection refused
sha1sum: pihole-FTL-arm-linux-gnueabihf.sha1: No such file or directory
  [✗] Downloading and Installing FTL
  Error: Download of binary from Github failed

This sheds some light i think ?

AzureMarker commented 6 years ago

We think we pinned down the problem. It was on our end. Try to install the development version again.

technicalpyro commented 6 years ago 
pi@raspberrypi:~ $ pihole checkout ftl development
  Please note that changing branches severely alters your Pi-hole subsystems
  Features that work on the master branch, may not on a development branch
  This feature is NOT supported unless a Pi-hole developer explicitly asks!
  Have you read and understood this? [y/N] y

  [✓] Detected ARM-hf architecture (armv7+)
  [✓] Branch development exists
  [✓] Installing FTL
pi@raspberrypi:~ $ pihole -up
  [i] Checking for updates...
  [i] Pi-hole Core:     up to date
  [i] FTL:              update available

  [i] FTL out of date

  [i] FTL Checks...
  [✓] Detected ARM-hf architecture (armv7+)
  [i] Checking for existing FTL binary...
  [i] Downloading and Installing FTL...curl: (7) Failed to connect to github-production-release-asset-2e65be.s3.amazonaws.com port 443: Connection refused
  [✗] Downloading and Installing FTL
  Error: URL not found

  [i] Web Interface:    up to date

  [i] FTL version is now at

  [✓] Starting pihole-FTL service

  [✓] Enabling pihole-FTL service to start on reboot
pi@raspberrypi:~ $ pihole -v
  Pi-hole version is v3.1-464-gf63267a (Latest: v3.1.4)
  AdminLTE version is v3.0.1-305-g40e6c5f (Latest: v3.1)
  FTL version is vDev-dce2377 (Latest: v2.11.1)

makes me unsure if it worked or not

AzureMarker commented 6 years ago

That looks like a GitHub issue, does it work now?

technicalpyro commented 6 years ago

yes everything is working now except the update i am on the ftl development branch it just seems to be a github issue preventing me from updating at the moment 

pi@raspberrypi:~ $ curl github-production-release-asset-2e65be.s3.amazonaws.com
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>4BDDB501006319B1</RequestId><HostId>OJqcFQRulCeFCBgdF4P/i8Q0DouFawlICWieHpGho18SDtJ2x3ykzqekw1ve2/MhAI9jxekHfK8=</H

perhaps the hard coded link is wrong on my end or something weird like that

AzureMarker commented 6 years ago

You might have hit the GitHub API too many times while checking for updates. Also, I'm not sure if the update command supports downloading the development version of FTL yet.

technicalpyro commented 6 years ago

waited 12+ hours and tried again same error

technicalpyro commented 6 years ago

seems to have worked now thanks for all the help