search

MichaIng / DietPi

Lightweight justice for your single-board computer!
https://dietpi.com/
GNU General Public License v2.0
4.8k stars 494 forks source link

DietPi Software | PiHole: "Queries over time" show only graph 1 hour #839

Closed k-plan closed 7 years ago

k-plan commented 7 years ago

170326-0001

Tested devices:

DietPi Version:

Installed software:

Install PiHole via dietpi-software

PiHole Version:

~# pihole -v
::: Pi-hole version is v2.13.2 (Latest version is v2.13.2)
::: Web-Admin version is v2.5.2 (Latest version is v2.5.2)

PiHole logfile:

root@NanoPi-Neo:~# cat /var/log/pihole.log
Mar 26 16:17:01 dnsmasq[9486]: gestartet, Version 2.72, Cachegröße 10000
Mar 26 16:17:01 dnsmasq[9486]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect
Mar 26 16:17:01 dnsmasq[9486]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 26 16:17:01 dnsmasq[9486]: Benutze Namensserver 192.168.1.1#53
Mar 26 16:17:01 dnsmasq[9486]: Benutze Namensserver 192.168.1.55#53
Mar 26 16:17:01 dnsmasq[9486]: /etc/hosts gelesen - 6 Adressen
Mar 26 16:17:01 dnsmasq[9486]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 26 16:17:10 dnsmasq[9486]: /etc/pihole/gravity.list gelesen - 347296 Adressen
Mar 26 16:21:41 dnsmasq[9486]: query[A] github.com from 192.168.1.102
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.1
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.55
Mar 26 16:21:41 dnsmasq[9486]: query[AAAA] github.com from 192.168.1.102
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.1
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.55
Mar 26 16:21:41 dnsmasq[9486]: reply github.com is 192.30.253.113
Mar 26 16:21:41 dnsmasq[9486]: reply github.com is 192.30.253.112
.....

Fiddle around a lot, e.g. fresh install pihole, different devices, starting with new images, install by hand or dietpi.txt, Ramlog 1 or 2 or no, but with no success. 😢

I think it is a problem in dietpi-logclear with dnsmasq:

https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-logclear#L108

because every hour at xx:17:01 dnsmasq log start new again.

Wondering, why no other DietPi user notice graphing issue in the PiHole web interface ... ❓

BTW: my main productive system, with a old and outdated PiHole installation, it works like a charm ...

170326-0002

Fourdee commented 7 years ago

@k-plan

Thanks for the report 👍

There is an exclusion for PiHole's log file clearing with DietPi-Ramlog. It should allow for 24 hour logs. Let me run some tests here and check source.

edit:

Log System: DietPi-Ramlog #2

Most likely this, Mode 2 probably moving the log contents, before 24h check is done.

Notes:

k-plan commented 7 years ago

@Fourdee

yes, I noticed this, but can't understand or debug, what your script is doing ... sorry for my disability. 😃

On the outdated PiHole installation, with Ramlog 2, graphing will works like a charm. Noticed as well, that here is no "preamble" in the log-file and it will always start at 00:00:xx :

root@Odroid-C2:~# cat /var/log/pihole.log
Mar 26 00:00:35 dnsmasq[32401]: query[AAAA] apple-mobile.query.yahooapis.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.3
Mar 26 00:00:35 dnsmasq[32401]: query[A] apple-mobile.query.yahooapis.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: query[AAAA] guzzoni.apple.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded guzzoni.apple.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: query[A] guzzoni.apple.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded guzzo
.....
Fourdee commented 7 years ago

@k-plan

On the outdated PiHole installation, with Ramlog 2, graphing will works like a charm.

Yep, checked code, looks fine. Shouldn't trigger a clear unless previous day.

Noticed as well, that here is no "preamble" in the log-file and it will always start at 00:00:xx

Thanks, i'll check 👍

k-plan commented 7 years ago

Yep, checked code, looks fine. Shouldn't trigger a clear unless previous day.

The first screenshot is taken from this device (NanoPi v146)

root@NanoPi-Neo:~# uptime
 21:04:54 up 7 days,  8:57,  1 user,  load average: 0,01, 0,04, 0,08

But it's the same, as well on RPi Zero W or OPi One (latest image version) updated to v147, running Pi-hole v2.13.2 with Admin-LTE v2.5.2

Fourdee commented 7 years ago

Tests with Ramlog mode 2:

Fourdee commented 7 years ago

@k-plan

Really strange, I cant seem to replicate this on RAMlog mode 1/2. PiHole logs are only cleared if day != today, or 5MB limit.

Just wondering if your PiHole log file is hitting 5MB> in size? in which case, it will be cleared to free up RAMlog (20MB tmpfs max): https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-logclear#L128-L136

Other thing could be locale related, causing a failed scrape. Running en_GB.UTF-8? Others should be fine, as long as en_GB.UTF-8 is still installed.

k-plan commented 7 years ago

@Fourdee

Just wondering if your PiHole log file is hitting 5MB> in size?

No, it's only my personal test device for my single computer. Rest of the hole network use the old PiHole installation on a other device.

root@NanoPi-Neo:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 102K Mär 27 16:13 /var/log/pihole.log

Other thing could be locale related, causing a failed scrape.

root@NanoPi-Neo:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX
root@NanoPi-Neo:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

But anyway ... shall we start form the scratch? (hope it is the last time, I do this ...) Which device? NanoPi Neo (v.1), OPi One or RPi Zero (W)? Last image version? Update to last DietPi? RAM-Log 2? May I change locale to de_DE? Install by hand or full automated?

k-plan commented 7 years ago

@Fourdee

OPi One installation:

Software:

root@OPi-One:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX

root@OPi-One:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

So choose for uninstall and after this run install via dietpi-software again.

After automated reboot, PiHole works.

Setup Pihole: cp /etc/pihole/adlists.default /etc/pihole/adlists.list
nano /etc/pihole/adlists.list pihole -g

Tested and work:

root@OPi-One:~# cat /var/log/pihole.log
Mar 27 17:30:16 dnsmasq[1011]: started, version 2.72 cachesize 10000
Mar 27 17:30:16 dnsmasq[1011]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 17:30:16 dnsmasq[1011]: warning: ignoring resolv-file flag because no-resolv is set
Mar 27 17:30:16 dnsmasq[1011]: using nameserver 192.168.1.1#53
Mar 27 17:30:16 dnsmasq[1011]: using nameserver 192.168.1.55#53
Mar 27 17:30:16 dnsmasq[1011]: read /etc/hosts - 2 addresses
Mar 27 17:30:16 dnsmasq[1011]: read /etc/pihole/local.list - 4 addresses
Mar 27 17:30:36 dnsmasq[1011]: read /etc/pihole/gravity.list - 204894 addresses
Mar 27 17:31:09 dnsmasq[1011]: query[A] api.github.com from 127.0.0.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.55
Mar 27 17:31:09 dnsmasq[1011]: query[AAAA] api.github.com from 127.0.0.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.55
Mar 27 17:31:09 dnsmasq[1011]: reply api.github.com is 192.30.253.116
Mar 27 17:31:09 dnsmasq[1011]: reply api.github.com is 192.30.253.117
Mar 27 17:31:10 dnsmasq[1011]: query[A] api.github.com from 127.0.0.1
Mar 27 17:31:10 dnsmasq[1011]: cached api.github.com is 192.30.253.117
Mar 27 17:31:10 dnsmasq[1011]: cached api.github.com is 192.30.253.116
Mar 27 17:31:10 dnsmasq[1011]: query[AAAA] api.github.com from 127.0.0.1
....

Graphing in "Queries over time" start working after 10 min.

After 47 min:

170327-0001

root@OPi-One:~# cat /var/log/pihole.log
Mar 27 18:17:02 dnsmasq[5818]: started, version 2.72 cachesize 10000
Mar 27 18:17:02 dnsmasq[5818]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 18:17:02 dnsmasq[5818]: warning: ignoring resolv-file flag because no-resolv is set
Mar 27 18:17:02 dnsmasq[5818]: using nameserver 192.168.1.1#53
Mar 27 18:17:02 dnsmasq[5818]: using nameserver 192.168.1.55#53
Mar 27 18:17:02 dnsmasq[5818]: read /etc/hosts - 2 addresses
Mar 27 18:17:02 dnsmasq[5818]: read /etc/pihole/local.list - 4 addresses
Mar 27 18:17:10 dnsmasq[5818]: read /etc/pihole/gravity.list - 343244 addresses
Mar 27 18:18:01 dnsmasq[5818]: query[A] raw.githubusercontent.com from 192.168.1.50
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.1
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.55
Mar 27 18:18:01 dnsmasq[5818]: query[AAAA] raw.githubusercontent.com from 192.168.1.50
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.1
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.55
Mar 27 18:18:01 dnsmasq[5818]: reply raw.githubusercontent.com is <CNAME>
Mar 27 18:18:01 dnsmasq[5818]: reply github.map.fastly.net is 151.101.112.133
Mar 27 18:18:01 dnsmasq[5818]: reply raw.githubusercontent.com is <CNAME>
Mar 27 18:18:01 dnsmasq[5818]: reply github.map.fastly.net is NODATA-IPv6
Mar 27 18:18:30 dnsmasq[5818]: query[A] api.github.com from 192.168.1.102
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.1
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:18:30 dnsmasq[5818]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:18:30 dnsmasq[5818]: reply api.github.com is 192.30.253.116
Mar 27 18:18:30 dnsmasq[5818]: reply api.github.com is 192.30.253.117
Mar 27 18:19:34 dnsmasq[5818]: query[A] api.github.com from 192.168.1.102
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.1
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:19:34 dnsmasq[5818]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:19:34 dnsmasq[5818]: reply api.github.com is 192.30.253.117
Mar 27 18:19:34 dnsmasq[5818]: reply api.github.com is 192.30.253.116
....

👎

Now installation on RPi-Zero-W is running ....

k-plan commented 7 years ago

@Fourdee

RPi-Zero-W installation:

Software:

root@RPi-Zero-W:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX

root@RPi-Zero-W:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

After automated reboot, PiHole works.

Setup Pihole: cp /etc/pihole/adlists.default /etc/pihole/adlists.list
nano /etc/pihole/adlists.list pihole -g

170327-0003

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 18:20:15 dnsmasq[946]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 18:20:15 dnsmasq[946]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntra$
Mar 27 18:20:15 dnsmasq[946]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 18:20:15 dnsmasq[946]: Benutze Namensserver 192.168.1.55#53
Mar 27 18:20:15 dnsmasq[946]: Benutze Namensserver 192.168.1.66#53
Mar 27 18:20:15 dnsmasq[946]: /etc/hosts gelesen - 2 Adressen
Mar 27 18:20:15 dnsmasq[946]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 18:20:29 dnsmasq[946]: /etc/pihole/gravity.list gelesen - 204894 Adressen
Mar 27 18:28:27 dnsmasq[946]: query[PTR] lb._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.LAN to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.LAN to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] lb._dns-sd._udp.0.10.16.172.in-addr.arpa from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.0.10.16.172.in-addr.arpa to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.0.10.16.172.in-addr.arpa to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] b._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded b._dns-sd._udp.LAN to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded b._dns-sd._udp.LAN to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] db._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded db._dns-sd._udp.LAN to 192.168.1.55
.......

170327-0004

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 19:17:02 dnsmasq[4170]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 19:17:02 dnsmasq[4170]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 19:17:02 dnsmasq[4170]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 19:17:02 dnsmasq[4170]: Benutze Namensserver 192.168.1.55#53
Mar 27 19:17:02 dnsmasq[4170]: Benutze Namensserver 192.168.1.66#53
Mar 27 19:17:02 dnsmasq[4170]: /etc/hosts gelesen - 2 Adressen
Mar 27 19:17:02 dnsmasq[4170]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 19:17:24 dnsmasq[4170]: /etc/pihole/gravity.list gelesen - 343244 Adressen
Mar 27 19:17:29 dnsmasq[4170]: query[A] gitter.im from 192.168.1.102
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.55
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.66
Mar 27 19:17:29 dnsmasq[4170]: query[AAAA] gitter.im from 192.168.1.102
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.55
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.66
Mar 27 19:17:29 dnsmasq[4170]: reply gitter.im is 54.209.199.250
Mar 27 19:17:29 dnsmasq[4170]: reply gitter.im is 54.174.166.63
Mar 27 19:17:39 dnsmasq[4170]: query[A] github.com from 192.168.1.102
Mar 27 19:17:39 dnsmasq[4170]: forwarded github.com to 192.168.1.55
Mar 27 19:17:39 dnsmasq[4170]: query[AAAA] github.com from 192.168.1.102
Mar 27 19:17:39 dnsmasq[4170]: forwarded github.com to 192.168.1.55
Mar 27 19:17:39 dnsmasq[4170]: reply github.com is 192.30.253.112
Mar 27 19:17:39 dnsmasq[4170]: reply github.com is 192.30.253.113
Mar 27 19:17:41 dnsmasq[4170]: query[A] live.github.com from 192.168.1.102
Mar 27 19:17:41 dnsmasq[4170]: forwarded live.github.com to 192.168.1.55
Mar 27 19:17:41 dnsmasq[4170]: query[AAAA] live.github.com from 192.168.1.102
Mar 27 19:17:41 dnsmasq[4170]: forwarded live.github.com to 192.168.1.55
Mar 27 19:17:41 dnsmasq[4170]: reply live.github.com is 192.30.253.124
Mar 27 19:17:41 dnsmasq[4170]: reply live.github.com is 192.30.253.125
Mar 27 19:18:41 dnsmasq[4170]: query[A] api.github.com from 192.168.1.102
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.55
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.66
Mar 27 19:18:41 dnsmasq[4170]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.55
Mar 27 19:18:41 dnsmasq[4170]: reply api.github.com is 192.30.253.117
Mar 27 19:18:41 dnsmasq[4170]: reply api.github.com is 192.30.253.116
......

Do you noticed "Every Hour at 17 Minutes " on ALL tested devices❓

170327-0005

update: 20:17:00

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 20:17:02 dnsmasq[8208]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 20:17:02 dnsmasq[8208]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 20:17:02 dnsmasq[8208]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 20:17:02 dnsmasq[8208]: Benutze Namensserver 192.168.1.55#53
Mar 27 20:17:02 dnsmasq[8208]: Benutze Namensserver 192.168.1.66#53
Mar 27 20:17:02 dnsmasq[8208]: /etc/hosts gelesen - 2 Adressen
Mar 27 20:17:02 dnsmasq[8208]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 20:17:25 dnsmasq[8208]: /etc/pihole/gravity.list gelesen - 343244 Adressen
Mar 27 20:21:24 dnsmasq[8208]: query[A] github.com from 192.168.1.102
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.55
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.66
Mar 27 20:21:24 dnsmasq[8208]: query[AAAA] github.com from 192.168.1.102
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.55
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.66
Mar 27 20:21:24 dnsmasq[8208]: reply github.com is 192.30.253.113
Mar 27 20:21:24 dnsmasq[8208]: reply github.com is 192.30.253.112
Mar 27 20:21:28 dnsmasq[8208]: query[A] avatars3.githubusercontent.com from 192.168.1.102
Mar 27 20:21:28 dnsmasq[8208]: forwarded avatars3.githubusercontent.com to 192.168.1.66
Mar 27 20:21:28 dnsmasq[8208]: query[AAAA] avatars3.githubusercontent.com from 192.168.1.102
Mar 27 20:21:28 dnsmasq[8208]: forwarded avatars3.githubusercontent.com to 192.168.1.66
Mar 27 20:21:28 dnsmasq[8208]: reply avatars3.githubusercontent.com is <CNAME>
Mar 27 20:21:28 dnsmasq[8208]: reply github.map.fastly.net is 151.101.112.133
Mar 27 20:21:28 dnsmasq[8208]: reply avatars3.githubusercontent.com is <CNAME>
Mar 27 20:21:28 dnsmasq[8208]: reply github.map.fastly.net is NODATA-IPv6
Mar 27 20:21:41 dnsmasq[8208]: query[A] avatars2.githubusercontent.com from 192.168.1.102
Mar 27 20:21:41 dnsmasq[8208]: forwarded avatars2.githubusercontent.com to 192.168.1.66
Mar 27 20:21:41 dnsmasq[8208]: query[AAAA] avatars2.githubusercontent.com from 192.168.1.102
Mar 27 20:21:41 dnsmasq[8208]: forwarded avatars2.githubusercontent.com to 192.168.1.66
Mar 27 20:21:41 dnsmasq[8208]: reply avatars2.githubusercontent.com is <CNAME>
Mar 27 20:21:41 dnsmasq[8208]: reply github.map.fastly.net is 151.101.112.133
Mar 27 20:21:41 dnsmasq[8208]: reply avatars2.githubusercontent.com is <CNAME>
Mar 27 20:21:41 dnsmasq[8208]: reply github.map.fastly.net is NODATA-IPv6
....

Yes, /etc/cron.hourly/ works ....

k-plan commented 7 years ago

for testing:

root@RPi-Zero-W:~# locale -a
C
C.UTF-8
en_GB.utf8
POSIX

root@RPi-Zero-W:~# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

170327-0006

Really strange, I cant seem to replicate this on RAMlog mode 1/2. PiHole logs are only cleared if day != today, or 5MB limit.

No, choose, add and switch locale=de_DE to default and you can see the magic ... 😃

BTW: Hope, with "Pi-hole FTL engine" all become better. 😃

Coming Soon To A Pi-hole Near You: FTL

Pi-hole FTL engine

Fourdee commented 7 years ago

@k-plan

No, choose, add and switch locale=de_DE to default and you can see the magic ...

Thanks, i'll check 👍

FTL

Looks like a change from php to C: https://github.com/pi-hole/FTL/blob/master/main.c should make for much reduced query times, espically with use of multithreading: https://github.com/pi-hole/FTL/blob/master/main.c#L43-L49

If IO on log file has any impact on query time, it "should" run faster on DietPi (RAMlog) 😉

Fourdee commented 7 years ago

@k-plan

https://github.com/pi-hole/FTL#pi-hole-ftl

This project is copyright under the latest version of the EUPL.

Yep, and Sonic is copyright of SEGA lmao

k-plan commented 7 years ago

Only change default locale=en_GB didn't help. Must remove all other language support.

If IO on log file has any impact on query time, it "should" run faster on DietPi (RAMlog) 😉

Flat file support will decease. Running database in RAMlog? 😉

FTL will help remediate the immediate need for a faster interface. The second component on the roadmap would be using a database instead of trying to query the entire text of the log file in it’s raw format.

Fourdee commented 7 years ago

@k-plan

Got it:

This will cause scrape to miss those entries, removing them.

I'll try a fix.

Fourdee commented 7 years ago

@k-plan

This commit should do it: https://github.com/Fourdee/DietPi/commit/3a6c6a3b1cf5fff8d69103a9fd78507b43b80419

k-plan commented 7 years ago

@Fourdee

Test is running on one device. We have to wait a little bit and see what's happens.

Thanks.

k-plan commented 7 years ago

Testing:

170330-0001

root@OPi-One:~# cat /DietPi/dietpi/dietpi-logclear | grep 'local month_day_today'
                                        ## local month_day_today=$(LANG=en_GB.UTF-8 date +'%b %e')
                                        local month_day_today=$(LC_ALL=en_GB.UTF-8 date +'%b %e')

root@OPi-One:~# date
Do 30. Mär 13:00:27 CEST 2017

root@OPi-One:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30

root@OPi-One:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
Mär 30

root@OPi-One:~# cat /var/log/pihole.log
Mar 30 10:58:42 dnsmasq[1061]: started, version 2.72 cachesize 10000
Mar 30 10:58:42 dnsmasq[1061]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 30 10:58:42 dnsmasq[1061]: warning: ignoring resolv-file flag because no-resolv is set
Mar 30 10:58:42 dnsmasq[1061]: using nameserver 192.168.1.56#53
Mar 30 10:58:42 dnsmasq[1061]: read /etc/hosts - 2 addresses
Mar 30 10:58:42 dnsmasq[1061]: read /etc/pihole/local.list - 4 addresses
Mar 30 10:59:03 dnsmasq[1061]: read /etc/pihole/gravity.list - 343242 addresses
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] lb._dns-sd._udp.LAN from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: forwarded lb._dns-sd._udp.LAN to 192.168.1.56
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] lb._dns-sd._udp.0.1.168.192.in-addr.arpa from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: forwarded lb._dns-sd._udp.0.1.168.192.in-addr.arpa to 192.168.1.56
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] 102.1.168.192.in-addr.arpa from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: config 192.168.1.102 is NXDOMAIN
....
root@OPi-One:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 101K Mär 30 13:06 /var/log/pihole.log

170330-0003

root@NanoPi-Neo:~# cat /DietPi/dietpi/.version
148

root@NanoPi-Neo:~# date
Do 30. Mär 13:07:46 CEST 2017

root@NanoPi-Neo:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30

root@NanoPi-Neo:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
Mär 30
root@NanoPi-Neo:~# cat /DietPi/dietpi/dietpi-logclear | grep 'local month_day'
                                        local month_day_today=$(LC_ALL=en_GB.UTF-8 date +'%b %e')

Mar 30 10:35:35 dnsmasq[1075]: gestartet, Version 2.72, Cachegröße 10000
Mar 30 10:35:35 dnsmasq[1075]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 30 10:35:35 dnsmasq[1075]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 30 10:35:35 dnsmasq[1075]: Benutze Namensserver 192.168.1.1#53
Mar 30 10:35:35 dnsmasq[1075]: Benutze Namensserver 192.168.1.55#53
Mar 30 10:35:35 dnsmasq[1075]: /etc/hosts gelesen - 6 Adressen
Mar 30 10:35:35 dnsmasq[1075]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 30 10:36:01 dnsmasq[1075]: /etc/pihole/gravity.list gelesen - 347304 Adressen
Mar 30 10:36:01 dnsmasq[1075]: query[A] github.com from 192.168.1.102
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.1
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.55
Mar 30 10:36:01 dnsmasq[1075]: query[AAAA] github.com from 192.168.1.102
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.1
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.55
Mar 30 10:36:01 dnsmasq[1075]: reply github.com is 192.30.253.113
Mar 30 10:36:01 dnsmasq[1075]: reply github.com is 192.30.253.112
....
root@NanoPi-Neo:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 131K Mär 30 13:09 /var/log/pihole.log

170330-0004

Different language in logfile content depends on device (NanoPi Neo vs. OrangePi One), but it seem to be equal.

Fourdee commented 7 years ago

@k-plan

Thanks, looks to be working with the change?

I believe you need to remove the ;, else, doesn't get applied to next command

LC_ALL=en_GB.UTF-8 date +'%b %e'

I left my C2 running today with the code change, de_DE default locale, seems to be working, logs from 01:00 onwards: image

Think we can mark this one as closed?

k-plan commented 7 years ago

@Fourdee

I believe you need to remove the ;, else, doesn't get applied to next command

Only copy your command from here 😃

But yes, you're right:

root@RPi-Zero-W:~# date
2017年 03月 30日 星期四 14:22:28 CEST

root@RPi-Zero-W:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
3月 30

root@RPi-Zero-W:~# LC_ALL=en_GB.UTF-8 date +'%b %e'
Mar 30

root@RPi-Zero-W:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30
root@RPi-Zero-W:~# locale -a -v

locale: zh_CN.utf8      archive: /usr/lib/locale/locale-archive
-------------------------------------------------------------------------------
    title | Chinese locale for Peoples Republic of China
    email | bug-glibc-locales@gnu.org
 language | Chinese
territory | P.R. of China
 revision | 0.1
     date | 2000-07-25
  codeset | UTF-8

locale: en_GB.utf8      archive: /usr/lib/locale/locale-archive
-------------------------------------------------------------------------------
    title | English locale for Britain
   source | RAP
  address | Sankt Jørgens Alle 8, DK-1615 København V, Danmark
  contact | Keld Simonsen
    email | bug-glibc-locales@gnu.org
 language | English
territory | Great Britain
 revision | 1.0
     date | 2000-06-28
  codeset | UTF-8

😄 ( ... and British form a Dane - nice!) :rofl:

Thanks, looks to be working with the change?

yes, I think so. It seem to be working on my Chinese locale RPi-Zero_W now.

170330-0005

root@RPi-Zero-W:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 81K 3月  30 14:39 /var/log/pihole.log
root@RPi-Zero-W:~# apt-get upgrade
正在读取软件包列表... 完成
正在分析软件包的依赖关系树
正在读取状态信息... 完成
正在对升级进行计算... 完成
下列软件包的版本将保持不变：
  libraspberrypi-bin libraspberrypi0 raspberrypi-bootloader raspberrypi-kernel
升级了 0 个软件包，新安装了 0 个软件包，要卸载 0 个软件包，有 4 个软件包未被升级。

:rofl:

Think we can mark this one as closed?

Yes, if it will run well on your C2 too.

Failure with locales in scripting, just as difficult every time. Sorry, better to remove language option in DietPi ... 😺

k-plan commented 7 years ago

@Fourdee

Testing results:

170330-0006

170330-0007

170330-0008

170330-0009

Fourdee commented 7 years ago

@k-plan

Only copy your command from here 😃

Yep, my bad 👍