jfb-pihole
commented
3 years ago Closed nikolaybotev closed 3 years ago
jfb-pihole
commented
3 years ago 
dschaper
commented
3 years ago Can you post the logs from when Pi-hole "crashes under it's own weight"? The log should show any errors.
Though 2500 queries in 10 minutes isn't any kind of load at all.
dschaper
commented
3 years ago Ah, can you confirm this line is missing the port at the end of 127.0.0.1?
-rw-r--r-- 1 root root 1414 Dec 1 20:15 /etc/dnsmasq.d/01-pihole.conf
addn-hosts=/etc/pihole/local.list
addn-hosts=/etc/pihole/custom.list
localise-queries
no-resolv
cache-size=10000
log-queries
log-facility=/var/log/pihole.log
local-ttl=2
log-async
server=127.0.0.1
domain-needed
expand-hosts
bogus-priv
interface=br0
server=/use-application-dns.net/
nikolaybotev
commented
3 years ago Hi,
Apologies, I did not mean to rip on Pi-hole and its performance - pi-hole definitely handles those thousands of queries without problem, the pi does not even get to any tangible CPU utilization.
What I observed was that for some reason after a while my internet connection seems to go down.
The actual problem is that after upgrade, the pihole-FTL process seems to start making lots of reverse DNS lookup queries (against itself), which I cannot explain in any sensible way.
I looked through the discourse thread, thanks for the references. It does not appear to be local client hostname lookups that the Pihole does once an hour. In my case the pihole-FTL is performing reverse DNS lookups for public IP addresses. It's also all IPv4 addresses, and NOT IPv6 addresses. Also, the pihole-FTL process is performing those lookups continuously, and it reaches more than 10,000 unique public IPs very quickly, and it fills up the DNS cache, which is at its default setting of 10,000 entries, and the UI starts flashing red on the cache evictions count to bring my attention to the fact that the cache is full and the pihole-FTL is having to evict entries from its cache early.
All of this seems very strange and scary, because I cannot trace the origin of those IP addresses and the queries!
As I shared the output of lsof it stops at the pihole-FTL process itself. So pihole-FTL is making those queries itself, and I have no idea where it is getting the list of those IPv4 addresses to lookup, and why it is even looking them up in the first place.
I have a home internet deployment with regular traffic to the usual sites, I have not had any unusual internet activity yesterday since the upgrade, and so it is all very strange.
Can you post the logs from when Pi-hole "crashes under it's own weight"? The log should show any errors.
Though 2500 queries in 10 minutes isn't any kind of load at all.
nikolaybotev
commented
3 years ago I do not have any info to share on why my internet stop working after awhile, and I would like to set that aside here.
I would like to focus your attention in this ticket on the continuous reverse DNS lookups against a large swath (1,000s maybe 10s of thousands) of distinct public IPv4 addresses, as shown in the screenshots of the graphs, and by the lsof output. This all happened after I upgraded pihole and I have no other changes to my environment that I can identify or report that I am aware of that happened yesterday.
I had problems gathering the debug log
DL6ER
commented
3 years ago seems to start making lots of reverse DNS lookup queries (against itself), which I cannot explain in any sensible way.
As @dschaper mentioned, your config contains:
server=127.0.0.1
which means Pi-hole is configured to query itself. I cannot imagine a scenario where this would be correct.
nikolaybotev
commented
3 years ago Some more details of my experience I can share here:
pihole-FTL never crashed on me, sorry for the confusion.
I did observe other strange behavior though, that I will share just for background info. Please note, I still feel the primary thing I would like to focus on in this ticket is the flood of reverse DNS lookups by the pihole-FTL process itself.
That said, for the record, here is what I experienced:
When I was submitting this ticket, I could not get pihole to upload the debug log. The debug log is pasted here inline. You can see that pihole reports failing to upload the log to your servers. I generated the log at least 5 or more times, with pi reboots and pihole restarts in between.
After I would generate the debug log, also, my internet connection would go down (remember my pi machine running pihole is also setup as my internet gateway / router).
After that I would try to restart the networking service, and it would fail to restart.
I would restart the pihole-FTL service and sometimes that would restore internet connectivity.
However, at some point last night while working on this my internet connection would start failing quite quickly after reboot of the pi.
At the end I was able to submit this ticket and I went to bed. Today I woke up to my internet connection down, and the raspberry pi was frozen to where I could not even ssh into it. It is running headless so I did not have other means to try to get into the machine. I just unplugged it and have switched back to my regular router for internet connectivity and put the pihole aside.
As I have to get back to my day, this will be the last info I can provide for today, and might not be able to get back to debugging / looking into this until tonight or a day or two later.
nikolaybotev
commented
3 years ago seems to start making lots of reverse DNS lookup queries (against itself), which I cannot explain in any sensible way.
As @dschaper mentioned, your config contains:
server=127.0.0.1which means Pi-hole is configured to query itself. I cannot imagine a scenario where this would be correct.
That is strange, and I will definitely look into it, however, I believe this should be 127.0.0.1#5353 as I have pihole configured to use a local instance of unbound recursive DNS server as its upstream DNS server as per the instructions here https://docs.pi-hole.net/guides/unbound/
nikolaybotev
commented
3 years ago seems to start making lots of reverse DNS lookup queries (against itself), which I cannot explain in any sensible way.
As @dschaper mentioned, your config contains:
server=127.0.0.1which means Pi-hole is configured to query itself. I cannot imagine a scenario where this would be correct.
For some reason the #5335 part got stripped from the debug log.
Here are the contents of my config files:
pi@Grogu ~
❯ cat /etc/pihole/pihole-FTL.conf
PRIVACYLEVEL=0
pi@Grogu ~
❯ cat /etc/pihole/setupVars.conf
WEBPASSWORD=91871995630c48de1153b8424dfc28b43fb3a560f744a4a0994fb20a8c5f73c2
DHCP_ACTIVE=true
DHCP_START=192.168.2.100
DHCP_END=192.168.2.251
DHCP_ROUTER=192.168.2.1
DHCP_LEASETIME=24
PIHOLE_DOMAIN=lan
DHCP_IPv6=true
DHCP_rapid_commit=true
BLOCKING_ENABLED=true
DNSMASQ_LISTENING=single
DNS_FQDN_REQUIRED=true
DNS_BOGUS_PRIV=true
DNSSEC=false
REV_SERVER=false
PIHOLE_INTERFACE=br0
IPV4_ADDRESS=192.168.2.1/24
IPV6_ADDRESS=
PIHOLE_DNS_1=127.0.0.1#5335
PIHOLE_DNS_2=
QUERY_LOGGING=true
INSTALL_WEB_SERVER=true
INSTALL_WEB_INTERFACE=true
LIGHTTPD_ENABLED=true
CACHE_SIZE=10000
pi@Grogu ~
❯ cat /etc/dnsmasq.d/01-pihole.conf
# Pi-hole: A black hole for Internet advertisements
# (c) 2017 Pi-hole, LLC (https://pi-hole.net)
# Network-wide ad blocking via your own hardware.
#
# Dnsmasq config for Pi-hole's FTLDNS
#
# This file is copyright under the latest version of the EUPL.
# Please see LICENSE file for your rights under this license.
###############################################################################
# FILE AUTOMATICALLY POPULATED BY PI-HOLE INSTALL/UPDATE PROCEDURE. #
# ANY CHANGES MADE TO THIS FILE AFTER INSTALL WILL BE LOST ON THE NEXT UPDATE #
# #
# IF YOU WISH TO CHANGE THE UPSTREAM SERVERS, CHANGE THEM IN: #
# /etc/pihole/setupVars.conf #
# #
# ANY OTHER CHANGES SHOULD BE MADE IN A SEPARATE CONFIG FILE #
# WITHIN /etc/dnsmasq.d/yourname.conf #
###############################################################################
addn-hosts=/etc/pihole/local.list
addn-hosts=/etc/pihole/custom.list
localise-queries
no-resolv
cache-size=10000
log-queries
log-facility=/var/log/pihole.log
local-ttl=2
log-async
server=127.0.0.1#5335
domain-needed
expand-hosts
bogus-priv
interface=br0
server=/use-application-dns.net/And some other stats that might be useful:
pi@Grogu ~
❯ ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.1 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::dea6:32ff:fe8c:73a3 prefixlen 64 scopeid 0x20<link>
ether dc:a6:32:8c:73:a3 txqueuelen 1000 (Ethernet)
RX packets 4846 bytes 629304 (614.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3483 bytes 1112588 (1.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether dc:a6:32:8c:73:a3 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 4417 bytes 307568 (300.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4417 bytes 307568 (300.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1500
inet 10.53.233.10 netmask 255.255.255.255 destination 10.64.64.64
ppp txqueuelen 3 (Point-to-Point Protocol)
RX packets 933 bytes 390454 (381.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1336 bytes 138845 (135.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether dc:a6:32:8c:73:a4 txqueuelen 1000 (Ethernet)
RX packets 4846 bytes 629304 (614.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3485 bytes 1197080 (1.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wwan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::b02c:8bff:fe3d:6729 prefixlen 64 scopeid 0x20<link>
ether b2:2c:8b:3d:67:29 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 29 bytes 4831 (4.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
pi@Grogu ~
❯ ip route
default dev ppp0 scope link
10.64.64.64 dev ppp0 proto kernel scope link src 10.53.233.10
192.168.2.0/24 dev br0 proto kernel scope link src 192.168.2.1
pi@Grogu ~
❯ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br0 state DOWN group default qlen 1000
link/ether dc:a6:32:8c:73:a3 brd ff:ff:ff:ff:ff:ff
3: wwan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether b2:2c:8b:3d:67:29 brd ff:ff:ff:ff:ff:ff
inet6 fe80::b02c:8bff:fe3d:6729/64 scope link
valid_lft forever preferred_lft forever
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether dc:a6:32:8c:73:a4 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether dc:a6:32:8c:73:a3 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.1/24 brd 192.168.2.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::dea6:32ff:fe8c:73a3/64 scope link
valid_lft forever preferred_lft forever
6: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 3
link/ppp
inet 10.53.233.10 peer 10.64.64.64/32 scope global ppp0
valid_lft forever preferred_lft forever
pi@Grogu ~
❯ sudo service pihole-FTL status
● pihole-FTL.service - LSB: pihole-FTL daemon
Loaded: loaded (/etc/init.d/pihole-FTL; generated)
Active: active (exited) since Wed 2020-12-02 11:45:16 EST; 1h 52min ago
Docs: man:systemd-sysv-generator(8)
Process: 2467 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS)
Dec 02 11:44:52 Grogu systemd[1]: Starting LSB: pihole-FTL daemon...
Dec 02 11:44:52 Grogu pihole-FTL[2467]: Not running
Dec 02 11:44:52 Grogu su[2485]: (to pihole) root on none
Dec 02 11:44:52 Grogu su[2485]: pam_unix(su:session): session opened for user pihole by (uid=0)
Dec 02 11:45:16 Grogu pihole-FTL[2467]: FTL started!
Dec 02 11:45:16 Grogu systemd[1]: Started LSB: pihole-FTL daemon.
pi@Grogu ~
❯ sudo service unbound status
● unbound.service - Unbound DNS server
Loaded: loaded (/lib/systemd/system/unbound.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-12-02 11:37:45 EST; 2h 0min ago
Docs: man:unbound(8)
Process: 737 ExecStartPre=/usr/lib/unbound/package-helper chroot_setup (code=exited, status=0/SUCCESS)
Process: 782 ExecStartPre=/usr/lib/unbound/package-helper root_trust_anchor_update (code=exited, status=0/SUCCESS)
Main PID: 791 (unbound)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/unbound.service
└─791 /usr/sbin/unbound -d
Dec 02 11:37:45 Grogu systemd[1]: Starting Unbound DNS server...
Dec 02 11:37:45 Grogu package-helper[782]: /var/lib/unbound/root.key has content
Dec 02 11:37:45 Grogu package-helper[782]: fail: the anchor is NOT ok and could not be fixed
Dec 02 11:37:45 Grogu systemd[1]: Started Unbound DNS server.
pi@Grogu ~
❯ sudo netstat -ltunp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 809/lighttpd
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 2699/pihole-FTL
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 776/sshd
tcp 0 0 127.0.0.1:5335 0.0.0.0:* LISTEN 791/unbound
tcp 0 0 127.0.0.1:5335 0.0.0.0:* LISTEN 791/unbound
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN 863/smbd
tcp 0 0 127.0.0.1:4711 0.0.0.0:* LISTEN 2699/pihole-FTL
tcp 0 0 0.0.0.0:8200 0.0.0.0:* LISTEN 868/minidlnad
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 863/smbd
tcp6 0 0 :::80 :::* LISTEN 809/lighttpd
tcp6 0 0 :::53 :::* LISTEN 2699/pihole-FTL
tcp6 0 0 :::22 :::* LISTEN 776/sshd
tcp6 0 0 :::445 :::* LISTEN 863/smbd
tcp6 0 0 ::1:4711 :::* LISTEN 2699/pihole-FTL
tcp6 0 0 :::139 :::* LISTEN 863/smbd
udp 0 0 0.0.0.0:53 0.0.0.0:* 2699/pihole-FTL
udp 0 0 0.0.0.0:67 0.0.0.0:* 2699/pihole-FTL
udp 0 0 192.168.2.255:137 0.0.0.0:* 740/nmbd
udp 0 0 192.168.2.1:137 0.0.0.0:* 740/nmbd
udp 0 0 0.0.0.0:137 0.0.0.0:* 740/nmbd
udp 0 0 192.168.2.255:138 0.0.0.0:* 740/nmbd
udp 0 0 192.168.2.1:138 0.0.0.0:* 740/nmbd
udp 0 0 0.0.0.0:138 0.0.0.0:* 740/nmbd
udp 0 0 127.0.0.1:5335 0.0.0.0:* 791/unbound
udp 0 0 127.0.0.1:5335 0.0.0.0:* 791/unbound
udp 0 0 0.0.0.0:5353 0.0.0.0:* 373/avahi-daemon: r
udp 0 0 10.53.233.10:58191 0.0.0.0:* 868/minidlnad
udp 0 0 239.255.255.250:1900 0.0.0.0:* 868/minidlnad
udp 0 0 0.0.0.0:48495 0.0.0.0:* 373/avahi-daemon: r
udp 0 0 192.168.2.1:49606 0.0.0.0:* 868/minidlnad
udp 0 0 0.0.0.0:55785 0.0.0.0:* 791/unbound
udp6 0 0 :::547 :::* 2699/pihole-FTL
udp6 0 0 :::53 :::* 2699/pihole-FTL
udp6 0 0 :::5353 :::* 373/avahi-daemon: r
udp6 0 0 :::53086 :::* 373/avahi-daemon: r
The DNS settings:
nikolaybotev
commented
3 years ago The DHCP settings (you can see I only have 11 clients!):
nikolaybotev
commented
3 years ago System settings (you can see there are only 179 cache insertions). I am going to keep the pi running, and see if it starts off on the IPv4 reverse DNS lookup spree again (usually it would take 15-30 minutes after reboot for it to start the flood, based on my experience yesterday).
nikolaybotev
commented
3 years ago Here is another sample from the query log from yesterday of IPs that pihole-FTL was looking up!
I tried one of them manually and got this:
❯ dig 185.120.22.23
; <<>> DiG 9.11.5-P4-5.1+deb10u2-Raspbian <<>> 185.120.22.23
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 17018
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;185.120.22.23. IN A
;; AUTHORITY SECTION:
. 3014 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2020120201 1800 900 604800 86400
;; Query time: 71 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Dec 02 13:46:13 EST 2020
;; MSG SIZE rcvd: 117
Client is always localhost, and when I use lsof it is always the pihole-FTL process itself!
I do see queries for my local clients on page 1:
Maybe because of my custom setup somehow pihole-FTL gets confused and thinks that all these public IPs (that it observes somehow?) are clients?
When they are really not? I can clearly see my 11 clients on the DHCP settings page, and pihole-FTL is setup to listen only on the local LAN br0 interface.
dschaper
commented
3 years ago Can you run
echo ">stats" | nc 127.0.0.1 4711
nikolaybotev
commented
3 years ago I do have a firewall blocking all but SSH on the internet interface:
❯ sudo iptables -nvL
Chain INPUT (policy ACCEPT 24130 packets, 2375K bytes)
pkts bytes target prot opt in out source destination
3381 234K f2b-sshd-aggressive tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 22
0 0 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
0 0 REJECT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 14876 packets, 8222K bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 REJECT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 20741 packets, 12M bytes)
pkts bytes target prot opt in out source destination
Chain f2b-sshd-aggressive (1 references)I see that FTL might be able to bind to a specific interface (https://github.com/pi-hole/FTL/search?q=SO_BINDTODEVICE) but not sure if that is enabled in the build, and if that option is set in dnsmasq config, and how to check the actual open listening socket (looked into netstat and ss and lsof and see no options to show the interface a socket was bound to via SO_BINDTODEVICE). https://unix.stackexchange.com/questions/54975/how-to-check-that-a-daemon-is-listening-on-what-interface...
nikolaybotev
commented
3 years ago Can you run
echo ">stats" | nc 127.0.0.1 4711
❯ echo ">stats" | nc 127.0.0.1 4711
domains_being_blocked 87641
dns_queries_today 82034
ads_blocked_today 1198
ads_percentage_today 1.460370
unique_domains 16740
queries_forwarded 63825
queries_cached 11561
clients_ever_seen 13
unique_clients 13
dns_queries_all_types 82034
reply_NODATA 0
reply_NXDOMAIN 47
reply_CNAME 48
reply_IP 144
privacy_level 0
status enabled
---EOM---.... and the flood has begun!
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
pihole-FTL 2699 pihole 23u IPv4 99673 0t0 UDP 127.0.0.1:56896->127.0.0.1:domain
pihole-FTL 2699 2700 telnet-IPv4 pihole 23u IPv4 99673 0t0 UDP 127.0.0.1:56896->127.0.0.1:domain
pihole-FTL 2699 2701 telnet-IPv6 pihole 23u IPv4 99673 0t0 UDP 127.0.0.1:56896->127.0.0.1:domain
pihole-FTL 2699 2703 database pihole 23u IPv4 99673 0t0 UDP 127.0.0.1:56896->127.0.0.1:domain
pihole-FTL 2699 2704 housekeeper pihole 23u IPv4 99673 0t0 UDP 127.0.0.1:56896->127.0.0.1:domain
pihole-FTL 2699 2705 DNS\x20client pihole 23u IPv4 99673 0t0 UDP 127.0.0.1:56896->127.0.0.1:domain
Dec 2 14:01:47 dnsmasq[2699]: forwarded 33.139.120.45.in-addr.arpa to 127.0.0.1
Dec 2 14:01:47 dnsmasq[2699]: reply 45.120.139.33 is NXDOMAIN
Dec 2 14:01:47 dnsmasq[2699]: query[PTR] 73.44.244.143.in-addr.arpa from 127.0.0.1
Dec 2 14:01:47 dnsmasq[2699]: forwarded 73.44.244.143.in-addr.arpa to 127.0.0.1
Dec 2 14:01:47 dnsmasq[2699]: reply 143.244.44.73 is NXDOMAIN
Dec 2 14:01:47 dnsmasq[2699]: query[PTR] 100.238.105.202.in-addr.arpa from 127.0.0.1
Dec 2 14:01:47 dnsmasq[2699]: forwarded 100.238.105.202.in-addr.arpa to 127.0.0.1
Dec 2 14:01:48 dnsmasq[2699]: reply 202.105.238.100 is NXDOMAIN
Dec 2 14:01:48 dnsmasq[2699]: query[PTR] 84.37.244.143.in-addr.arpa from 127.0.0.1
Dec 2 14:01:48 dnsmasq[2699]: forwarded 84.37.244.143.in-addr.arpa to 127.0.0.1
Dec 2 14:01:48 dnsmasq[2699]: reply 143.244.37.84 is NXDOMAIN
Dec 2 14:01:48 dnsmasq[2699]: query[PTR] 24.45.17.84.in-addr.arpa from 127.0.0.1
Dec 2 14:01:48 dnsmasq[2699]: forwarded 24.45.17.84.in-addr.arpa to 127.0.0.1
Dec 2 14:01:48 dnsmasq[2699]: reply 84.17.45.24 is unn-84-17-45-24.cdn77.com
Dec 2 14:01:48 dnsmasq[2699]: query[PTR] 242.104.107.86.in-addr.arpa from 127.0.0.1
Dec 2 14:01:48 dnsmasq[2699]: forwarded 242.104.107.86.in-addr.arpa to 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: reply 86.107.104.242 is NXDOMAIN
Dec 2 14:01:49 dnsmasq[2699]: query[PTR] 99.53.110.79.in-addr.arpa from 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: forwarded 99.53.110.79.in-addr.arpa to 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: reply 79.110.53.99 is NXDOMAIN
Dec 2 14:01:49 dnsmasq[2699]: query[PTR] 50.60.146.156.in-addr.arpa from 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: forwarded 50.60.146.156.in-addr.arpa to 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: reply 156.146.60.50 is unn-156-146-60-50.cdn77.com
Dec 2 14:01:49 dnsmasq[2699]: query[PTR] 5.131.29.154.in-addr.arpa from 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: forwarded 5.131.29.154.in-addr.arpa to 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: reply 154.29.131.5 is NXDOMAIN
Dec 2 14:01:49 dnsmasq[2699]: query[PTR] 126.75.67.172.in-addr.arpa from 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: forwarded 126.75.67.172.in-addr.arpa to 127.0.0.1
Dec 2 14:01:49 dnsmasq[2699]: reply 172.67.75.126 is NXDOMAINYou can see it is a steady stream of ~3 queries per second.
nikolaybotev
commented
3 years ago DNS cache insertions is at 850+ and steadily growing:
... as I am writing this to almost 1,000:
It will reach 10,000 soon and start evicting...
So these are all DISTINCT IP addresses ... Where is pihole getting those... I have no idea...
The most embarrassing prospect is that my pi has somehow been hacked and infected with some malware / virus ... :-S
nikolaybotev
commented
3 years ago The DHCP stuff... all looks good:
pi@Grogu ~
❯ cat /etc/dnsmasq.d/02-pihole-dhcp.conf
###############################################################################
# DHCP SERVER CONFIG FILE AUTOMATICALLY POPULATED BY PI-HOLE WEB INTERFACE. #
# ANY CHANGES MADE TO THIS FILE WILL BE LOST ON CHANGE #
###############################################################################
dhcp-authoritative
dhcp-range=192.168.2.100,192.168.2.251,24h
dhcp-option=option:router,192.168.2.1
dhcp-leasefile=/etc/pihole/dhcp.leases
#quiet-dhcp
domain=lan
local=/lan/
dhcp-rapid-commit
#quiet-dhcp6
#enable-ra
dhcp-option=option6:dns-server,[::]
dhcp-range=::100,::1ff,constructor:br0,ra-names,slaac,24h
ra-param=*,0,0
pi@Grogu ~
❯ cat /etc/pihole/dhcp.leases
1607013493 a4:83:e7:21:41:da 192.168.2.198 ---- 01:a4:83:e7:21:41:da
1606940231 52:71:2b:d8:36:5d 192.168.2.159 ---- 01:52:71:2b:d8:36:5d
1606959773 4a:fb:79:d4:33:6e 192.168.2.147 ---- 01:4a:fb:79:d4:33:6e
1606935729 3c:e1:a1:15:fc:86 192.168.2.191 ---- *
1607013529 90:70:65:6b:a4:9c 192.168.2.106----- ff:65:6b:a4:9c:00:01:00:01:23:9b:17:51:90:70:65:6b:a4:9c
1606960373 cc:d2:81:74:96:34 192.168.2.195 --- 01:cc:d2:81:74:96:34
1607013508 b8:27:eb:b7:f8:c4 192.168.2.203 ---- 01:b8:27:eb:b7:f8:c4
1606976302 24:4b:fe:2f:ac:b0 192.168.2.100 ---- 01:24:4b:fe:2f:ac:b0
1607013522 b8:27:eb:45:54:39 192.168.2.122 ---- 01:b8:27:eb:45:54:39
1606936327 50:14:79:12:5b:6d 192.168.2.115 ---- *
1606971296 ee:7c:c9:92:c9:2c 192.168.2.110 ---- 01:ee:7c:c9:92:c9:2c
duid 00:01:00:01:26:83:06:d4:dc:a6:32:8c:73:a3
(masked client host names with ----)
nikolaybotev
commented
3 years ago More config:
❯ ls /etc/dnsmasq.d
01-pihole.conf 02-pihole-dhcp.conf 04-pihole-static-dhcp.conf
pi@Grogu ~
❯ cat /etc/dnsmasq.d/04-pihole-static-dhcp.conf
dhcp-host=24:4B:FE:2F:AC:B0,192.168.2.100,----
stats again: queries have already gone up by 2,000 but unique domains still at ~16K - I imagine most of these 16K are from the flooding yesterday!
❯ echo ">stats" | nc 127.0.0.1 4711
domains_being_blocked 87641
dns_queries_today 84047
ads_blocked_today 1203
ads_percentage_today 1.431342
unique_domains 16740
queries_forwarded 65797
queries_cached 11597
clients_ever_seen 13
unique_clients 13
dns_queries_all_types 84047
reply_NODATA 0
reply_NXDOMAIN 913
reply_CNAME 48
reply_IP 195
privacy_level 0
status enabled
---EOM---DNS cache insertions matches up the 2K increase above:
DNS cache insertions: | 2664
-- | --
Bumped the cache size to 20,000 and restarted pihole-FTL will leave it running and see what happens.
Double checked that my pihole-FTL binary is not compromised / genuine:
pi@Grogu ~
❯ sha1sum pihole-FTL-armv7-linux-gnueabihf
974dacfe58b34ea2c89be67271fa1998b81583ac pihole-FTL-armv7-linux-gnueabihf
pi@Grogu ~
❯ cat pihole-FTL-armv7-linux-gnueabihf.sha1
974dacfe58b34ea2c89be67271fa1998b81583ac pihole-FTL-armv7-linux-gnueabihf
pi@Grogu ~
❯ sha1sum /usr/bin/pihole-FTL
974dacfe58b34ea2c89be67271fa1998b81583ac /usr/bin/pihole-FTL
Bumped the cache size to 20,000
Why? You haven't seen any cache evictions.
So far everything here points to an application on the Pi-hole device querying for those PTR records. What other applications are running? Any kind of torrenting application or something that would be accessing remote nodes?
nikolaybotev
commented
3 years ago Bumped the cache size to 20,000
Why? You haven't seen any cache evictions.
So far everything here points to an application on the Pi-hole device querying for those PTR records. What other applications are running? Any kind of torrenting application or something that would be accessing remote nodes?
Yesterday I saw cache evictions. It only takes a few 10s of minutes (once flooding starts) to get to the cache size.
After I saw the total domains at 16,700, I figured let me keep this thing running with a larger cache size and when / if the flood starts again (after changing the setting and restarting pihole-FTL) then I can see if the total number of domains it queries goes beyond 16,700 (with a larger cache size).
nikolaybotev
commented
3 years ago I will set a cron job to track all the open files by pihole-FTL... in the hope of observing where pihole-FTL is getting all those IP addresses it is so eager to do reverse DNS lookup on.
dschaper
commented
3 years ago I can see if the total number of domains it queries goes beyond 16,700 (with a larger cache size).
And what do you think that information will reveal to you?
dschaper
commented
3 years ago I will set a cron job to track all the open files by pihole-FTL... in the hope of observing where pihole-FTL is getting all those IP addresses it is so eager to do reverse DNS lookup on.
That's not going to show anything and you're not answering my question. What other networking applications are you running on the same node?
nikolaybotev
commented
3 years ago So far everything here points to an application on the Pi-hole device querying for those PTR records. What other applications are running? Any kind of torrenting application or something that would be accessing remote nodes?
That's what I thought as well! However, lsof output which I shared from the very beginning here, clearly shows that pihole-FTL is the one querying for those records!
Unless I am misunderstanding the output of lsof, however, I tested my assumptions by running nc and I clearly see that nc which I know is the client shows up in the lsof output as I would expect:
❯ nc -u4 localhost 53❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
nc 9774 pi 3u IPv4 144345 0t0 UDP 127.0.0.1:56392->127.0.0.1:domain When the flood happens (as you can see in my earlier comments) the only local client printed by lsof is pihole-FTL itself!
I invite you to inspect the output of netstat which I shared in earlier comments and which lists all local networking applications running (basically unbound, samba with nmbd, and minidlna, none of which show up in the lsof output during the flood).
dschaper
commented
3 years ago clearly shows that pihole-FTL is the one querying for those records!
Sure it is, something is asking it for the information.
netstat -ltunp
That shows things in the listening state, which doesn't show client requests.
I see now that you are using this Raspberry Pi as a router, which in itself is a bad idea, it's just not made to handle that task. pihole-FTL is not going to randomly decide to start looking up PTR records for IPv4 addresses with no query coming in for that IPv4 address in the first place.
EDIT: Sorry, missed that you intentionally called nc
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain nc 9774 pi 3u IPv4 144345 0t0 UDP 127.0.0.1:56392->127.0.0.1:domain
Why is nc (netcat) asking Pi-hole for DNS information?
nikolaybotev
commented
3 years ago What other ways (other than tcp/udp port 53) are there for that something to ask pihole-FTL for the information?
nikolaybotev
commented
3 years ago I stopped all other network services, including unbound!
The flood kept going!
pi@Grogu ~
❯ echo ">stats" | nc 127.0.0.1 4711
domains_being_blocked 87641
dns_queries_today 70582
ads_blocked_today 1150
ads_percentage_today 1.629311
unique_domains 16666
queries_forwarded 55005
queries_cached 9622
clients_ever_seen 13
unique_clients 13
dns_queries_all_types 70582
reply_NODATA 3
reply_NXDOMAIN 1442
reply_CNAME 86
reply_IP 369
privacy_level 0
status enabled
---EOM---
^C
pi@Grogu ~
❯ echo ">stats" | nc 127.0.0.1 4711
domains_being_blocked 87641
dns_queries_today 70609
ads_blocked_today 1150
ads_percentage_today 1.628688
unique_domains 16666
queries_forwarded 55032
queries_cached 9622
clients_ever_seen 13
unique_clients 13
dns_queries_all_types 70609
reply_NODATA 3
reply_NXDOMAIN 1442
reply_CNAME 86
reply_IP 369
privacy_level 0
status enabled
---EOM---
Observe the queries are going up from 70582 to 70609. The only client showing up on lsof is still pihole-FTL!
pi@Grogu ~
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
pihole-FTL 5630 pihole 22u IPv4 188775 0t0 UDP 127.0.0.1:43062->127.0.0.1:domain
pihole-FTL 5630 5631 telnet-IPv4 pihole 22u IPv4 188775 0t0 UDP 127.0.0.1:43062->127.0.0.1:domain
pihole-FTL 5630 5632 telnet-IPv6 pihole 22u IPv4 188775 0t0 UDP 127.0.0.1:43062->127.0.0.1:domain
pihole-FTL 5630 5634 database pihole 22u IPv4 188775 0t0 UDP 127.0.0.1:43062->127.0.0.1:domain
pihole-FTL 5630 5635 housekeeper pihole 22u IPv4 188775 0t0 UDP 127.0.0.1:43062->127.0.0.1:domain
pihole-FTL 5630 5636 DNS\x20client pihole 22u IPv4 188775 0t0 UDP 127.0.0.1:43062->127.0.0.1:domain There is only sshd left and pihole-FTL in terms of services running that listen on the network:
❯ sudo netstat -ltunp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 5630/pihole-FTL
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 776/sshd
tcp 0 0 127.0.0.1:4711 0.0.0.0:* LISTEN 5630/pihole-FTL
tcp6 0 0 :::53 :::* LISTEN 5630/pihole-FTL
tcp6 0 0 :::22 :::* LISTEN 776/sshd
tcp6 0 0 ::1:4711 :::* LISTEN 5630/pihole-FTL
udp 0 0 0.0.0.0:53 0.0.0.0:* 5630/pihole-FTL
udp 0 0 0.0.0.0:67 0.0.0.0:* 5630/pihole-FTL
udp 0 0 0.0.0.0:34414 0.0.0.0:* 5630/pihole-FTL
udp 0 0 0.0.0.0:10979 0.0.0.0:* 5630/pihole-FTL
udp 0 0 0.0.0.0:25864 0.0.0.0:* 5630/pihole-FTL
udp 0 0 0.0.0.0:37132 0.0.0.0:* 5630/pihole-FTL
udp 0 0 0.0.0.0:31501 0.0.0.0:* 5630/pihole-FTL
udp 0 0 0.0.0.0:29087 0.0.0.0:* 5630/pihole-FTL
udp6 0 0 :::547 :::* 5630/pihole-FTL
udp6 0 0 :::53 :::* 5630/pihole-FTL
Unless something (malware?) is connecting to the telnet interface of pihole-FTL at 4711? I am going to look into that next.
All network services that were killed for this test:
755 sudo service smbd stop
756 sudo service nmbd stop
757 sudo service minidlna stop
764 sudo service avahi-daemon stop
765 sudo service lighttpd stop
766 sudo service dhcpcd stop
770 sudo service unbound stopThere do not appear to be any active connections to the pihole-FTL telnet interface at 4711:
❯ sudo lsof +c 15 -n |grep :4711
pihole-FTL 5630 pihole 14u IPv4 125810 0t0 TCP 127.0.0.1:4711 (LISTEN)
pihole-FTL 5630 pihole 16u IPv6 125813 0t0 TCP [::1]:4711 (LISTEN)
pihole-FTL 5630 5631 telnet-IPv4 pihole 14u IPv4 125810 0t0 TCP 127.0.0.1:4711 (LISTEN)
pihole-FTL 5630 5631 telnet-IPv4 pihole 16u IPv6 125813 0t0 TCP [::1]:4711 (LISTEN)
pihole-FTL 5630 5632 telnet-IPv6 pihole 14u IPv4 125810 0t0 TCP 127.0.0.1:4711 (LISTEN)
pihole-FTL 5630 5632 telnet-IPv6 pihole 16u IPv6 125813 0t0 TCP [::1]:4711 (LISTEN)
pihole-FTL 5630 5634 database pihole 14u IPv4 125810 0t0 TCP 127.0.0.1:4711 (LISTEN)
pihole-FTL 5630 5634 database pihole 16u IPv6 125813 0t0 TCP [::1]:4711 (LISTEN)
pihole-FTL 5630 5635 housekeeper pihole 14u IPv4 125810 0t0 TCP 127.0.0.1:4711 (LISTEN)
pihole-FTL 5630 5635 housekeeper pihole 16u IPv6 125813 0t0 TCP [::1]:4711 (LISTEN)
pihole-FTL 5630 5636 DNS\x20client pihole 14u IPv4 125810 0t0 TCP 127.0.0.1:4711 (LISTEN)
pihole-FTL 5630 5636 DNS\x20client pihole 16u IPv6 125813 0t0 TCP [::1]:4711 (LISTEN)Maybe they are coming and going too quickly for me to catch? Will try and block 4711 using iptables.
nikolaybotev
commented
3 years ago The only active ssh session is my own:
❯ sudo lsof +c 15 -n |grep :ssh
sshd 776 root 3u IPv4 18789 0t0 TCP *:ssh (LISTEN)
sshd 776 root 4u IPv6 18791 0t0 TCP *:ssh (LISTEN)
sshd 1352 root 3u IPv4 56101 0t0 TCP 192.168.2.1:ssh->192.168.2.198:59758 (ESTABLISHED)
sshd 1371 pi 3u IPv4 56101 0t0 TCP 192.168.2.1:ssh->192.168.2.198:59758 (ESTABLISHED)
sshd 1882 root 3u IPv4 83497 0t0 TCP 192.168.2.1:ssh->192.168.2.198:59770 (ESTABLISHED)
sshd 1888 pi 3u IPv4 83497 0t0 TCP 192.168.2.1:ssh->192.168.2.198:59770 (ESTABLISHED)The web UI stopped working, but the flood storm apparently continues based on lsof output, with continously fresh connections (see the 3 different client port numbers in the 3 separate runs of lsof) between pihole-FTL and itself to query DNS, after successfully blocking all connections on port 4711 from anywhere over udp and tcp and ipv4 and ipv6:
pi@Grogu ~
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
pihole-FTL 5630 pihole 22u IPv4 220075 0t0 UDP 127.0.0.1:53553->127.0.0.1:domain
pihole-FTL 5630 5631 telnet-IPv4 pihole 22u IPv4 220075 0t0 UDP 127.0.0.1:53553->127.0.0.1:domain
pihole-FTL 5630 5632 telnet-IPv6 pihole 22u IPv4 220075 0t0 UDP 127.0.0.1:53553->127.0.0.1:domain
pihole-FTL 5630 5634 database pihole 22u IPv4 220075 0t0 UDP 127.0.0.1:53553->127.0.0.1:domain
pihole-FTL 5630 5635 housekeeper pihole 22u IPv4 220075 0t0 UDP 127.0.0.1:53553->127.0.0.1:domain
pihole-FTL 5630 5636 DNS\x20client pihole 22u IPv4 220075 0t0 UDP 127.0.0.1:53553->127.0.0.1:domain
pi@Grogu ~
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
pihole-FTL 5630 pihole 22u IPv4 222390 0t0 UDP 127.0.0.1:38006->127.0.0.1:domain
pihole-FTL 5630 5631 telnet-IPv4 pihole 22u IPv4 222390 0t0 UDP 127.0.0.1:38006->127.0.0.1:domain
pihole-FTL 5630 5632 telnet-IPv6 pihole 22u IPv4 222390 0t0 UDP 127.0.0.1:38006->127.0.0.1:domain
pihole-FTL 5630 5634 database pihole 22u IPv4 222390 0t0 UDP 127.0.0.1:38006->127.0.0.1:domain
pihole-FTL 5630 5635 housekeeper pihole 22u IPv4 222390 0t0 UDP 127.0.0.1:38006->127.0.0.1:domain
pihole-FTL 5630 5636 DNS\x20client pihole 22u IPv4 222390 0t0 UDP 127.0.0.1:38006->127.0.0.1:domain
pi@Grogu ~
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
pihole-FTL 5630 pihole 22u IPv4 222483 0t0 UDP 127.0.0.1:37421->127.0.0.1:domain
pihole-FTL 5630 5631 telnet-IPv4 pihole 22u IPv4 222483 0t0 UDP 127.0.0.1:37421->127.0.0.1:domain
pihole-FTL 5630 5632 telnet-IPv6 pihole 22u IPv4 222483 0t0 UDP 127.0.0.1:37421->127.0.0.1:domain
pihole-FTL 5630 5634 database pihole 22u IPv4 222483 0t0 UDP 127.0.0.1:37421->127.0.0.1:domain
pihole-FTL 5630 5635 housekeeper pihole 22u IPv4 222483 0t0 UDP 127.0.0.1:37421->127.0.0.1:domain
pihole-FTL 5630 5636 DNS\x20client pihole 22u IPv4 222483 0t0 UDP 127.0.0.1:37421->127.0.0.1:domain
pi@Grogu ~
❯ echo ">stats" | nc 127.0.0.1 4711
pi@Grogu ~
❯ sudo iptables -nvL INPUT
Chain INPUT (policy ACCEPT 130K packets, 14M bytes)
pkts bytes target prot opt in out source destination
23342 1566K f2b-sshd-aggressive tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 22
22718 17M ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
36 1860 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
5685 510K REJECT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
0 0 REJECT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:4711 reject-with icmp-port-unreachable
24 1440 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4711 reject-with icmp-port-unreachable
pi@Grogu ~
❯ sudo ip6tables -nvL INPUT
Chain INPUT (policy ACCEPT 219 packets, 58910 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all eth1 * ::/0 ::/0 state RELATED,ESTABLISHED
0 0 ACCEPT tcp eth1 * ::/0 ::/0 state NEW tcp dpt:22
112 22736 REJECT all eth1 * ::/0 ::/0 reject-with icmp6-port-unreachable
0 0 REJECT tcp * * ::/0 ::/0 tcp dpt:4711 reject-with icmp6-port-unreachable
0 0 REJECT udp * * ::/0 ::/0 udp dpt:4711 reject-with icmp6-port-unreachable
pi@Grogu ~
❯ At this point I have exhausted all ideas I have for diagnosing this, beyond debugging pihole-FTL locally, which I might do at some point... or try out AdGuard? Who knows...
nikolaybotev
commented
3 years ago Here is one more thing I looked at to share: a tcpdump look at the traffic (all over lo by the way, as already established between pihole-FTL and itself!):
And a couple of other notes:
❯ sudo tcpdump -vv -n -s 1500 -i lo 'port 53'
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 1500 bytes
15:40:04.508767 IP (tos 0x0, ttl 64, id 36536, offset 0, flags [DF], proto UDP (17), length 124)
127.0.0.1.53 > 127.0.0.1.47707: [bad udp cksum 0xfe7b -> 0x3ff3!] 27660 q: PTR? 104.198.228.168.in-addr.arpa. 1/0/0 104.198.228.168.in-addr.arpa. PTR maxfibra-168-228-198-104.yune.com.br. (96)
15:40:04.509522 IP (tos 0x0, ttl 64, id 36537, offset 0, flags [DF], proto UDP (17), length 73)
127.0.0.1.34837 > 127.0.0.1.53: [bad udp cksum 0xfe48 -> 0x9c56!] 46945+ PTR? 150.239.238.41.in-addr.arpa. (45)
15:40:05.196028 IP (tos 0x0, ttl 64, id 36552, offset 0, flags [DF], proto UDP (17), length 117)
127.0.0.1.53 > 127.0.0.1.34837: [bad udp cksum 0xfe74 -> 0x3996!] 46945 q: PTR? 150.239.238.41.in-addr.arpa. 1/0/0 150.239.238.41.in-addr.arpa. PTR host-41.238.239.150.tedata.net. (89)
15:40:05.196770 IP (tos 0x0, ttl 64, id 36553, offset 0, flags [DF], proto UDP (17), length 72)
127.0.0.1.52426 > 127.0.0.1.53: [bad udp cksum 0xfe47 -> 0x6c29!] 2469+ PTR? 187.171.6.107.in-addr.arpa. (44)
15:40:05.861526 IP (tos 0x0, ttl 64, id 36587, offset 0, flags [DF], proto UDP (17), length 125)
127.0.0.1.53 > 127.0.0.1.52426: [bad udp cksum 0xfe7c -> 0x73ca!] 2469 q: PTR? 187.171.6.107.in-addr.arpa. 1/0/0 187.171.6.107.in-addr.arpa. PTR sh-ams-nl-gp1-wk112.internet-census.org. (97)
15:40:05.862271 IP (tos 0x0, ttl 64, id 36588, offset 0, flags [DF], proto UDP (17), length 71)
127.0.0.1.36376 > 127.0.0.1.53: [bad udp cksum 0xfe46 -> 0x7ecc!] 43394+ PTR? 201.73.163.1.in-addr.arpa. (43)
15:40:06.472822 IP (tos 0x0, ttl 64, id 36599, offset 0, flags [DF], proto UDP (17), length 118)
127.0.0.1.53 > 127.0.0.1.36376: [bad udp cksum 0xfe75 -> 0xfc5d!] 43394 q: PTR? 201.73.163.1.in-addr.arpa. 1/0/0 201.73.163.1.in-addr.arpa. PTR 1-163-73-201.dynamic-ip.hinet.net. (90)
15:40:06.473610 IP (tos 0x0, ttl 64, id 36600, offset 0, flags [DF], proto UDP (17), length 71)
127.0.0.1.36479 > 127.0.0.1.53: [bad udp cksum 0xfe46 -> 0xf515!] 23967+ PTR? 130.91.73.80.in-addr.arpa. (43)
15:40:07.180389 IP (tos 0x0, ttl 64, id 36672, offset 0, flags [DF], proto UDP (17), length 130)
127.0.0.1.53 > 127.0.0.1.36479: [bad udp cksum 0xfe81 -> 0xefe7!] 23967 NXDomain q: PTR? 130.91.73.80.in-addr.arpa. 0/1/0 ns: 91.73.80.in-addr.arpa. SOA ns1.dv.rt.ru. oleg.sakha.ru. 2017091100 14400 1800 1209600 86400 (102)
15:40:07.181542 IP (tos 0x0, ttl 64, id 36675, offset 0, flags [DF], proto UDP (17), length 72)
127.0.0.1.44194 > 127.0.0.1.53: [bad udp cksum 0xfe47 -> 0xe89a!] 44387+ PTR? 3.144.123.193.in-addr.arpa. (44)
15:40:07.252518 IP (tos 0x0, ttl 64, id 36679, offset 0, flags [DF], proto UDP (17), length 156)
127.0.0.1.53 > 127.0.0.1.44194: [bad udp cksum 0xfe9b -> 0x65a5!] 44387 NXDomain q: PTR? 3.144.123.193.in-addr.arpa. 0/1/0 ns: 123.193.in-addr.arpa. SOA ns1.p78.dns.oraclecloud.net. hostmaster.oracle.com. 10 3600 600 604800 1800 (128)
15:40:07.253518 IP (tos 0x0, ttl 64, id 36682, offset 0, flags [DF], proto UDP (17), length 73)
127.0.0.1.52949 > 127.0.0.1.53: [bad udp cksum 0xfe48 -> 0x950d!] 59514+ PTR? 74.118.200.185.in-addr.arpa. (45)
15:40:07.409720 IP (tos 0x0, ttl 64, id 36698, offset 0, flags [DF], proto UDP (17), length 98)
127.0.0.1.53 > 127.0.0.1.52949: [bad udp cksum 0xfe61 -> 0x63d7!] 59514 q: PTR? 74.118.200.185.in-addr.arpa. 1/0/0 74.118.200.185.in-addr.arpa. PTR adscore.com. (70)
15:40:07.410566 IP (tos 0x0, ttl 64, id 36699, offset 0, flags [DF], proto UDP (17), length 73)
127.0.0.1.54446 > 127.0.0.1.53: [bad udp cksum 0xfe48 -> 0xb467!] 7396+ PTR? 234.155.96.156.in-addr.arpa. (45)
15:40:08.369142 IP (tos 0x0, ttl 64, id 36767, offset 0, flags [DF], proto UDP (17), length 73)
127.0.0.1.53 > 127.0.0.1.54446: [bad udp cksum 0xfe48 -> 0x33e5!] 7396 ServFail q: PTR? 234.155.96.156.in-addr.arpa. 0/0/0 (45)
15:40:08.369442 IP (tos 0x0, ttl 64, id 36768, offset 0, flags [DF], proto UDP (17), length 73)
127.0.0.1.46484 > 127.0.0.1.53: [bad udp cksum 0xfe48 -> 0xd381!] 7396+ PTR? 234.155.96.156.in-addr.arpa. (45)
15:40:08.372558 IP (tos 0x0, ttl 64, id 36773, offset 0, flags [DF], proto UDP (17), length 73)
127.0.0.1.53 > 127.0.0.1.46484: [bad udp cksum 0xfe48 -> 0x52ff!] 7396 ServFail q: PTR? 234.155.96.156.in-addr.arpa. 0/0/0 (45)
15:40:08.373658 IP (tos 0x0, ttl 64, id 36776, offset 0, flags [DF], proto UDP (17), length 72)
127.0.0.1.43634 > 127.0.0.1.53: [bad udp cksum 0xfe47 -> 0x2973!] 4891+ PTR? 53.114.81.103.in-addr.arpa. (44)
15:40:08.595015 IP (tos 0x0, ttl 64, id 36794, offset 0, flags [DF], proto UDP (17), length 160)
127.0.0.1.53 > 127.0.0.1.43634: [bad udp cksum 0xfe9f -> 0x13b4!] 4891 NXDomain q: PTR? 53.114.81.103.in-addr.arpa. 0/1/0 ns: 103.in-addr.arpa. SOA ns.apnic.net. read-txt-record-of-zone-first-dns-admin.apnic.net. 53339 7200 1800 604800 3600 (132)
15:40:08.596157 IP (tos 0x0, ttl 64, id 36797, offset 0, flags [DF], proto UDP (17), length 74)
127.0.0.1.51400 > 127.0.0.1.53: [bad udp cksum 0xfe49 -> 0xffef!] 17328+ PTR? 187.122.255.141.in-addr.arpa. (46)
^C
20 packets captured
40 packets received by filter
0 packets dropped by kernelSome of those domains above look pretty sketchy by the way. Definitely looks like a compromise of some sort... I just hope not a rootkit that masks the real origin of all requests.
PromoFaux
commented
3 years ago The next step is probably to grab a fresh SD card, flash raspbian onto it and install Pi-hole fresh. That should rule out (or confirm) a compromised system ...
nikolaybotev
commented
3 years ago As a last step before I go full wipe-out and start fresh, I am building pihole-FTL from source (v5.3.1 tag first), and will see how that does.
nikolaybotev
commented
3 years ago FYI, for FTL devs, the instructions here https://docs.pi-hole.net/ftldns/compile/#debian-ubuntu-raspbian should be updated for 5.3.1 to include libreadline-dev:
sudo apt install build-essential libgmp-dev m4 cmake libidn11-dev libreadline-devFTL can compile without readline (it just skips the capability in this case), however, I guess you hit a intermediate situation where the library is installed but the headers were missing. Thanks for the heads up, we'll change it.
nikolaybotev
commented
3 years ago FTL can compile without readline (it just skips the capability in this case), however, I guess you hit a intermediate situation where the library is installed but the headers were missing. Thanks for the heads up, we'll change it.
Gotcha. That's probably what it was.
nikolaybotev
commented
3 years ago So, custom build of pihole-FTL still flooding, and now it even established a TCP connection to itself! Previously it would only use UDP connections:
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
pihole-FTL 27063 pihole 22u IPv4 680503 0t0 TCP 127.0.0.1:58134->127.0.0.1:domain (ESTABLISHED)
pihole-FTL 27063 pihole 23u IPv4 680504 0t0 TCP 127.0.0.1:domain->127.0.0.1:58134 (ESTABLISHED)UDP connections still show up sometimes:
❯ sudo lsof +c 15 -n |grep 127.0.0.1:domain
pihole-FTL 25098 pihole 22u IPv4 681868 0t0 UDP 127.0.0.1:37380->127.0.0.1:domain
pihole-FTL 25098 25099 telnet-IPv4 pihole 22u IPv4 681868 0t0 UDP 127.0.0.1:37380->127.0.0.1:domain
pihole-FTL 25098 25100 telnet-IPv6 pihole 22u IPv4 681868 0t0 UDP 127.0.0.1:37380->127.0.0.1:domain
pihole-FTL 25098 25102 database pihole 22u IPv4 681868 0t0 UDP 127.0.0.1:37380->127.0.0.1:domain
pihole-FTL 25098 25103 housekeeper pihole 22u IPv4 681868 0t0 UDP 127.0.0.1:37380->127.0.0.1:domain
pihole-FTL 25098 25104 DNS\x20client pihole 22u IPv4 681868 0t0 UDP 127.0.0.1:37380->127.0.0.1:domain
pihole-FTL 27063 pihole 22u IPv4 680503 0t0 TCP 127.0.0.1:58134->127.0.0.1:domain (ESTABLISHED)
pihole-FTL 27063 pihole 23u IPv4 680504 0t0 TCP 127.0.0.1:domain->127.0.0.1:58134 (ESTABLISHED)And the queries keep flowing at 5-10 / second.
nikolaybotev
commented
3 years ago Is there something in the pihole-FTL database (one of the two .db files it manages) that could cause FTL to keep sending reverse DNS lookups?
I did find 2 places in the FTL code that seem to construct and maybe issue reverse DNS lookups...
dschaper
commented
3 years ago Just for my own edification, what is your /etc/resolv.conf set to?
nikolaybotev
commented
3 years ago ❯ cat /etc/resolv.conf
# Generated by resolvconf
domain fios-router.home
nameserver 127.0.0.1Based on my fresh memory, that's /etc/resolv.conf has been from before I started seeing the issue (from before I upgraded to 5.3 yesterday and started seeing all this flooding of requests).
Versions
Pi-hole version is v5.2 (Latest: v5.2) AdminLTE version is v5.2 (Latest: v5.2) FTL version is v5.3.1 (Latest: v5.3.1)
Platform
Expected behavior
Pihole-FTL has no business doing reverse DNS lookups using itself as the DNS server!
Actual behavior / bug
Pihole-FTL gets caught in. vicious cycle of perpetual reverse DNS lookups.
Steps to reproduce
Steps to reproduce the behavior:
Launch pi-hole. Wait a few minutes.
Debug Token
Screenshots
Upgraded to 5.3.1 at 12:50 PM ET:
Very shortly, starting at 1:00 PM ET the flood started:
You can see here how the top client is 127.0.0.1 below:
You can see how pihole-FTL is the actual client process here:
Additional context
Pi-hole is running on a Raspberry Pi functioning as a gateway between LAN on br0 (split to wlan0 and eth0) and WAN on eth1 (USB ethernet outbound to Internet).
Here is the routing table:
Everything is properly configured - iptables nat, sysctl ipv4 forwarding, statid IP address on br0 with DHCP, pi-hole listening on br0 only.
Also, the pi debug log fails to upload. Here it is embedded:
pihole_debug.log
``` This process collects information from your Pi-hole, and optionally uploads it to a unique and random directory on tricorder.pi-hole.net. The intent of this script is to allow users to self-diagnose their installations. This is accomplished by running tests against our software and providing the user with links to FAQ articles when a problem is detected. Since we are a small team and Pi-hole has been growing steadily, it is our hope that this will help us spend more time on development. NOTE: All log files auto-delete after 48 hours and ONLY the Pi-hole developers can access your data via the given token. We have taken these extra steps to secure your data and will work to further reduce any personal information gathered. *** [ INITIALIZING ] [i] 2020-12-02:00:06:33 debug log has been initialized. *** [ INITIALIZING ] Sourcing setup variables [i] Sourcing /etc/pihole/setupVars.conf... *** [ DIAGNOSING ]: Core version [i] Core: v5.2 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249) [i] Remotes: origin https://github.com/pi-hole/pi-hole.git (fetch) origin https://github.com/pi-hole/pi-hole.git (push) [i] Branch: master [i] Commit: v5.2-0-gfee1b8b *** [ DIAGNOSING ]: Web version [i] Web: v5.2 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249) [i] Remotes: origin https://github.com/pi-hole/AdminLTE.git (fetch) origin https://github.com/pi-hole/AdminLTE.git (push) [i] Branch: master [i] Commit: v5.2-0-g2c2d9f5 *** [ DIAGNOSING ]: FTL version [✓] FTL: v5.3.1 *** [ DIAGNOSING ]: lighttpd version [i] 1.4.53 *** [ DIAGNOSING ]: php version [i] 7.3.19 *** [ DIAGNOSING ]: Operating system [i] dig return code: 9 [i] dig response: ;; connection timed out; no servers could be reached [✗] Distro: Raspbian [✗] Error: Raspbian is not a supported distro (https://docs.pi-hole.net/main/prerequisites/) *** [ DIAGNOSING ]: SELinux [i] SELinux not detected *** [ DIAGNOSING ]: FirewallD [i] Firewalld service inactive *** [ DIAGNOSING ]: Processor [✓] armv7l *** [ DIAGNOSING ]: Networking [✓] IPv4 address(es) bound to the br0 interface: 192.168.2.1/24 matches the IP found in /etc/pihole/setupVars.conf [✓] IPv6 address(es) bound to the br0 interface: fe80::dea6:32ff:fe8c:73a3 does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127) ^ Please note that you may have more than one IP address listed. As long as one of them is green, and it matches what is in /etc/pihole/setupVars.conf, there is no need for concern. The link to the FAQ is for an issue that sometimes occurs when the IPv6 address changes, which is why we check for it. [i] Default IPv4 gateway: eth1 72.74.66.1 * Pinging eth1 72.74.66.1... [✗] Gateway did not respond. (https://discourse.pi-hole.net/t/why-is-a-default-gateway-important-for-pi-hole/3546) *** [ DIAGNOSING ]: Ports in use *:22 sshd (IPv4) *:22 sshd (IPv6) 127.0.0.1:5335 unbound (IPv4) 127.0.0.1:5335 unbound (IPv4) [80] is in use by lighttpd [80] is in use by lighttpd *:445 smbd (IPv6) *:139 smbd (IPv6) *:445 smbd (IPv4) *:139 smbd (IPv4) *:8200 minidlnad (IPv4) [53] is in use by pihole-FTL [53] is in use by pihole-FTL [4711] is in use by pihole-FTL [4711] is in use by pihole-FTL *** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain [✓] format.prod.cloud.ogury.io is 0.0.0.0 via localhost (127.0.0.1) [✓] format.prod.cloud.ogury.io is 0.0.0.0 via Pi-hole (192.168.2.1) [✗] Failed to resolve doubleclick.com via a remote, public DNS server (8.8.8.8) *** [ DIAGNOSING ]: Discovering active DHCP servers (takes 10 seconds) Scanning all your interfaces for DHCP servers Timeout: 10 seconds * Received 300 bytes from br0:192.168.2.1 Offered IP address: 192.168.2.128 Server IP address: 192.168.2.1 Relay-agent IP address: N/A DHCP options: Message type: DHCPOFFER (2) server-identifier: 192.168.2.1 lease-time: 120 ( 2m ) renewal-time: 60 ( 1m ) rebinding-time: 105 ( 1m 45s ) netmask: 255.255.255.0 broadcast: 192.168.2.255 dns-server: 192.168.2.1 domain-name: "lan" router: 192.168.2.1 --- end of options --- DHCP packets received on interface br0: 1 DHCP packets received on interface lo: 0 DHCP packets received on interface wlan0: 0 DHCP packets received on interface eth0: 0 DHCP packets received on interface wwan0: 0 DHCP packets received on interface eth1: 0 *** [ DIAGNOSING ]: Pi-hole processes [✓] lighttpd daemon is active [✓] pihole-FTL daemon is active *** [ DIAGNOSING ]: Pi-hole-FTL full status ● pihole-FTL.service - LSB: pihole-FTL daemon Loaded: loaded (/etc/init.d/pihole-FTL; generated) Active: active (exited) since Tue 2020-12-01 21:17:26 EST; 2h 49min ago Docs: man:systemd-sysv-generator(8) Process: 782 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=0/SUCCESS) Dec 01 21:17:03 Grogu systemd[1]: Starting LSB: pihole-FTL daemon... Dec 01 21:17:03 Grogu pihole-FTL[782]: Not running Dec 01 21:17:03 Grogu su[825]: (to pihole) root on none Dec 01 21:17:03 Grogu su[825]: pam_unix(su:session): session opened for user pihole by (uid=0) Dec 01 21:17:26 Grogu pihole-FTL[782]: FTL started! Dec 01 21:17:26 Grogu systemd[1]: Started LSB: pihole-FTL daemon. *** [ DIAGNOSING ]: Setup variables DHCP_ACTIVE=true DHCP_START=192.168.2.100 DHCP_END=192.168.2.251 DHCP_ROUTER=192.168.2.1 DHCP_LEASETIME=24 PIHOLE_DOMAIN=lan DHCP_IPv6=true DHCP_rapid_commit=true BLOCKING_ENABLED=true DNSMASQ_LISTENING=single DNS_FQDN_REQUIRED=true DNS_BOGUS_PRIV=true DNSSEC=false REV_SERVER=false PIHOLE_INTERFACE=br0 IPV4_ADDRESS=192.168.2.1/24 IPV6_ADDRESS= PIHOLE_DNS_1=127.0.0.1#5335 PIHOLE_DNS_2= QUERY_LOGGING=true INSTALL_WEB_SERVER=true INSTALL_WEB_INTERFACE=true LIGHTTPD_ENABLED=true CACHE_SIZE=10000 *** [ DIAGNOSING ]: Dashboard and block page [✗] Block page X-Header: X-Header does not match or could not be retrieved. HTTP/1.1 200 OK Content-type: text/html; charset=UTF-8 Expires: Wed, 02 Dec 2020 05:07:05 GMT Cache-Control: max-age=0 Date: Wed, 02 Dec 2020 05:07:05 GMT Server: lighttpd/1.4.53 [✓] Web interface X-Header: X-Pi-hole: The Pi-hole Web interface is working! *** [ DIAGNOSING ]: Gravity List and Database -rw-rw-r-- 1 pihole pihole 5382144 Dec 1 19:01 /etc/pihole/gravity.db *** [ DIAGNOSING ]: Info table property value -------------------- ---------------------------------------- version 13 updated 1606845576 gravity_count 87641 Last gravity run finished at: Tue Dec 1 12:59:36 EST 2020 ----- First 10 Gravity Domains ----- localhost.localdomain n2019cov.000webhostapp.com webmail-who-int.000webhostapp.com 010sec.com 01mspmd5yalky8.com 0byv9mgbn0.com ns6.0pendns.org dns.0pengl.com ios.0pengl.com 0x4fc271.tk *** [ DIAGNOSING ]: Groups id enabled name date_added date_modified description ---- ------- -------------------------------------------------- ------------------- ------------------- -------------------------------------------------- 0 1 Default 2020-06-22 01:37:52 2020-06-22 01:37:52 The default group *** [ DIAGNOSING ]: Domainlist (0/1 = exact white-/blacklist, 2/3 = regex white-/blacklist) id type enabled group_ids domain date_added date_modified comment ---- ---- ------- ------------ ---------------------------------------------------------------------------------------------------- ------------------- ------------------- -------------------------------------------------- 2 0 1 0 cb.sailthru.com 2020-12-01 19:01:04 2020-12-01 19:01:04 WCVB Boston Local News (link.wcvb.com CNAME) *** [ DIAGNOSING ]: Clients *** [ DIAGNOSING ]: Adlists id enabled group_ids address date_added date_modified comment ---- ------- ------------ ---------------------------------------------------------------------------------------------------- ------------------- ------------------- -------------------------------------------------- 1 1 0 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts 2020-06-22 01:37:52 2020-06-22 01:37:52 Migrated from /etc/pihole/adlists.list 2 1 0 https://mirror1.malwaredomains.com/files/justdomains 2020-06-22 01:37:52 2020-06-22 01:37:52 Migrated from /etc/pihole/adlists.list 3 1 0 https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt 2020-06-22 01:37:52 2020-06-22 01:37:52 Migrated from /etc/pihole/adlists.list 4 1 0 https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt 2020-06-22 01:37:52 2020-06-22 01:37:52 Migrated from /etc/pihole/adlists.list *** [ DIAGNOSING ]: contents of /etc/pihole -rw-r--r-- 1 root root 38 Dec 1 12:59 /etc/pihole/local.list 192.168.2.1 Grogu 192.168.2.1 pi.hole -rw-r--r-- 1 root root 234 Dec 1 12:50 /etc/pihole/logrotate /var/log/pihole.log { su root root daily copytruncate rotate 5 compress delaycompress notifempty nomail } /var/log/pihole-FTL.log { su root root weekly copytruncate rotate 3 compress delaycompress notifempty nomail } -rw-rw-r-- 1 pihole root 15 Dec 1 20:13 /etc/pihole/pihole-FTL.conf PRIVACYLEVEL=0 *** [ DIAGNOSING ]: contents of /etc/dnsmasq.d -rw-r--r-- 1 root root 1414 Dec 1 20:15 /etc/dnsmasq.d/01-pihole.conf addn-hosts=/etc/pihole/local.list addn-hosts=/etc/pihole/custom.list localise-queries no-resolv cache-size=10000 log-queries log-facility=/var/log/pihole.log local-ttl=2 log-async server=127.0.0.1 domain-needed expand-hosts bogus-priv interface=br0 server=/use-application-dns.net/ -rw-r--r-- 1 root root 647 Dec 1 12:50 /etc/dnsmasq.d/02-pihole-dhcp.conf dhcp-authoritative dhcp-range=192.168.2.100,192.168.2.251,24h dhcp-option=option:router,192.168.2.1 dhcp-leasefile=/etc/pihole/dhcp.leases domain=lan local=/lan/ dhcp-rapid-commit dhcp-option=option6:dns-server,[::] dhcp-range=::100,::1ff,constructor:br0,ra-names,slaac,24h ra-param=*,0,0 *** [ DIAGNOSING ]: contents of /etc/lighttpd -rw-r--r-- 1 root root 0 Dec 1 12:50 /etc/lighttpd/external.conf -rw-r--r-- 1 root root 4066 Dec 1 12:50 /etc/lighttpd/lighttpd.conf server.modules = ( "mod_access", "mod_accesslog", "mod_auth", "mod_expire", "mod_compress", "mod_redirect", "mod_setenv", "mod_rewrite" ) server.document-root = "/var/www/html" server.error-handler-404 = "/pihole/index.php" server.upload-dirs = ( "/var/cache/lighttpd/uploads" ) server.errorlog = "/var/log/lighttpd/error.log" server.pid-file = "/run/lighttpd.pid" server.username = "www-data" server.groupname = "www-data" server.port = 80 accesslog.filename = "/var/log/lighttpd/access.log" accesslog.format = "%{%s}t|%V|%r|%s|%b" index-file.names = ( "index.php", "index.html", "index.lighttpd.html" ) url.access-deny = ( "~", ".inc", ".md", ".yml", ".ini" ) static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" ) compress.cache-dir = "/var/cache/lighttpd/compress/" compress.filetype = ( "application/json", "application/vnd.ms-fontobject", "application/xml", "font/eot", "font/opentype", "font/otf", "font/ttf", "image/bmp", "image/svg+xml", "image/vnd.microsoft.icon", "image/x-icon", "text/css", "text/html", "text/javascript", "text/plain", "text/xml" ) mimetype.assign = ( ".ico" => "image/x-icon", ".jpeg" => "image/jpeg", ".jpg" => "image/jpeg", ".png" => "image/png", ".svg" => "image/svg+xml", ".css" => "text/css; charset=utf-8", ".html" => "text/html; charset=utf-8", ".js" => "text/javascript; charset=utf-8", ".json" => "application/json; charset=utf-8", ".map" => "application/json; charset=utf-8", ".txt" => "text/plain; charset=utf-8", ".eot" => "application/vnd.ms-fontobject", ".otf" => "font/otf", ".ttc" => "font/collection", ".ttf" => "font/ttf", ".woff" => "font/woff", ".woff2" => "font/woff2" ) include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port include_shell "find /etc/lighttpd/conf-enabled -name '*.conf' -a ! -name 'letsencrypt.conf' -printf 'include \"%p\" ' 2>/dev/null" $HTTP["url"] =~ "^/admin/" { setenv.add-response-header = ( "X-Pi-hole" => "The Pi-hole Web interface is working!", "X-Frame-Options" => "DENY" ) $HTTP["url"] =~ "\.(eot|otf|tt[cf]|woff2?)$" { setenv.add-response-header = ( "Access-Control-Allow-Origin" => "*" ) } } $HTTP["url"] =~ "^/admin/\.(.*)" { url.access-deny = ("") } expire.url = ( "" => "access plus 0 seconds" ) include_shell "cat external.conf 2>/dev/null" *** [ DIAGNOSING ]: contents of /etc/cron.d -rw-r--r-- 1 root root 1754 Dec 1 12:50 /etc/cron.d/pihole 7 4 * * 7 root PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log 00 00 * * * root PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole flush once quiet @reboot root /usr/sbin/logrotate /etc/pihole/logrotate */10 * * * * root PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker local 49 17 * * * root PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker remote @reboot root PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker remote reboot *** [ DIAGNOSING ]: contents of /var/log/lighttpd -rw-r--r-- 1 www-data www-data 2934 Dec 2 00:07 /var/log/lighttpd/error.log -----head of error.log------ 2020-11-29 00:00:54: (server.c.1759) logfiles cycled UID = 0 PID = 5472 2020-12-01 12:50:45: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 12:50:46: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 12:50:46: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-01 12:53:38: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 12:53:49: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 12:53:49: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-01 13:09:14: (mod_fastcgi.c.421) FastCGI-stderr: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /var/www/html/admin/api_db.php on line 121 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-01 19:51:28: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 19:51:49: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 19:51:49: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-01 21:16:55: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 21:17:04: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 21:17:04: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-02 00:07:05: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-02 00:07:05: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-02 00:07:05: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 -----tail of error.log------ 2020-12-01 12:50:45: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 12:50:46: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 12:50:46: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-01 12:53:38: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 12:53:49: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 12:53:49: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-01 13:09:14: (mod_fastcgi.c.421) FastCGI-stderr: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /var/www/html/admin/api_db.php on line 121 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-01 19:37:43: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-01 19:51:28: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 19:51:49: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 19:51:49: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-01 21:16:55: (server.c.2059) server stopped by UID = 0 PID = 1 2020-12-01 21:17:04: (server.c.1464) server started (lighttpd/1.4.53) 2020-12-01 21:17:04: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-02 00:02:31: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-02 00:07:05: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-02 00:07:05: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 2020-12-02 00:07:05: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: viewPort in /var/www/html/pihole/index.php on line 73 2020-12-02 00:07:05: (mod_fastcgi.c.421) FastCGI-stderr: PHP Notice: Undefined variable: serverName in /var/www/html/pihole/index.php on line 74 *** [ DIAGNOSING ]: contents of /var/log -rw-r--r-- 1 pihole pihole 129 Dec 2 00:04 /var/log/pihole-FTL.log -----head of pihole-FTL.log------ [2020-12-02 00:04:11.048 2019M] Resizing "FTL-strings" from 471040 to (475136 * 1) == 475136 (/dev/shm: 8.3MB used, 2.0GB total) -----tail of pihole-FTL.log------ [2020-12-02 00:04:11.048 2019M] Resizing "FTL-strings" from 471040 to (475136 * 1) == 475136 (/dev/shm: 8.3MB used, 2.0GB total) *** [ DIAGNOSING ]: contents of /dev/shm -rw------- 1 pihole pihole 339968 Dec 1 21:17 /dev/shm/FTL-clients -rw------- 1 pihole pihole 152 Dec 1 21:17 /dev/shm/FTL-counters -rw------- 1 pihole pihole 262144 Dec 1 21:17 /dev/shm/FTL-dns-cache -rw------- 1 pihole pihole 327680 Dec 1 21:17 /dev/shm/FTL-domains -rw------- 1 pihole pihole 28 Dec 1 23:30 /dev/shm/FTL-lock -rw------- 1 pihole pihole 20480 Dec 1 21:17 /dev/shm/FTL-overTime -rw------- 1 pihole pihole 4096 Dec 1 21:17 /dev/shm/FTL-per-client-regex -rw------- 1 pihole pihole 6881280 Dec 1 23:59 /dev/shm/FTL-queries -rw------- 1 pihole pihole 12 Dec 1 21:17 /dev/shm/FTL-settings -rw------- 1 pihole pihole 475136 Dec 2 00:04 /dev/shm/FTL-strings -rw------- 1 pihole pihole 28672 Dec 1 21:17 /dev/shm/FTL-upstreams *** [ DIAGNOSING ]: Pi-hole diagnosis messages *** [ DIAGNOSING ]: Locale LANG= *** [ DIAGNOSING ]: Pi-hole log -rw-r--r-- 1 pihole pihole 227185 Dec 2 00:07 /var/log/pihole.log -----head of pihole.log------ Dec 2 00:00:04 dnsmasq[2019]: reply configuration.apple.com is