search

nin9s / elk-hole

elasticsearch, logstash and kibana configuration for pi-hole visualiziation
MIT License
204 stars 36 forks source link

Grok Parse Failing After Pihole Update #48

Closed bjmaynard01 closed 2 years ago

bjmaynard01 commented 2 years ago

I recently updated pihole on my instance to be this version:

Pi-hole version is v5.11.4 (Latest: v5.11.4) AdminLTE version is v5.13 (Latest: v5.13) FTL version is v5.16.1 (Latest: v5.16.1)

Now I am getting grok parse failures. I'm pretty sure that pihole has modified the format that information is logged in. Below are a few sample lines from my new log file. I have blanked out my internal IPs, but changed nothing else about the format. Also, just FYI, the log location has changed to /var/log/pihole/pihole.log

Jul 15 12:36:11 dnsmasq[2020]: query[A] zwyr157wwiu6eior.com from XXX.XXX.XXX.XXX Jul 15 12:36:11 dnsmasq[2020]: forwarded zwyr157wwiu6eior.com to 208.67.220.220 Jul 15 12:36:11 dnsmasq[2020]: reply zwyr157wwiu6eior.com is 104.17.207.102 Jul 15 12:36:11 dnsmasq[2020]: reply zwyr157wwiu6eior.com is 104.16.160.101 Jul 15 12:36:13 dnsmasq[2020]: query[A] signaler-pa.clients6.google.com from XXX.XXX.XXX.XXX Jul 15 12:36:13 dnsmasq[2020]: forwarded signaler-pa.clients6.google.com to 208.67.220.220 Jul 15 12:36:13 dnsmasq[2020]: reply signaler-pa.clients6.google.com is 172.253.115.95 Jul 15 12:36:17 dnsmasq[2020]: query[A] downloads.napps-2.com from XXX.XXX.XXX.XXX Jul 15 12:36:17 dnsmasq[2020]: forwarded downloads.napps-2.com to 208.67.220.220 Jul 15 12:36:17 dnsmasq[2020]: reply downloads.napps-2.com is 172.67.146.24 Jul 15 12:36:17 dnsmasq[2020]: reply downloads.napps-2.com is 104.21.39.138 Jul 15 12:36:39 dnsmasq[2020]: query[A] video.us.bytedance.map.fastly.net from XXX.XXX.XXX.XXX Jul 15 12:36:39 dnsmasq[2020]: forwarded video.us.bytedance.map.fastly.net to 208.67.220.220 Jul 15 12:36:39 dnsmasq[2020]: reply video.us.bytedance.map.fastly.net is 146.75.30.73 Jul 15 12:36:47 dnsmasq[2020]: query[A] www.google.com from XXX.XXX.XXX.XXX Jul 15 12:36:47 dnsmasq[2020]: forwarded www.google.com to 208.67.220.220 Jul 15 12:36:47 dnsmasq[2020]: reply www.google.com is 172.253.122.99 Jul 15 12:36:47 dnsmasq[2020]: reply www.google.com is 172.253.122.103 Jul 15 12:36:47 dnsmasq[2020]: reply www.google.com is 172.253.122.104 Jul 15 12:36:47 dnsmasq[2020]: reply www.google.com is 172.253.122.105 Jul 15 12:36:47 dnsmasq[2020]: reply www.google.com is 172.253.122.106 Jul 15 12:36:47 dnsmasq[2020]: reply www.google.com is 172.253.122.147

I am currently trying to debug the filters, but am having little luck.

nin9s commented 2 years ago

I don’t have that kind of problem with the most recent version. can you check at which position in the log file your grokfailure starts? you may me able to see which fields still have content and where it starts to fail.

for me its a symlink to the „new“ location already

ls -lath /var/log/pihole.log
lrwxrwxrwx 1 pihole pihole 26 Jul 17 08:07 /var/log/pihole.log -> /var/log/pihole/pihole.log

can you confirm?

bjmaynard01 commented 2 years ago

Yes, can confirm that path changed. I also found out what my issue was. When I updated pihole it reverted the dnsmasq.d setting I had because I used the default config instead of using an appending conf file....lesson learned. Was missing the log-queries=extra line from my configs.