search

scottmuc / infrastructure

Documentation / Automation for personal third-party infrastructure
The Unlicense
11 stars 2 forks source link

Use Loki to query top requested domains #77

Closed scottmuc closed 4 months ago

scottmuc commented 4 months ago

As part of implementing #76 I was able to get a better idea of the shape and volume of my DNS traffic.

Stats for the last 24hrs

Here's my dnsmasq

image

And for unbound

image

Noteworthy Observations

Todo

scottmuc commented 4 months ago

Loki and Promtail installed

The above commits gets me at least this far.

promtail

image

loki

image

Not sure how to see fail2ban results from this. Guessing I might need to do some more configuration.

scottmuc commented 4 months ago

Switching DNS frontends

The above commit makes unbound be the first hop for a DNS query. With this configuration it has 3 things it can do with the request (in order of likelihood):

Now I can see which hosts performed certain queries (this is my Ripe Atlas device):

[1720504961] unbound[269366:0] query: 192.168.2.16 topOLogY6.DyndNs.AtLaS.Ripe.NEt. AAAA IN
[1720504961] unbound[269366:0] reply: 192.168.2.16 topOLogY6.DyndNs.AtLaS.Ripe.NEt. AAAA IN NOERROR 0.045840 0 77

In my dnsmasq logs, I can see that hosts are appending the search domain if the initial query is refused:

2024-07-09T06:36:57.547949+01:00 pippin dnsmasq[269326]: query[A] region1.app-measurement.com.middle-earth.internal from 192.168.2.10
2024-07-09T06:36:57.548913+01:00 pippin dnsmasq[269326]: config region1.app-measurement.com.middle-earth.internal is NXDOMAIN

I can also see what HTTPS requests are looking like (this would happen when I connect to grafana):

2024-07-09T07:05:00.952685+01:00 pippin dnsmasq[269326]: query[HTTPS] _3000._https.pippin.middle-earth.internal from 192.168.2.10
2024-07-09T07:05:00.954117+01:00 pippin dnsmasq[269326]: config _3000._https.pippin.middle-earth.internal is NXDOMAIN
2024-07-09T07:05:00.954774+01:00 pippin dnsmasq[269326]: query[A] pippin.middle-earth.internal from 192.168.2.10
2024-07-09T07:05:00.955408+01:00 pippin dnsmasq[269326]: /etc/hosts pippin.middle-earth.internal is 192.168.2.10

Lastly, the dashboard reveals that dnsmasq doesn't count queries it doesn't forward:

image

scottmuc commented 4 months ago

First Loki Data in Dashboard

Can't say I full understand what I'm doing but I can take a line in /var/log/unbound/unbound.log that looks like:

[1720637568] unbound[269366:0] reply: 192.168.2.197 ocsp.r2m01.amazontrust.com. HTTPS IN NOERROR 0.034493 0 125

and parse it with:

{dns="reply"} | pattern `<_> <_> <_> <client> <domain> <type> <_> <_> <lookup_time> <cache> <_> `

image

I can see that the above is powered by the following in my promtail/config.yml

https://github.com/scottmuc/infrastructure/blob/b9b2802aff16960bedb528bbb47faee2f0a72e7c/devices/pippin/root/etc/promtail/config.yml#L25-L29

Now I think I can start thinking about what types of information I want to know from the logs. First ones that come to mind:

I don't need a running log of requests like I've pasted above, but it was enough to learn a little bit how the whole system comes together.

scottmuc commented 4 months ago

Success!

Turned out this required more learning how to configure grafana visualizations. Still... I can answer the questions I posed to myself as tasks. With the screenshot below you can answer the following:

image

scottmuc commented 4 months ago

Summary

The dashboard still needs work, but very pleased that I have a facility to inquire more about my DNS traffic. It shows that I have a lot to learn about grafana visualizations. Setting up the log collection, ingestion into loki, and requesting from grafana was the easier part.

Some interesting observations: