Time out for Live Queries

[xaquib666]

Hello,

I am getting an error only for the "Live Queries" which I find strange as everything else is working.

Status: 504. Message: Get "http://192.168.1.27:3100/loki/api/v1/query_range?direction=backward&end=1721298679278000000&limit=100000&query=%7Bdns%3D%22reply%22%7D+%7C+pattern+%60%3C_%3E+%3C_%3E+%3C_%3E+%3Cclient%3E+%3Cdomain%3E+%3Ctype%3E+%3C_%3E+%3C_%3E+%3Clookup_time%3E+%3Ccache%3E+%3C_%3E%60&start=1721212279278000000&step=300000ms": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Can you please help?

Initially "Live Queries" worked too, after few hours it stopped working.

I have already tried increasing the timeout to 100s for both Loki and Prometheus, but it also did not help.

Thanks!

[ar51an]

The above Live Queries error is related to Loki, it has nothing to do with Promoetheus.

What version of Loki you are using? Are you using the Loki config.yml provided in the dashboard release?

Try modifying below values in /etc/loki/config.yml:

http_server_read_timeout: 120s
http_server_write_timeout: 120s
grpc_server_max_recv_msg_size: 20971520
grpc_server_max_send_msg_size: 20971520

[xaquib666]

Yes I was using Loki config.yml provided in the dashboard release.

After modification of the config.yml now everything again works! 💯

Thanks a lot!

One quick question though, Background I have a home lab setup with approximately 60 local domain names. 2 raspberrypi 4b with (unbound+pihole) and (unbound+adguard) running as redundant DNS servers.

1. Is raspberrypi 4b strong and stable for this purpose? Or is it the right hardware?
2. During setup I had to restart a number of time everytime the cache was rebuilt from scratch. and during rebuild time (several days) new websites were obviously slow. Any way to make the cache persistant and retain through restarts?

Thanks again!

[ar51an]

1) Raspberrypi is very powerful. You can overclock it as well. I have been running Rasspberrypi 4 overclocked at 2.0GHz for 3+ years now. You need some cooling if you overclock it, like mini fan or place it in temperature controlled environment (AC). https://github.com/ar51an/raspberrypi-fan-control

If you are not satisfied with RP4 performance and want to upgrade then buy some mini PC (like HP or DELL). They are more powerful and slightly larger then RP4. You can get a good refurbished older model under 100$ on ebay.

2) Yes if you are willing to compile unbound. Redis module does exactly that, it persists cache through reboots. Redis is not available on the distro unbound version. https://github.com/ar51an/unbound-redis (Remove the adblock/blocklist part as you are already using pihole and adguard, just compile the unbound and use it with redis for persistent cache).

If you want to stick with distro version of unbound, then the solution is to use couple of scripts and services. They will save and reload the cache on reboots. I wrote and used these scripts and services long time ago before I started using the redis. They worked perfectly for preserving the cache back then, may require some changes now. Pasting below if you want to use them.

unbound-savecache.sh

#!/bin/bash

# Config
CACHE=/opt/unbound/cache.dump

## Saving unbound cache
{ echo -e "Saving Unbound Cache"; } 2> /dev/null
[ -f $CACHE ] && rm $CACHE
# Save cache
if unbound-control dump_cache > $CACHE; then
    chmod 0600 $CACHE
    echo -e "Cache saved ..."
else
    echo -e "Failed to save cache ..."
    rm $CACHE
fi

unbound-savecache.service

[Unit]
Description=Unbound save cache on shutdown/reboot
DefaultDependencies=no
Requires=network.target
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/true
ExecStop=/opt/unbound/scripts/unbound-savecache.sh

[Install]
WantedBy=multi-user.target

unbound-loadcache.sh

#!/bin/bash

# Config
CACHE=/opt/unbound/cache.dump

## Loading unbound cache
{ echo -e "Loading Unbound Cache"; } 2> /dev/null
# Load cache
if [ -f $CACHE ]; then
    cat $CACHE | unbound-control load_cache 1> /dev/null
    echo -e "Cache loaded ..."
    rm $CACHE
else
    echo -e "Failed to load cache, file does not exist ..."
fi

unbound-loadcache.service

[Unit]
Description=Unbound load cache on boot
After=unbound.service
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=false
ExecStart=/opt/unbound/scripts/unbound-loadcache.sh

[Install]
WantedBy=multi-user.target

Note: Do not start the services, just enable them with below cmds. They run automatically on reboot. sudo systemctl enable unbound-savecache.service sudo systemctl enable unbound-loadcache.service

[xaquib666]

Thanks!

The script method would be perfect for me 👍🏾

A naive question, can cou please tell me how to run a script and where to place the service files? Never done it before.

I plan to put the scripts in /opt/unbound/scripts/ and services in /etc/systemd/system/ both with root permission.

And run the the script sudo sh ./script-name-here.sh as root.

sounds ok?

[ar51an]

Yes, both scripts to /opt/unbound/scripts/ and services to /etc/systemd/system/. Do sudo chown root:root to all 4 files and sudo chmod +x to both scripts only. You do not need to run them manually, services will run them automatically on reboot. Just enable the services as mentioned in the previous comment.

[xaquib666]

Hello,

Is there a way to check if the script is working without restarting?

I had to restart and looks like the cache was not restored.

Thanks!

[ar51an]

I just tested the scripts & services now and they worked. Scripts worked fine individually and services worked fine on reboot in running the scripts automatically.

Here is how you can test the scripts without reboot and services:

sudo /opt/unbound/scripts/unbound-savecache.sh Above cmd should save the /opt/unbound/cache.dump file. Check the output of the script on the shell and check if the file exists.

sudo /opt/unbound/scripts/unbound-loadcache.sh Above cmd will reload the cache from /opt/unbound/cache.dump file and delete it. Check the output of the script on the shell.

The cache counts in the dashboard will not be exactly the same. If this is what made you think that it did not work. Here is why:

When unbound is running it contains various types of records and data in cache, like negative data, bogus data, internal queries it sent to retrieve records, DNSSEC validation records ... Many of these things are just part of the running process. Unbound does not save everything as part of saving cache, otherwise you will get wrong response. Bottom line is save cache is saving everything what Unbound allows to save and these are queries that client (you) sent.

Things you should be looking at to see if the cache is loaded after the restart:

1) The cache counts in the dashboard will not be same but they won't be as low as you start the unbound without any cache. (Explained above) 2) Webpages are opening faster, just like they do after cache is built. 3) In the dashboard, most of the values in the 'Reply' column under 'Live Queries' panel are '0 s'. 4) 'Cache Hits' count in the dashboard will start much higher if cache is reloaded. Without reloading the cache it start much lower. 5) This list will go on and on and but the above are some basic things you should check.

[xaquib666]

Awesome! Yes it does work! I was looking at the size and count whicht didnt match so confused me.

Thanks a lot!