Adding adblocking to the mix to cut down bandwidth needs for web browsing even more?

[dhents]

First of all i want to thank everyone involved for their work in this project (and the related ones).

I have an idea for something we already implemented in our LAN which might be interesting for your project too, since caching updates and common downloads is all about saving internet bandwith. Please bear with me in the following text where i briefly describe what we implemented in our setup, what our gains from it are and how it is possible to add to this project.

We are currently running an older version from one of the older lancache repos for steam only at our Lanparties (about 300 players using a few 16mbit DSL lines balanced by a zeroshell) and realized that adding a dns-based adblocker to our setup cut down our webtraffic by at least 20%.

We use pihole as a web advertisement filtering solution. They live over here: github.com/pi-hole

Basically our border firewall drops all DNS requests to the outside that do not originate from one of our designated LAN DNS servers in exact the way we want to forward them from one to another. Our users must use the DNS server we tell them to, everything else won't work.

Web traffic (calls to 80 and 443) do get routed through one of our DSL lines, the cache uses another one. All other traffic from users, in this case only things like voice tools or playing onlinegames balances over the others. So cutting down web traffic by cutting out advertisements helps everyone surf faster, also since it is being blocked on the network level, mobile devices don't show any ads (which is a nice side-effect). I actually use pihole not only in our lanparty setup, but also at home. I realize that adding this would be a change of scope from caching to caching+filtering in which i can understand arguments of being opposed to that, but i still think this is worth mentioning as an idea.

Our DNS setup is currently daisy-chained like this ( "--> means 'forwards to'"): Client-PCs --> pihole --> cache --> gateway, so only the domains we want to have resolved (no bads/ads, cachable content only over the cache) do actually resolve and point to where we want them.

I have to admit i didn't yet find time to upgrade our setup to your seemingly great implementation here. But my thoughts already revolve around how to keep pihole in our setup once we switched to lancache, and i think this should probably be fairly easy for us. Since this project already implements 20 additional virtual hosts by ip adress aliases i propose to add one more to bind piholes services to it (unbound is already limited to the first one anyways), and have unbound forward everything it does not already have records for based on our caching needs to pihole which in turn forwards everything not on the blacklists to either our border gateway, or google dns directly.

We will be including pihole in our setup anyways, you decide if maybe this idea is helpful to you/your lanparty guests or not, or to include it or maybe just offer the option to. I'm fairly new to github and am willing to help, so this is the first step of me trying to be helpful and sharing ideas. I also want to hear your opinions on this matter before i try to figure out this pull request thing unasked. I didn't use github before, but i have some experience with gitlab and bitbucket from work. I hope i wrote this in the right place since i only just read that the issue tracker is also for discussions, ideas and feature requests.

[nexusofdoom]

so something like this

https://www.reddit.com/r/pihole/comments/7blq3j/how_to_install_pihole_with_unbound/

[elico]

@nexusofdoom @dhents or dnsmasq or similar caching proxies.

[dhents]

Yes, something like this. Integration into the DNS server we already have makes even more sense.

[billthecatt]

dnsmasq is generally a proxy and doesn't in practice actually cache very well at all, in my experience, vs unbound.. To experience for yourself, spin up dnsmasq and ask for google.com three times, all three requests go to the network every time (use wireshark and watch it go nuts..) Then ask for the same thing again, and you'll see all three requests over and over. Sure dns TTL is coming into play, and my dnsmasq config may be sub optimal for caching, but I get much much better bandwidth reduction using unbound for DNS from LAN clients.

I thought I was doing a good thing having three dnsmasq processes stood up for redundancy for clients, but each dnsmasq instance amplifies the traffic and you end up with 3x the amount you really wanted. dnsmasq doesn't just ask the first server in the config, it asks all three, all the time, and then gives the client the quickest reply.

TL:DR. Use unbound instead of dnsmasq..

[It0w]

Hi ;)

I have written this little script for my Opnsense which acts as a DNS server for me

#!/usr/local/bin/bash

tmp1="$(mktemp)" && wget --no-cache -q -c -O "$tmp1" https://raw.githubusercontent.com/oznu/dns-zone-blacklist/master/unbound/unbound-nxdomain.blacklist
tmp2="$(mktemp)" && sort <$tmp1 >$tmp2 && rm $tmp1
tmp3="$(mktemp)" && cat /root/lancache.conf "$tmp2" >"$tmp3"

if [ "$1" == change ]; then
        tmp4="$(mktemp)" && diff -bBus "$tmp3" "/var/unbound/ads_and_lancache.conf" > "$tmp4"
        add_lines=`cat "$tmp4" | grep ^+ | wc -l`
        del_lines=`cat "$tmp4" | grep ^- | wc -l`
        #add_lines=`expr $add_lines - 1`
        #del_lines=`expr $del_lines - 1`
        rm "$tmp4"
        printf "Total added lines:  "
        printf "%10s\n" "$add_lines"
        printf "Total deleted lines:"
        printf "%10s\n" "$del_lines"
fi

mv "$tmp3" /var/unbound/ads_and_lancache.conf
rm "$tmp2"
chown unbound:unbound /var/unbound/ads_and_lancache.conf
chroot -u unbound -g unbound / unbound-control -c /var/unbound/unbound.conf reload
exit 0

Of course you had to add the blacklist in the configuration file

include: /var/unbound/ads_and_lancache.conf

run the script on a daily base via cron

keep in mind !!this will not work out of the box!! the script must be adapted to your needs