search

opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.3k stars 734 forks source link

unexpected Unbound DNS blocklist caching #7207

Closed alexandervlpl closed 2 months ago

alexandervlpl commented 8 months ago

Describe the bug

The "Update Unbound DNSBLs" cron job cannot actually update blocklists more frequently than every 20 hours because of a local, hardcoded cache managed by the blocklist scripting. This is undocumented, undesired and very unintuitive.

To Reproduce

Steps to reproduce the behavior:

  1. Enable Unbound DNS blocklist.
  2. Add blocklist URL. Probably also happens with the predefined blocklists, but haven't tried that.
  3. Enable "Update Unbound DNSBLs" cron.

Expected behavior

Ideally the update scripting should respect cache control headers sent by the blocklist service. That's the standard way to tell clients how frequently data is updated and how many requests they should be making. At least there should be a setting for this instead of the very long hardcoded cache_ttl = 72000. In my case the URL is hosted by me and I want to update frequently to respond to abuse on my network. Is this local cache even necessary?

Environment

OPNsense 23.1.11 (amd64)

swhite2 commented 8 months ago

The main function of the cache is to prevent spamming URLs due to misconfigured cron jobs. We can add a field to control the cache duration, but it should have a documented minimum of an hour.

Cache control headers are unreliable as they may not be set by the service maintaining the list.

alexandervlpl commented 8 months ago

Maybe you could add the cache TTL field, give it a reasonable default like 1 hour but allow it to be any value? Or perhaps exclude custom URLs, since this is an "advanced" option and already the user's responsibility? And document this cache feature so people like me aren't left wondering why their blocklist didn't update.

As a webmaster, I respect the "prevent spamming" concern. :+1: These lists are almost static though, I'm sure there's lots of caching on their side and many of the ones I've seen use a CDN. Even the worst possible * * * * * cron every minute probably isn't abuse in this case, but I may be wrong.

OPNsense-bot commented 2 months ago

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.