Bug: Advanced Blocking App doesn't respect allowListUrls, !allowlist, or explicit Allow

[shawnkhall]

The Advanced Blocking App doesn't respect allowListUrls, !allowlist, or explicit Allow when a domain also exists in a list from adblockListUrls.

I've added allowlists within the main Settings=>Blocking=>Block List URLs block using the !allowlist syntax, within the Advanced Blocking config as allowListUrls, and in no instance will it actually observe domains from these allowlists.

When using the allowListUrls option it is downloading and parsing the allowlist, it just isn't respecting the allowlist.

[2022-07-04 00:54:40 Local] DNS App [Advanced Blocking]: Advanced Blocking app successfully downloaded allow list (3.53 KB): https://example.com/s/allowlist.txt
[2022-07-04 00:54:40 Local] DNS App [Advanced Blocking]: Advanced Blocking app is reading allow list from: https://example.com/s/allowlist.txt
[2022-07-04 00:54:40 Local] DNS App [Advanced Blocking]: Advanced Blocking app read allow list file (222 domains) from: https://example.com/s/allowlist.txt

I even defined individual domains explicitly within the global Allow section but they weren't observed.

The only way I found to get the Advanced Blocking App to allow an individual domain is to explicitly list is within the Advanced Blocking App config in the allowed array. For large allowlists this is a problem.

Since I don't want to recreate the wheel, I believe that the allowlists should always take precedence over blocklists. An explicit Allowed/Allow or !allowlist entry should supercede anything appearing in the blocklists in the Advanced Blocking App.

[ShreyasZare]

Thanks for the feedback. The DNS server's built in blocking features and the Advanced Blocking app are totally independent of each other.

Also, the DNS apps get a chance to process the request first so Advanced Blocking app will get to process the request before the DNS server's built in blocking feature. So, you should either use the built-in blocking features or use the Advanced Blocking app to avoid any confusion with the config.

The Advanced Blocking app's allowListUrls works as expected. In some case it may happen that the app will allow the request to move further based on the allowed list and it will reach the DNS server's built-in blocking feature which may then block the domain as per its block lists.

Do try disabling the DNS server's built in blocking feature and try with only Advanced Blocking app.

[shawnkhall]

Here's a sample config that demonstrates that the allowListUrls does not work as expected.

{
  "enableBlocking": true,
  "blockListUrlUpdateIntervalHours": 12,
  "networkGroupMap": {
    "0.0.0.0/0": "everyone",
    "192.168.1.0/24": "local",
  },
  "groups": [
    {
      "name": "everyone",
      "enableBlocking": true,
      "allowTxtBlockingReport": true,
      "blockAsNxDomain": false,
      "blockingAddresses": [
        "0.0.0.0",
        "::"
      ],
      "allowed": [],
      "blocked": [],
      "allowListUrls": [],
      "blockListUrls": [],
      "allowedRegex": [],
      "blockedRegex": [],
      "regexAllowListUrls": [],
      "regexBlockListUrls": [],
      "adblockListUrls": [ 
        "https://example.com/s/tdns/adblocklist.txt"
      ]
    },{
      "name": "local",
      "enableBlocking": true,
      "allowTxtBlockingReport": true,
      "blockAsNxDomain": false,
      "blockingAddresses": [
        "0.0.0.0",
        "::"
      ],
      "allowed": [],
      "blocked": [],
      "allowListUrls": [
        "https://example.com/s/tdns/allowlist.txt"
      ],
      "blockListUrls": [],
      "allowedRegex": [],
      "blockedRegex": [],
      "regexAllowListUrls": [],
      "regexBlockListUrls": [],
      "adblockListUrls": []
    }
  ]
}

The items within the urls at adblockListUrls are expected to be blocked everywhere (including on "local"). This works.

[Adblock Plus 1.1]
! Title: Sample blocklist for Technitium DNS Server

||anydesk.com^
||gotomypc.com^
||microsoft.com^
||teamviewer.com^

The items within the urls at allowListUrls are expected to be allowed when at local (anywhere on 192.168.1.0/24). This does not happen.

anydesk.com
gotomypc.com
teamviewer.com

If I check the txt record for any of the domains listed in adblocklist.txt I get the same result, even though some are listed in allowlist.txt. The txt record clearly shows that the Advanced Blocking App is responsible, and the specific list. Two of these three records should have been allowed -- microsoft.com should continue to be blocked. It's not applying the local group, only everyone.

[2022-07-04 00:20:33 Local] [192.168.1.59:58569] [UDP] QNAME: anydesk.com; QTYPE: TXT; QCLASS: IN; RCODE: NoError; ANSWER: ["source=advanced-blocking-app; group=everyone; blockListUrl=https://example.com/s/tdns/adblocklist.txt; domain=anydesk.com"]
[2022-07-04 00:20:45 Local] [192.168.1.59:58571] [UDP] QNAME: microsoft.com; QTYPE: TXT; QCLASS: IN; RCODE: NoError; ANSWER: ["source=advanced-blocking-app; group=everyone; blockListUrl=https://example.com/s/tdns/adblocklist.txt; domain=microsoft.com"]
[2022-07-04 00:21:06 Local] [192.168.1.59:57701] [UDP] QNAME: teamviewer.com; QTYPE: TXT; QCLASS: IN; RCODE: NoError; ANSWER: ["source=advanced-blocking-app; group=everyone; blockListUrl=https://example.com/s/tdns/adblocklist.txt; domain=teamviewer.com"]

If I completely eliminate the everyone group and use only the local group (applying the same adblockListUrls and allowListUrls within the singular local group) then it will block and allow the listed domains correctly, but this effectively eliminates the only real benefit of the Advanced Blocking App: network segregation.

[shawnkhall]

Is there a diagram or list identifying exactly what order lists and records are processed in?

[ShreyasZare]

Thanks for the config. The network to group mapping selects a single group for the request IP address. So, the everyone group will never be selected from your local network since you have the local network's subnet mapped to local group. Each group is independent and do not share anything with each other.

You will need to move the adblockListUrls into the local group to make it work.

[shawnkhall]

Understood. Thank you.

With all the other testing to try to figure out what was going on here I found another issue. It looks like the cache behavior is inconsistent. If a list returns an error the first time it is requested it will not be attempted again (even if you edit the config file) unless you restart the Technitium DNS service.

[ShreyasZare]

Thanks for the feedback. When a list fails to download, it will be retried only when the update interval that is configure elapses or when the App starts (when DNS server starts for example).

Saving the app config does not trigger update unless you remove the list url once and then add it back again in the config. Here too if the block list cache already has the file downloaded for the list url then the app avoids to download it again.

The app basically tries to avoid frequent download of the lists this way.

In your case when the list url failed with error, you can try removing the url from the config once, save it, and again edit it to add the list url. This should trigger the download. Let me know if that worked.

[ShreyasZare]

Closing this issue. Reopen if needed.