Whitelist not applying to Wildcard lists

[devipasigner]

Hello, thanks for making this awesome piece of software and all the updates recently. Im trying out the wildcard and adblock syntax support and the blocking works but theres a few issues.

Im trying to use a whitelist by putting a "!" in front of the whitelist url but it is not being applied to the lists im using (hagezi wildcard + adblock tlds).

And another thing, can you make it so technitium can read more advanced adblock rules? Like for example I want to use hagezis abused tlds list and it works except for the whitelist part. See https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

[ShreyasZare]

Thanks for the feedback. Please share the exact block list you are using so that I will try to reproduce the issue here.

For Adblock plus format, the format itself supports adding exception using @@||. So for this format, its recommended to use this syntax instead of ! in the config.

[devipasigner]

For Adblock plus format, the format itself supports adding exception using @@||. So for this format, its recommended to use this syntax instead of ! in the config.

I’m talking about adding allowlists. If I wanted to add an allow list I would put a ! Before the URL right? Since we are adding a list and not a rule

[devipasigner]

Thanks for the feedback. Please share the exact block list you are using so that I will try to reproduce the issue here.

thank you, I’m using the hagezis ultimate and threat blocklists. I’m trying to use his referral allowlist.

blacklists (in wildcard): https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif.txt

Whitelists (in Adblock): https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt

[devipasigner]

Abused tld lists:

depends if you rather adopt Adblock ublock or AdGuard home syntax variations

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

[ShreyasZare]

thank you, I’m using the hagezis ultimate and threat blocklists. I’m trying to use his referral allowlist.

blacklists (in wildcard): https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif.txt

Whitelists (in Adblock): https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt

The white list that you are using in Adblock format already uses the @@|| notation to denote that the domain names in there are excluded from blocking. So, as I said, adding the ! before Adblock URL in the config wont work as expected since the format already has notation to add domain to allow list. The ! config is required for formats like hosts file format where there is no such option.

[ShreyasZare]

Abused tld lists:

depends if you rather adopt Adblock ublock or AdGuard home syntax variations

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

[devipasigner]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

Yes, if you could add this it would be great. Thanks

[devipasigner]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

I understand now, thank you!

[ShreyasZare]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

Yes, if you could add this it would be great. Thanks

From the format documentation, this is designed for browser plugins and not possible to be implemented with DNS. DNS server does not know which website you are on to know how to respond to the next request.

[devipasigner]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

Yes, if you could add this it would be great. Thanks

From the format documentation, this is designed for browser plugins and not possible to be implemented with DNS. DNS server does not know which website you are on to know how to respond to the next request.

It’s working with AdGuard home.. this can be used to exclude a certain domain from the tld.

for example look at the hagezi list. We block .gq TLDS but for example want inege.gq to be excluded

[ShreyasZare]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

Yes, if you could add this it would be great. Thanks

From the format documentation, this is designed for browser plugins and not possible to be implemented with DNS. DNS server does not know which website you are on to know how to respond to the next request.

It’s working with AdGuard home.. this can be used to exclude a certain domain from the tld.

for example look at the hagezi list. We block .gq TLDS but for example want inege.gq to be excluded

Which specific hagezi list are you refering to? The syntax for exclusion is @@|| for adblock formats.

The uBlock documentation clearly says with example that the filter applies only to the specified domain names. That is, when you are on the specified domain's website, that the filtered domain will be blocked otherwise it wont be blocked.

If you have any documentation source which says something different then do share it here.

[devipasigner]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

Yes, if you could add this it would be great. Thanks

From the format documentation, this is designed for browser plugins and not possible to be implemented with DNS. DNS server does not know which website you are on to know how to respond to the next request.

It’s working with AdGuard home.. this can be used to exclude a certain domain from the tld. for example look at the hagezi list. We block .gq TLDS but for example want inege.gq to be excluded

Which specific hagezi list are you refering to? The syntax for exclusion is @@|| for adblock formats.

The uBlock documentation clearly says with example that the filter applies only to the specified domain names. That is, when you are on the specified domain's website, that the filtered domain will be blocked otherwise it wont be blocked.

If you have any documentation source which says something different then do share it here.

See here; https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt

[devipasigner]

Oh and I have another request. Can you support regex in blocklists? I want to be able to use special lists like this (which works for adguard home). I miss having this functionality.

https://github.com/DandelionSprout/adfilt/blob/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareAdGuardHome.txt

might have to scroll down

[devipasigner]

Another request: can you change the default blocking mode to null ip rather than NXDOMAIN? NXDOMAIN can cause devices to try to use other DNS providers which is why it isn’t the default for pihole or adguard home anymore. And another small request: can the NXDOMAIN app include apples private relay domains? Apple has documented this.

[ShreyasZare]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

Yes, if you could add this it would be great. Thanks

From the format documentation, this is designed for browser plugins and not possible to be implemented with DNS. DNS server does not know which website you are on to know how to respond to the next request.

It’s working with AdGuard home.. this can be used to exclude a certain domain from the tld. for example look at the hagezi list. We block .gq TLDS but for example want inege.gq to be excluded

Which specific hagezi list are you refering to? The syntax for exclusion is @@|| for adblock formats. The uBlock documentation clearly says with example that the filter applies only to the specified domain names. That is, when you are on the specified domain's website, that the filtered domain will be blocked otherwise it wont be blocked. If you have any documentation source which says something different then do share it here.

See here; https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt

That list is using AdGuard syntax and not Adblock syntax so it not supported yet.

[ShreyasZare]

Oh and I have another request. Can you support regex in blocklists? I want to be able to use special lists like this (which works for adguard home). I miss having this functionality.

https://github.com/DandelionSprout/adfilt/blob/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareAdGuardHome.txt

might have to scroll down

The problem with regex is that the regex syntax is not a standard and so the syntax they use may not work with what dotnet supports.

[ShreyasZare]

Another request: can you change the default blocking mode to null ip rather than NXDOMAIN? NXDOMAIN can cause devices to try to use other DNS providers which is why it isn’t the default for pihole or adguard home anymore.

The default blocking type was recently updated to NXDOMAIN since it works better with Extended DNS Errors as they get cached and relayed to all clients. You can change the option to use 0.0.0.0 address if you wish to from the Settings.

And another small request: can the NXDOMAIN app include apples private relay domains? Apple has documented this.

Can you provide links to it? I don't have any apple devices so I wont be able to test those domains.

[devipasigner]

Another request: can you change the default blocking mode to null ip rather than NXDOMAIN? NXDOMAIN can cause devices to try to use other DNS providers which is why it isn’t the default for pihole or adguard home anymore.

The default blocking type was recently updated to NXDOMAIN since it works better with Extended DNS Errors as they get cached and relayed to all clients. You can change the option to use 0.0.0.0 address if you wish to from the Settings.

And another small request: can the NXDOMAIN app include apples private relay domains? Apple has documented this.

Can you provide links to it? I don't have any apple devices so I wont be able to test those domains.

No problem, here: https://developer.apple.com/support/prepare-your-network-for-icloud-private-relay

[devipasigner]

Oh and I have another request. Can you support regex in blocklists? I want to be able to use special lists like this (which works for adguard home). I miss having this functionality. https://github.com/DandelionSprout/adfilt/blob/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareAdGuardHome.txt might have to scroll down

The problem with regex is that the regex syntax is not a standard and so the syntax they use may not work with what dotnet supports.

Hmm, so it wouldnt be possible to intergrate?

[devipasigner]

thank you, I’m using the hagezis ultimate and threat blocklists. I’m trying to use his referral allowlist. blacklists (in wildcard): https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif.txt Whitelists (in Adblock): https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt

The white list that you are using in Adblock format already uses the @@|| notation to denote that the domain names in there are excluded from blocking. So, as I said, adding the ! before Adblock URL in the config wont work as expected since the format already has notation to add domain to allow list. The ! config is required for formats like hosts file format where there is no such option.

Can you please try my setup? Whitelisting is still not working for me.

My black/allowlists are as follows: (copied directly)

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/dyndns.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt

Domains used to test: www.googleadservices.com

Should be whitelisted by https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt Blacklisted by https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate.txt

Ultimate only blocks *googleadservices.com But the whitelist should allow www.googleadservices.com

[devipasigner]

Abused tld lists: depends if you rather adopt Adblock ublock or AdGuard home syntax variations https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

The uBlock format is kind of extended Adblock format. The DNS server will be able to read the TLD domain names in there but not the domain names that are followed by the domain= syntax.

Yes, if you could add this it would be great. Thanks

From the format documentation, this is designed for browser plugins and not possible to be implemented with DNS. DNS server does not know which website you are on to know how to respond to the next request.

It’s working with AdGuard home.. this can be used to exclude a certain domain from the tld. for example look at the hagezi list. We block .gq TLDS but for example want inege.gq to be excluded

Which specific hagezi list are you refering to? The syntax for exclusion is @@|| for adblock formats. The uBlock documentation clearly says with example that the filter applies only to the specified domain names. That is, when you are on the specified domain's website, that the filtered domain will be blocked otherwise it wont be blocked. If you have any documentation source which says something different then do share it here.

See here; https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds.txt

That list is using AdGuard syntax and not Adblock syntax so it not supported yet.

Would be great if you could have support for both, since they are very similar. Also, what about ublock origin? Is that the standard for adblock syntax? He also has a ublock version, https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

Again, works the same

[ShreyasZare]

The problem with regex is that the regex syntax is not a standard and so the syntax they use may not work with what dotnet supports.

Hmm, so it wouldnt be possible to intergrate?

It wont not be feasible. There is support for regex in the Advanced Blocking app but it requires using the syntax that works with .NET runtime. So, it will be required to convert the syntax if its not compatible to be usable.

[ShreyasZare]

Can you please try my setup? Whitelisting is still not working for me.

My black/allowlists are as follows: (copied directly)

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/dyndns.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt

Domains used to test: www.googleadservices.com

Should be whitelisted by https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt Blacklisted by https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate.txt

Ultimate only blocks *googleadservices.com But the whitelist should allow www.googleadservices.com

This issue is due to how the blocking implementation works in the DNS server's built in blocking feature. Will get that changed so that this case that you mentioned will work. This change will be available in the next service update this is planned to be released soon.

If you use this with the Advanced Blocking app then it would work as expected.

[ShreyasZare]

Would be great if you could have support for both, since they are very similar. Also, what about ublock origin? Is that the standard for adblock syntax? He also has a ublock version, https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt

Again, works the same

It would be best if you can use the supported block list file format from the project. Adding support for all types of formats will be difficult to maintain since they keep changing and are not documented/partially documented in many cases.

The uBlock list you mention is problem since its made for the ublock browser plugin. The DNS server wont be able to use the domain= part since it cannot know on which website you already are on. If the block list is assigning a different meaning to the domain= syntax then that is unfortunate decision to do so by the maintainer.

[devipasigner]

Would be great if you could have support for both, since they are very similar. Also, what about ublock origin? Is that the standard for adblock syntax? He also has a ublock version, https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt Again, works the same

It would be best if you can use the supported block list file format from the project. Adding support for all types of formats will be difficult to maintain since they keep changing and are not documented/partially documented in many cases.

The uBlock list you mention is problem since its made for the ublock browser plugin. The DNS server wont be able to use the domain= part since it cannot know on which website you already are on. If the block list is assigning a different meaning to the domain= syntax then that is unfortunate decision to do so by the maintainer.

What’s the proper way to implement this then? Adguard home uses $denyallow, unlock uses $doc_domain

It would be great if you could implement one of them, since a lot of lists are using the Adblock format these days and some list maintainer are providing special options for dns like adguard home where as pihole and other softwares are left behind

[devipasigner]

Would be great if you could have support for both, since they are very similar. Also, what about ublock origin? Is that the standard for adblock syntax? He also has a ublock version, https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/spam-tlds-ublock.txt Again, works the same

It would be best if you can use the supported block list file format from the project. Adding support for all types of formats will be difficult to maintain since they keep changing and are not documented/partially documented in many cases.

The uBlock list you mention is problem since its made for the ublock browser plugin. The DNS server wont be able to use the domain= part since it cannot know on which website you already are on. If the block list is assigning a different meaning to the domain= syntax then that is unfortunate decision to do so by the maintainer.

👍 thanks for your help

[ShreyasZare]

What’s the proper way to implement this then? Adguard home uses $denyallow, unlock uses $doc_domain

The problem is that each app uses Adblock like format and then extends it based on their features. A lot of options are not clearly documented and thus creates problems to maintain such an implementation.

It would be great if you could implement one of them, since a lot of lists are using the Adblock format these days and some list maintainer are providing special options for dns like adguard home where as pihole and other softwares are left behind

I would recommend that you use other formats when they are available (like wildcard list, hosts, or domains) or use lists that have standard Adblock format. Any other format like uBlock or Adguard will be difficult to support since they are designed for the specific product usage/features.

[ShreyasZare]

Technitium DNS Server v11.0.1 is now available that fixes the allow list issue. Do update and let me know your feedback.