search

TechnitiumSoftware / DnsServer

Technitium DNS Server
https://technitium.com/dns/
GNU General Public License v3.0
4.17k stars 409 forks source link

Block lists duplicate/duplicates, deduplication #1003

Open gpz1100 opened 3 weeks ago

gpz1100 commented 3 weeks ago

How does Technitium handle block lists containing duplicates?

That is, say list A and B both contain the same domain, is there any sort of deduplication performed for the final master list?

ShreyasZare commented 3 weeks ago

Thanks for asking. Yes, the blocking feature that is built into the DNS server does deduplication to save memory. The Advanced Blocking app however does not do that so as to allow the same block list that is loaded into memory to be reused by multiple groups to save memory when there are many groups with overlapping block lists.

gpz1100 commented 3 weeks ago

Thank you for your reply. On the same topic, with multiple forwarders configured, does the DNS use a parallel, round robin,or other type of lookup? Is this configurable?

ShreyasZare commented 3 weeks ago

Thank you for your reply. On the same topic, with multiple forwarders configured, does the DNS use a parallel, round robin,or other type of lookup? Is this configurable?

Currently, the DNS server will randomly select forwarders based on the Forwarder Concurrency that is configured and use the fastest response. If none of those selected forwarders answer then it will randomly select from the remaining ones and repeat.

The next update changes this such that instead of random selection, the latency of the upstream is recorded and used to select the fastest ones and then concurrently query them.

chmichael commented 3 weeks ago

That's awesome combined with sequental order it will always use the fastest forwarder!

btw it handles the wildcard black lists the same ? eg. the 1st list has: subdomain.mydomain.com the 2nd list has: *.mydomain.com

I guess it discards the item from the 1st list and keeps the 2nd right ?

ShreyasZare commented 3 weeks ago

btw it handles the wildcard black lists the same ? eg. the 1st list has: subdomain.mydomain.com the 2nd list has: *.mydomain.com

I guess it discards the item from the 1st list and keeps the 2nd right ?

The deduplication process does exact match. This level of check is not done since it will increase loading time significantly and also increase memory usage.