search

TechnitiumSoftware / DnsServer

Technitium DNS Server
https://technitium.com/dns/
GNU General Public License v3.0
4.25k stars 418 forks source link

Adblocking lists and ram usage #111

Closed ghost closed 4 years ago

ghost commented 4 years ago

Ram usage can drastically grow up using big AdBlocking hosts like https://block.energized.pro/ultimate/formats/hosts.txt. I've got 4 gigs of ram and it tried to use all of it.

Previously i have been using Unbound and ram usage was no problem with blocklists like this. Even adding this list to ublock generate smaller ram usage (like 200-300MB more).

ShreyasZare commented 4 years ago

Thanks for the feedback.

DNS Server will use up a lot of memory that is available on the system while loading block lists. This is since C# is garbage collected language and there is no direct control to ask it to free unused memory. After a while, the garbage collector (GC) would free up the space and the RAM usage will go down in few 100s of MBs as per the actual size of the data that is loaded in the memory. It may take 10-15 mins or even an hour for this process to occur depending on the GC algorithm.

This is not an issue with Unbound since its written in C and they can immediately free up memory that was used while loading the list.

ghost commented 4 years ago

Thanks for explanation :) That explains why ram usage stays even after you remove blocking list

So it is a no go with weaker systems like mine if i want to do a DNS Sinkhole. 10-15 mins is a long time where you can't do anything on the system.

Is there no way to force GC? I've seen browser have a simple button to do this when you monitor memory usage.

ShreyasZare commented 4 years ago

Thanks for explanation :) That explains why ram usage stays even after you remove blocking list

So it is a no go with weaker systems like mine if i want to do a DNS Sinkhole. 10-15 mins is a long time where you can't do anything on the system.

Is there no way to force GC? I've seen browser have a simple button to do this when you monitor memory usage.

There is GC.Collect(); method to do that which I had tried to fix this issue a while back but it practically does not work and GC takes its own time to free memory.

There are many string manipulations that are done while parsing the block list file which is causing this since strings are immutable and thus each operation allocates new memory. There could be ways to optimize this by reducing string usage which can reduce memory usage a bit. Will try to do some tests on it and see if anything can be improved in next release.

ghost commented 4 years ago

Ok, thanks :)

It really is GC problem. After enabling Steven Host list ram usage has been sitting on 150MB, after like an hour dropped to normal (15-20MB).

Thanks for help. Waiting for what will you come up to improve this.