Open tschimbr opened 7 years ago
Without having double checked it, I suspect that it might be, because I'm storing a little bit of data about each visited page in memory (e.g. the hash value of its content). It might be that this, when accumulated, turns out to be a lot of data. I could solve that problem by bucketing all the information into domain-buckets (e.g. "www.eonum.ch/about" etc. will be in the bucket "www.eonum.ch") and then only holding the currently used bucket in memory and storing everything else on disk. I think this will reduce the memory usage but increase disk access (which shouldn't be a huge problem).
I just remembered that I'm caching requests as well. I have to check if I'm already deleting them after they are used or not. If I'm not, that'll be the main problem.
The current memory usage tends to get very high after some time. Ist there anything that could be garbage collected?