Memory leakage

[GoogleCodeExporter]

If I chose MaxConcurrentThreads >1 I experiance linear psychical memory growth. 
However on MaxConcurrentThreads =1 memory usage is stable.

For example (see app.config attached)
CrawlConfiguration crawlConfig = 
AbotConfigurationSectionHandler.LoadFromXml().Convert();
crawlConfig.MaxConcurrentThreads = 2;
crawlConfig.IsExternalPageLinksCrawlingEnabled = true;
crawlConfig.IsExternalPageCrawlingEnabled = false;
crawlConfig.MaxRobotsDotTextCrawlDelayInSeconds = 5;

crawler = new PoliteWebCrawler(crawlConfig, null, null, null, null, null, null, 
null, null);

Maybe it's something with BlockingCollection<Action> _actionsToExecute in 
Abot.Crawl not being disposed. Or event subscription.

Well, did not manage to resolve this, maybe have some ideas?

Original issue reported on code.google.com by sandis.a...@gmail.com on 9 May 2013 at 8:22

Attachments:

• Capture.PNG
• App.config
• Capture1.PNG

[GoogleCodeExporter]

Thanks for creating a bug! I appreciate you taking the time to do so.

I assume you are using v1.1 since ProducerConsumerThreadManager was the default 
IThreadManager used in that version.

Abot v1.1.1 (current stable release) no longer uses the 
ProducerConsumerThreadManager (shown as the problem in your profiler result) by 
default due to some of the issues that you point out along with a few others. 

In Abot v1.2 (currently under development) the ProducerConsumerThreadManager 
impl was completely removed. The ManualThreadManager is still the default 
IThreadManager but there is also a chance that the TaskThreadManager may become 
the new default. Both of these implementations have been updated to avoid a 
busy wait.

Would you be willing to run your tests on the following?

-Abot v1.1.1 (current stable release)
new PoliteWebCrawler()

-Abot v1.2 (currently under development)
new PoliteWebCrawler()

-Abot v1.2 (currently under development)using the TaskThreadManager instead...
new PoliteWebCrawler(null, null, new TaskThreadManager(MaxNumberOfThreads), 
null, null, null, null, null, null);

I have attached the latest v1.2 (currently under development) to this ticket.

Original comment by sjdir...@gmail.com on 10 May 2013 at 5:41

Attachments:

• v1.2.zip

[GoogleCodeExporter]

Thanks for reply.

You were correct about me using version v1.1 and ProducerConsumerThreadManager.

Problem when MaxConcurrentThreads >1 did not recur in v1.1.1 and v1.2

Original comment by sandis.a...@gmail.com on 14 May 2013 at 8:26

[GoogleCodeExporter]

Marking this as invalid since the bug was specific to an older version. Thanks 
for checking back in so we can be sure before closing this issue!!

Original comment by sjdir...@gmail.com on 14 May 2013 at 8:50

• Changed state: Invalid