search

divyang4481 / abot

Automatically exported from code.google.com/p/abot
Apache License 2.0
0 stars 0 forks source link

Test TaskThreadManager to see if it can become the default IThreadManager #98

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Test against ManualThreadManager for (Speed/cpu/memory)

Original issue reported on code.google.com by sjdir...@gmail.com on 2 May 2013 at 10:58

GoogleCodeExporter commented 9 years ago 
The following unit tests had to have their wait times bumped up from 20ms to 
80ms to make test pass for TaskThreadManager. This tells me that the 
ManualThreadManager is still much faster getting work done. Just need to test 
what it costs in memory and cpu to create new threads all the time.

ThreadManagerTest.DoWork_WorkItemsEqualToThreads_WorkIsCompletedAsync()
ThreadManagerTest.DoWork_MoreWorkThenThreads_WorkIsCompletedAsync()

Original comment by sjdir...@gmail.com on 4 May 2013 at 12:12

GoogleCodeExporter commented 9 years ago 
If this becomes the default pass in the _crawlContext.CancellationSourceToken 
into the TaskThreadManager's constructor to be able to cancel everything in 
unison.

Original comment by sjdir...@gmail.com on 4 May 2013 at 12:16

GoogleCodeExporter commented 9 years ago 
While observing the running of the integration tests I noticed that the 
ManualThreadManager would peek its cpu usage at 100% the last 5 seconds before 
completing the integration tests. The TaskThreadManager peeks at about 68%. 

Both implementations ran the integration tests in about 35-36 seconds 
consistently. 

Looks like the ManualThread manager still is a little more performant than 
TaskThreadManager but does use more cpu. Haven't made a decision on which 
should be default. I suspect the TaskThreadManager will not work as well as 
ManaualThreadManager when many instances of PoliteWebCrawler have been created. 
Seems like thread pool threads take a while to get scheduled.

Original comment by sjdir...@gmail.com on 4 May 2013 at 4:47

GoogleCodeExporter commented 9 years ago 
Adding to my last comment...

I think the TaskThreadManager has proven to be stable and safe but will need to 
do the multiple instance test to see if one or the other handles it better.

Original comment by sjdir...@gmail.com on 4 May 2013 at 4:48

GoogleCodeExporter commented 9 years ago 
This issue was closed by revision r534.

Original comment by sjdir...@gmail.com on 27 Jun 2013 at 6:12