search

flipkart-incubator / databuilderframework

A data driven execution engine
33 stars 29 forks source link

MultiThreadedExecutor runs executor even when there is one builder in the respective rank. #9

Closed gokulvanan closed 8 years ago

gokulvanan commented 8 years ago

Parallelization with overhead of context switch would be beneficial if there are more than one builders in the same rank.

santanusinha commented 8 years ago

Unless there is a demonstrated gain in performance due to this, it would introduce a lot of unnecessary complexity.

gokulvanan commented 8 years ago

agreed will try out a test case and post some numbers here to take this forward.

gokulvanan commented 8 years ago

Below is the output from running a new test case in "issue_9" branch of my fork. se - simpleExecutor me- multiThreadedExecutor ome - omptimizedMultiThreadedExecutor - avoid using thread pool if size among the same rank is 1.

From below stats we can se that when concurrency is more that builder thread pool size "ome" gives considerable performance gains compared "me" - Again this is assuming the dataflow execution graph has many single nodes in between parallel nodes in topology.

Running com.flipkart.databuilderframework.cmplxscenariotest.ContextSwitchOverHeadTest SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. running unbounded queue test with builder size 150 and concurrecy 100 se 3590 me 2401 me run: context switches over threshold count 0, max latent switch: 0, rejections 0 ome 2091 ome run: context switches over threshold count 0, max latent switch: 0, rejections 0 testWhenConccurencyIsMoreThanBuilderSize running unbounded queue test with builder size 50 and concurrecy 150 se 2518 me 6874 me run: context switches over threshold count 15112, max latent switch: 263, rejections 0 ome 3993 ome run: context switches over threshold count 10120, max latent switch: 288, rejections 0 testWhenConccurencyIsSameAsBuilderSize running unbounded queue test with builder size 150 and concurrecy 150 se 2556 me 2362 me run: context switches over threshold count 342, max latent switch: 73, rejections 0 ome 1581 ome run: context switches over threshold count 196, max latent switch: 80, rejections 0 Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.18 sec