search

intel-cloud / cosbench

a benchmark tool for cloud object storage service
Other
573 stars 242 forks source link

Fixes for issues #265 and #264 #266

Closed zoran-rajic closed 9 years ago

zoran-rajic commented 9 years ago

Fixes for issues:

zoran-rajic commented 9 years ago

Hello Yaguang,

I was able to reproduce the issue, using 0.4.2.c1 (version 3ae8e70, tag:v0.4.2.c1, Apr 8 2015), by using "concurrency = 2" and running 2 workloads in parallel.

Test-setup:

[from system.log @ controller-host -- comments annotated with "[Zoran..]" ] 2015-04-13 15:46:29,345 [INFO] [WorkloadProcessor] - ============================================ 2015-04-13 15:46:29,346 [INFO] [WorkloadProcessor] - successfully ran stage s1-init 2015-04-13 15:46:29,346 [INFO] [WorkloadProcessor] - begin to run stage s2-prepare 2015-04-13 15:46:29,347 [INFO] [WorkloadProcessor] - ============================================ 2015-04-13 15:46:29,347 [INFO] [WorkloadProcessor] - START WORK: prepare 2015-04-13 15:46:28,004 [INFO] [AbstractCommandTasklet] - time drift between controller and driver-driver-81 is 1 mSec 2015-04-13 15:46:28,005 [INFO] [AbstractCommandTasklet] - time drift between controller and driver-driver-82 is 1 mSec 2015-04-13 15:46:27,000 [INFO] [AbstractCommandTasklet] - time drift between controller and driver-driver-80 is -1006 mSec

[Zoran: the controller and driver-80 are on the same physical host (but different JVMs), and yet, a significant time-drift has been detected]

2015-04-13 15:46:27,001 [WARN] [AbstractCommandTasklet] - time drift is still longer than tolerable time drift 300 mSec after 3 times of synchronization 2015-04-13 15:46:27,001 [INFO] [StageRunner] - successfully booted all tasks in stage s2-prepare 2015-04-13 15:46:27,217 [INFO] [StageRunner] - successfully submitted all tasks in stage s2-prepare 2015-04-13 15:46:27,222 [INFO] [COSBDriverService] - handler=M6B4F65C85F 2015-04-13 15:46:27,232 [INFO] [MissionHandler] - mission M6B4F65C85F has been authed successfully 2015-04-13 15:46:27,425 [INFO] [StageRunner] - successfully authenticated all tasks in stage s2-prepare 2015-04-13 15:46:27,631 [INFO] [StageRunner] - successfully launched all tasks in stage s2-prepare 2015-04-13 15:46:28,208 [WARN] [OperatorContext] - heavy atomic op overhead detected: 2.88261ms 2015-04-13 15:46:28,534 [INFO] [WorkloadProcessor] - begin to process workload w8 2015-04-13 15:46:28,535 [INFO] [WorkloadProcessor] - begin to run stage s1-init 2015-04-13 15:46:28,535 [INFO] [WorkloadProcessor] - ============================================ 2015-04-13 15:46:28,536 [INFO] [WorkloadProcessor] - START WORK: init 2015-04-13 15:46:28,005 [ERROR] [AbstractAgent] - unexpected exception java.lang.ArrayIndexOutOfBoundsException: -51 at com.intel.cosbench.bench.Counter.doAdd(Counter.java:65) at com.intel.cosbench.driver.model.OperatorContext.doAddSample(OperatorContext.java:76) at com.intel.cosbench.driver.model.OperatorContext.addSample(OperatorContext.java:70) at com.intel.cosbench.driver.agent.WorkAgent.onSampleCreated(WorkAgent.java:211) at com.intel.cosbench.driver.operator.Preparer.operate(Preparer.java:99) at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76) at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197) at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177) at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [Zoran: repeats 2 times] 2015-04-13 15:46:28,009 [INFO] [AbstractCommandTasklet] - time drift between controller and driver-driver-81 is 2 mSec 2015-04-13 15:46:28,008 [ERROR] [AbstractAgent] - unexpected exception java.lang.ArrayIndexOutOfBoundsException: -50 at com.intel.cosbench.bench.Counter.doAdd(Counter.java:65) at com.intel.cosbench.driver.model.OperatorContext.doAddSample(OperatorContext.java:76) at com.intel.cosbench.driver.model.OperatorContext.addSample(OperatorContext.java:70) at com.intel.cosbench.driver.agent.WorkAgent.onSampleCreated(WorkAgent.java:211) at com.intel.cosbench.driver.operator.Preparer.operate(Preparer.java:99) at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76) at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197) at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177) at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-04-13 15:46:28,005 [ERROR] [AbstractAgent] - unexpected exception java.lang.ArrayIndexOutOfBoundsException: -47 at com.intel.cosbench.bench.Counter.doAdd(Counter.java:65) at com.intel.cosbench.driver.model.OperatorContext.doAddSample(OperatorContext.java:76) at com.intel.cosbench.driver.model.OperatorContext.addSample(OperatorContext.java:70) at com.intel.cosbench.driver.agent.WorkAgent.onSampleCreated(WorkAgent.java:211) at com.intel.cosbench.driver.operator.Preparer.operate(Preparer.java:99) at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76) at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197) at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177) at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [Zoran: etc...]

I have tested also with the default "concurrency=1", and that seems to be working OK.

Best regards, Zoran

ywang19 commented 9 years ago

Zoran,

could you withdraw the fix for #264? It's fine to merge fixes for #265, for #264, it's still in question, you could create another PR for it later on.

-yaguang

ywang19 commented 9 years ago

applied the fixes for #265 (S3: Need configurable max_connections for AmazonS3Client) on 0.4.2.0 branch.