Cosbench 0.4.2.c4 gave data integrity error at the end of the read tests

[jianghaitao]

Cosbench v0.4.2.c4 against Ceph (Hammer) / radosgw / HAproxy's HTTP end point. All seems fine except that at the end of the read test, it failed the task and gave following errors

com.amazonaws.AmazonClientException: Unable to verify integrity of data download.  Client calculated content hash didn't match hash calculated by Amazon S3.  The data may be corrupt.
        at com.amazonaws.services.s3.internal.DigestValidationInputStream.validateMD5Digest(DigestValidationInputStream.java:79)
        at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:61)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
        at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:147)
        at com.intel.cosbench.driver.operator.Reader.copyLarge(Reader.java:120)
        at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:92)
        at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
        at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
        at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
        at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
        at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

[ywang19]

could you paste the workload config file?

[ywang19]

here is a thread about the exception from AWS: https://github.com/awslabs/aws-sdk-android-samples/issues/62

[ywang19]

about setting the property:

1. stop all cosbench processes (controller, drivers)
2. edit cosbench-start.sh, locate the java launching line, and add "-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true"
3. restart cosbench processes.

[jianghaitao]

@ywang19 Thank you very much! -D option helped. Tests completed successfully.

[srujun]

I was able to reproduce this issue using this conf file using version 0.4.2.c4: ceph-s3-config.txt

If I set it to just one worker running the read operation, it succeeds with 100% success rate. But any additional workers results in a less than 5% success rate and the eventual failure as mentioned above. However, upon setting hashCheck=true in the read operation, the workload does finish successfully. And all of this is without setting the disableGetObjectMD5Validation=true property.

My question is, if I set the javac property and also keep the hashCheck=true setting for the any operations, will it still check for data integrity or not?

[ericychoi]

Just wanted to chime in that I am seeing the exact same issue; @srujun any fix you found for this? I was wondering if this is an issue in RadosGW / Ceph or cosbench..

[jamie-liu]

@ericychoi I think it's an issue in aws-java-sdk and it has been fixed in 1.5.0 release. https://forums.aws.amazon.com/thread.jspa?threadID=126007

@ywang19 What's the aws-java-sdk version used in Cosbench 0.4.2.c4?

[hrchu]

1.10.76 in master branch, but seems that it is not yet released.

https://github.com/intel-cloud/cosbench/commit/d7a4d7c77a3a9ca49e246eb401c4b41d3506a2d5

[CMADHANKUMAR]

about setting the property:

1. stop all cosbench processes (controller, drivers)
2. edit cosbench-start.sh, locate the java launching line, and add "-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true"
3. restart cosbench processes.

Right one thankyou