search

intel-cloud / cosbench

a benchmark tool for cloud object storage service
Other
573 stars 242 forks source link

S3 md5 hash fails #363

Open MostafaNabi opened 7 years ago

MostafaNabi commented 7 years ago

Hello,

I've been getting md5 errors where the hashes do not match. See the error stack below.

com.amazonaws.AmazonClientException: Unable to verify integrity of data download. Client calculated content hash didn't match hash calculated by Amazon S3. The data may be corrupt. at com.amazonaws.services.s3.internal.DigestValidationInputStream.validateMD5Digest(DigestValidationInputStream.java:79) at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:61) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155) at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:147) at com.intel.cosbench.driver.operator.Reader.copyLarge(Reader.java:120) at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:92) at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69) at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76) at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197) at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177) at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44) at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) at java.lang.Thread.run(Thread.java:748)

I have tried on a Scality S3 server and a CEPH cluster and the error was consistent. I saw this thread on the github here: https://github.com/intel-cloud/cosbench/issues/320 and the issue seems to be same as mine. I also noticed the more workers I have the lower my succes ratio, with a 100% success ratio for one worker.

I turned off the md5 checks and got a 100% success ratio. My question is does anyone know why the checks fail?

My config file:

<?xml version="1.0" encoding="UTF-8" ?>

merrett commented 6 years ago

Same issue here.

MostafaNabi commented 6 years ago

@merrett Did you find a solution to the problem?

merrett commented 6 years ago

I just had to turn the MD5 off.

edit cosbench-start.sh

find the java start line and add "-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true"

so you have:

/usr/bin/nohup java -Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true -Dcosbench.tomcat.config=$TOMCAT_CONFIG -server -cp main/* org.eclipse.equinox.launcher.Main -configuration $OSGI_CONFIG -console $OSGI_CONSOLE_PORT 1 > $BOOT_LOG 2>&1 &