search

intel-cloud / cosbench

a benchmark tool for cloud object storage service
Other
573 stars 242 forks source link

S3 storag type: Encountered an exception and couldn't reset the stream to retry #329

Open ksingh7 opened 8 years ago

ksingh7 commented 8 years ago

Hello Guys

I am facing a strange problem with S3 storage type. I am creating 40 containers each with 256 objects.

The problem is while using S3 storage type COSbench is not creating all 256 objects inside first containers, i.e. in container 41 it creates less objects < 256 , however for rest all 39 containers its creating 256 objects each.

And the strange thing is that if i use SWIFT storage type , COSBench creates all objects in each container and everything works fine.

So looks like something wrong with S3 Storage driver type ?

Here is my workload file

<?xml version="1.0" encoding="UTF-8"?>
 <workload name="1-rgw-1-client-s3" description="1-rgw-1-client-s3" config="">
    <storage type="s3" config="accesskey=4QDZVBFX62G3O17YHWBB;secretkey=I4ZTGXtxrxKJVtA5AIMmTvn2xMAtsivfjd8pLavI;timeout=600000;endpoint=http://10.5.13.140:8080" />
    <workflow config="">
        <!-- Create containers -->
        <workstage name="init">
            <work name="Initialize all containers" type="init" workers="8" config="containers=r(41,80)"/>
        </workstage>

        <!-- Create Objects -->
        <workstage name="prepare-objects">
            <work name="prepare-all-objects" type="prepare" workers="32" driver="client1" interval="5" config="containers=r(41,50);objects=r(1,256);sizes=c(1)MB" />
            <work name="prepare-all-objects" type="prepare" workers="32" driver="client1" interval="5" config="containers=r(51,60);objects=r(1,256);sizes=c(32)MB" />
            <work name="prepare-all-objects" type="prepare" workers="32" driver="client1" interval="5" config="containers=r(61,70);objects=r(1,256);sizes=c(64)MB" />
            <work name="prepare-all-objects" type="prepare" workers="32" driver="client1" interval="5" config="containers=r(71,80);objects=r(1,256);sizes=c(128)MB" />
        </workstage>
    </workflow>
</workload>

Here are the mission logs notice containers41 which is failing 

2016-07-19 16:27:32,655 [INFO] [NoneStorage] - performing PUT at /mycontainers49/myobjects232
2016-07-19 16:27:33,031 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects225
2016-07-19 16:27:33,183 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects226
2016-07-19 16:27:33,495 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects227
2016-07-19 16:27:33,642 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects228
2016-07-19 16:27:33,753 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects229
2016-07-19 16:27:34,837 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects230
2016-07-19 16:27:34,975 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects231
2016-07-19 16:27:35,091 [INFO] [NoneStorage] - performing PUT at /mycontainers50/myobjects232
2016-07-19 16:27:43,524 [WARN] [S3Storage] - below exception encountered when creating object myobjects201 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,526 [WARN] [S3Storage] - below exception encountered when creating object myobjects153 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,527 [WARN] [S3Storage] - below exception encountered when creating object myobjects57 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,527 [WARN] [S3Storage] - below exception encountered when creating object myobjects129 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,527 [WARN] [S3Storage] - below exception encountered when creating object myobjects169 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,529 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects202
2016-07-19 16:27:43,529 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects154
2016-07-19 16:27:43,530 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects170
2016-07-19 16:27:43,530 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects58
2016-07-19 16:27:43,530 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects130
2016-07-19 16:27:43,533 [WARN] [S3Storage] - below exception encountered when creating object myobjects145 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,534 [WARN] [S3Storage] - below exception encountered when creating object myobjects33 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,535 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects146
2016-07-19 16:27:43,536 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects34
2016-07-19 16:27:43,537 [WARN] [S3Storage] - below exception encountered when creating object myobjects41 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,538 [WARN] [S3Storage] - below exception encountered when creating object myobjects161 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,539 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects42
2016-07-19 16:27:43,540 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects162
2016-07-19 16:27:43,550 [WARN] [S3Storage] - below exception encountered when creating object myobjects65 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,551 [WARN] [S3Storage] - below exception encountered when creating object myobjects193 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,552 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects66
2016-07-19 16:27:43,553 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects194
2016-07-19 16:27:43,557 [WARN] [S3Storage] - below exception encountered when creating object myobjects73 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,559 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects74
2016-07-19 16:27:43,560 [WARN] [S3Storage] - below exception encountered when creating object myobjects17 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,561 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects18
2016-07-19 16:27:43,564 [WARN] [S3Storage] - below exception encountered when creating object myobjects9 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,566 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects10
2016-07-19 16:27:43,569 [WARN] [S3Storage] - below exception encountered when creating object myobjects25 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,571 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects26
2016-07-19 16:27:43,574 [WARN] [S3Storage] - below exception encountered when creating object myobjects81 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,576 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects82
2016-07-19 16:27:43,581 [WARN] [S3Storage] - below exception encountered when creating object myobjects177 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,583 [WARN] [S3Storage] - below exception encountered when creating object myobjects241 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,583 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects178
2016-07-19 16:27:43,585 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects242
2016-07-19 16:27:43,585 [WARN] [S3Storage] - below exception encountered when creating object myobjects49 at mycontainers41: Encountered an exception and couldn't reset the stream to retry
2016-07-19 16:27:43,586 [INFO] [NoneStorage] - performing PUT at /mycontainers41/myobjects50
ksingh7 commented 8 years ago

And the debug logs do not show any JAVA exception

lucaspeng12138 commented 1 year ago

@ksingh7 I'm facing the same problem, any suggestion please?