search

open-io / oio-swift

Openio flavor of the Openstack Swift Gateway and middlewares, allowing access to OpenIO SDS via an Openstack Swift gateway.
https://www.openio.io
Apache License 2.0
23 stars 21 forks source link

Code retour 204 during put (then put fail) when file size is greater than multipart size #55

Closed sebastienlapierre closed 6 years ago

sebastienlapierre commented 7 years ago

oio-version: 4.0.0-1 oio-swift: 1.0.0-0.b4

packages:

python2-swiftclient-3.3.0-1.el7.noarch
openio-sds-swift-1.0.0-0.b4.el7.oio.noarch
openio-sds-swift-plugin-swift3-1.12.0.b0-0.el7.oio.noarch
python-swift-2.13.0-1.el7.noarch
openstack-swift-proxy-2.13.0-1.el7.noarch
openstack-swift-container-2.13.0-1.el7.noarch

client: s3cmd-1.6.1-1.el7.noarch

Each time we try to put a file with a greater size than the default mutlipart size (15M) we encountered this error with s3cmd:

DEBUG: s3cmd version 1.6.1
DEBUG: ConfigParser: Reading file 'config_test'
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: signature_v2->True
DEBUG: ConfigParser: access_key->pr...28_chars...y
DEBUG: ConfigParser: secret_key->xq...13_chars...f
DEBUG: ConfigParser: host_base->xxxxxxxx
DEBUG: ConfigParser: host_bucket->xxxxxxxxxx
DEBUG: ConfigParser: socket_timeout->100
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'put' using UTF-8
DEBUG: Unicodising 'vingtMBfile' using UTF-8
DEBUG: Unicodising 's3://openiotest/vingtMBfile' using UTF-8
DEBUG: Command: put
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
INFO: Compiling list of local files...
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: Unicodising '' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: Applying --exclude/--include
DEBUG: CHECK: vingtMBfile
DEBUG: PASS: u'vingtMBfile'
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: doing file I/O to read md5 of vingtMBfile
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
INFO: Summary: 1 local files to upload
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': 'uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1501663079/atime:1501663132/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/ctime:1501663079'}
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: DeUnicodising u'vingtMBfile' using UTF-8
DEBUG: String 'vingtMBfile' encoded to 'vingtMBfile'
DEBUG: CreateRequest: resource[uri]=/vingtMBfile?uploads
DEBUG: Using signature v2
DEBUG: SignHeaders: 'POST\n\napplication/octet-stream\n\nx-amz-date:Wed, 02 Aug 2017 08:53:03 +0000\nx-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1501663079/atime:1501663132/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/ctime:1501663079\nx-amz-storage-class:STANDARD\n/openiotest/vingtMBfile?uploads'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(openiotest): xxxxxxxxxxxxx
DEBUG: ConnMan.get(): creating new connection: https://xxxxxxxxxxxxxx
DEBUG: Using ca_certs_file None
DEBUG: httplib.HTTPSConnection() has both context and check_hostname
DEBUG: non-proxied HTTPSConnection(xxxxxxxxxxxxxxx)
DEBUG: format_uri(): /openiotest/vingtMBfile?uploads
DEBUG: Sending request method_string='POST', uri='/openiotest/vingtMBfile?uploads', headers={'x-amz-meta-s3cmd-attrs': 'uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1501663079/atime:1501663132/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/ctime:1501663079', 'content-type': 'application/octet-stream', 'Authorization': 'AWS XXXX:XXXXX:XXXXX=', 'x-amz-date': 'Wed, 02 Aug 2017 08:53:03 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0 bytes)
DEBUG: Response: {'status': 500, 'headers': {'transfer-encoding': 'chunked', 'server': 'nginx/1.13.3', 'x-amz-id-2': 'tx016597d721b04644ba766-00598192e8', 'connection': 'keep-alive', 'x-amz-request-id': 'tx016597d721b04644ba766-00598192e8', 'x-trans-id': 'tx016597d721b04644ba766-00598192e8', 'date': 'Wed, 02 Aug 2017 08:53:03 GMT', 'content-type': 'application/xml', 'x-openstack-request-id': 'tx016597d721b04644ba766-00598192e8'}, 'reason': 'Internal Server Error', 'data': "<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>InternalError</Code><Message>unexpected status code 204</Message><RequestId>tx016597d721b04644ba766-00598192e8</RequestId></Error>"}
DEBUG: ConnMan.put(): connection put back to pool (https://xxxxxxxxxxxx)
DEBUG: S3Error: 500 (Internal Server Error)
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: server: nginx/1.13.3
DEBUG: HttpHeader: x-amz-id-2: tx016597d721b04644ba766-00598192e8
DEBUG: HttpHeader: connection: keep-alive
DEBUG: HttpHeader: x-amz-request-id: tx016597d721b04644ba766-00598192e8
DEBUG: HttpHeader: x-trans-id: tx016597d721b04644ba766-00598192e8
DEBUG: HttpHeader: date: Wed, 02 Aug 2017 08:53:03 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: x-openstack-request-id: tx016597d721b04644ba766-00598192e8
DEBUG: ErrorXML: Code: 'InternalError'
DEBUG: ErrorXML: Message: 'unexpected status code 204'
DEBUG: ErrorXML: RequestId: 'tx016597d721b04644ba766-00598192e8'
WARNING: Retrying failed request: /vingtMBfile?uploads
WARNING: 500 (InternalError): unexpected status code 204
WARNING: Waiting 3 sec...

Related swift error:

2017-08-02T12:11:03.632649+03:00 oio1 OIO,OPENIO,oioswift,1: err  500 Internal Server Error: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 119, in __call__#012    resp = self.handle_request(req)#012  File "/usr/lib/python2.7/site-packages/swift3/middleware.py", line 150, in handle_request#012    res = handler(req)#012  File "/usr/lib/python2.7/site-packages/swift3/controllers/base.py", line 59, in wrapped#012    return func(self, req)#012  File "/usr/lib/python2.7/site-packages/swift3/controllers/base.py", line 71, in check_container#012    return func(self, req)#012  File "/usr/lib/python2.7/site-packages/swift3/controllers/multi_upload.py", line 340, in POST#012    req.get_response(self.app, 'PUT', container, '')#012  File "/usr/lib/python2.7/site-packages/swift3/request.py", line 1353, in get_response#012    headers, body, query)#012  File "/usr/lib/python2.7/site-packages/swift3/request.py", line 1338, in _get_response#012    raise InternalError('unexpected status code %d' % status)#012InternalError: 500 Internal Server Error (txn: txcee6b435347f48d687c4f-0059819727) (client_ip: 149.202.213.199)
2017-08-02T12:11:03.634296+03:00 oio1 OIO,OPENIO,oioswift,1: info  149.202.213.199 149.202.213.199 02/Aug/2017/09/11/03 POST /openiotest/vingtMBfile%3Fuploads HTTP/1.0 500 - - - - 182 - txcee6b435347f48d687c4f-0059819727 - 0.0364 - - 1501665063.596266985 1501665063.632709026 -

The workaround is to force the multipart size greater than the filesize. (option: --multipart-chunk-size=xx)

kamel-rahim commented 7 years ago

This is an error of the middleware swift3. It should be fix by swift3#7.