search

nspcc-dev / neofs-s3-gw

NeoFS S3 Protocol Gateway
GNU Affero General Public License v3.0
7 stars 15 forks source link

UploadPart fails with 500 on the first part of the multipart object #969

Closed evgeniiz321 closed 2 weeks ago

evgeniiz321 commented 2 months ago
  1. Create multipart upload: 
    
COMMAND: 'aws --no-verify-ssl --no-paginate s3api create-multipart-upload --bucket 5676e6e6-98c3-4d37-b31b-10dd54fca1d9 --key 183b408e-13c3-4a7b-ad30-894a1165faea --endpoint-url https://localhost:56919'
OUTPUT:
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
"Bucket": "5676e6e6-98c3-4d37-b31b-10dd54fca1d9",
"Key": "183b408e-13c3-4a7b-ad30-894a1165faea",
"UploadId": "CPHxp7nGwfxEdyHyyEUK8Fvo8joHfHmi7EprTTn56xjZ"
}

RC: 0 Start / End / Elapsed 20:58:49.367753 / 20:58:50.026036 / 0:00:00.658283

2. Try to upload first part:

COMMAND: 'aws --no-verify-ssl --no-paginate s3api upload-part --bucket 5676e6e6-98c3-4d37-b31b-10dd54fca1d9 --key 183b408e-13c3-4a7b-ad30-894a1165faea --upload-id CPHxp7nGwfxEdyHyyEUK8Fvo8joHfHmi7EprTTn56xjZ --part-number 1 --body /home/runner/work/neofs-testcases/neofs-testcases/neofs-testcases/TemporaryDir/TestFilesDir/183b408e-13c3-4a7b-ad30-894a1165faea_part_1 --endpoint-url https://localhost:56919' OUTPUT: urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 0): We encountered an internal error, please try again. RC: 254 Start / End / Elapsed 20:58:50.652536 / 20:58:51.629319 / 0:00:00.976783

In s3 gw logs I see errors like these:

error handler/util.go:29 call method {"status": 500, "request_id": "19611a1a-5376-456d-a650-d956cd5a6b9c", "method": "UploadPart", "bucket": "df4547b0-db3b-4671-a111-305e35c4222c", "object": "32936272-3246-4371-ac45-fbed0a33843c", "description": "could not upload a part", "error": "copy payload with buffer: status: code = 2048 message = access to object operation denied", "uploadID": "8uhUVrYJtW8SkRfGDWFgpwh8VcbfEKyqgxk5mXEW8uv2", "Key": "32936272-3246-4371-ac45-fbed0a33843c"} error handler/util.go:29 call method {"status": 500, "request_id": "3b4c1523-8da4-4c68-a55d-ed86f75f5480", "method": "UploadPart", "bucket": "df4547b0-db3b-4671-a111-305e35c4222c", "object": "32936272-3246-4371-ac45-fbed0a33843c", "description": "could not upload a part", "error": "copy payload with buffer: status: code = 2048 message = access to object operation denied", "uploadID": "8uhUVrYJtW8SkRfGDWFgpwh8VcbfEKyqgxk5mXEW8uv2", "Key": "32936272-3246-4371-ac45-fbed0a33843c"}



Complete logs available here - https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/444-1719005333/index.html#suites/0865de5bb52b9f1109887595bf1f9535/5058435b934945cd/.

It can be a test issue as well, but right now it is not clear what is the reason.
carpawell commented 4 weeks ago

See https://github.com/nspcc-dev/neofs-node/pull/2913, it now behaves differently and all object parts seem to be PUT successfully.

carpawell commented 3 weeks ago

Can confirm my local fails were a local problem with outdated aws cli. @evgeniiz321, you can consider closing this issue.

roman-khimov commented 2 weeks ago

Fixed in the node, nothing to do for gateway (except for things already done).