search

tus / tusd

Reference server implementation in Go of tus: the open protocol for resumable file uploads
https://tus.github.io/tusd
MIT License
2.95k stars 467 forks source link

s3 ListParts error on final concatenation #1015

Closed mosi-kha closed 6 months ago

mosi-kha commented 9 months ago

Describe the bug in v1 everything is okay but in v2, in concatenation mode, after all chunks are uploaded and after call final url to concat files, tusd server raise below error and return 500 status code:

2023/10/01 11:52:46.232313 level=ERROR event=InternalServerError method=POST path="" requestId="" message="operation error S3: ListParts, https response error StatusCode: 404, RequestID: tx000004d1ec4f709b0aa19-0065195d8d-8fc3f44f-ir-thr-at1, HostID: 8fc3f44f-ir-thr-at1-ir-thr, api error NoSuchKey: UnknownError"
2023/10/01 11:52:46.232415 level=INFO event=ResponseOutgoing method=POST path="" requestId="" status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: ListParts, https response error StatusCode: 404, RequestID: tx000004d1ec4f709b0aa19-0065195d8d-8fc3f44f-ir-thr-at1, HostID: 8fc3f44f-ir-thr-at1-ir-thr, api error NoSuchKey: UnknownError\n"

full logs in my test:

2023/10/01 11:49:49.146290 Using 'https://s3.ir-thr-at1.arvanstorage.ir/tusd-test' as S3 endpoint and bucket for storage.
2023/10/01 11:49:49.146367 Using 0.00MB as maximum size.
2023/10/01 11:49:49.146376 Using 'http://localhost:43649/upload/tusd' as the endpoint for hooks
2023/10/01 11:49:49.146399 Enabled hook events: pre-create, pre-finish, post-finish
2023/10/01 11:49:49.146401 Supported tus extensions: creation,creation-with-upload,termination,concatenation,creation-defer-length
2023/10/01 11:49:49.146404 Using 0.0.0.0:1080 as address to listen.
2023/10/01 11:49:49.146405 Using /files/ as the base path.
2023/10/01 11:49:49.146423 Using /metrics as the metrics path.
2023/10/01 11:49:49.146660 You can now upload files to: http://[::]:1080/files/
INFO 2023-10-01 11:49:51,325 spexa.apps.upload.tests setUp 42347 140679901499392 setUp
INFO 2023-10-01 11:49:51,709 spexa.base.clients get_web_client_host 42347 140679901499392 request has not origin or referer header!
INFO 2023-10-01 11:49:51,788 celery.app.trace info 42347 140679901499392 Task spexa.apps.accounts.emails.email_create_session[3c50f7ad-bdff-46ae-b743-2665070b878a] succeeded in 0.03619024199724663s: None
DEBUG 2023-10-01 11:49:57,919 spexa.apps.upload.tests test_upload_concatenation 42347 140679901499392 temp chunk size: 10000
2023/10/01 11:50:28.111273 level=INFO event=RequestIncoming method=POST path="" requestId=""
2023/10/01 11:50:28.111319 level=DEBUG event=HookInvocationStart type=pre-create id=""
DEBUG 2023-10-01 11:50:28,148 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-create
2023/10/01 11:50:28.160851 level=DEBUG event=HookInvocationFinish type=pre-create id=""
2023/10/01 11:50:31.200212 level=INFO event=UploadCreated method=POST path="" requestId="" id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj size=10000 url=http://0.0.0.0:1080/files/f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj
2023/10/01 11:50:31.200342 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj status=201 body=""
2023/10/01 11:50:31.298119 level=INFO event=RequestIncoming method=PATCH path=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj requestId=""
2023/10/01 11:50:31.656802 level=INFO event=ChunkWriteStart method=PATCH path=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj requestId="" id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj maxSize=10000 offset=0
2023/10/01 11:50:32.014368 level=INFO event=ChunkWriteComplete method=PATCH path=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj requestId="" id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj bytesWritten=10000
2023/10/01 11:50:32.378348 level=DEBUG event=HookInvocationStart type=pre-finish id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj
DEBUG 2023-10-01 11:50:32,395 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-finish
2023/10/01 11:50:32.398157 level=DEBUG event=HookInvocationFinish type=pre-finish id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj
2023/10/01 11:50:32.398206 level=INFO event=UploadFinished method=PATCH path=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj requestId="" id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj size=10000
2023/10/01 11:50:32.398253 level=INFO event=ResponseOutgoing method=PATCH path=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj requestId="" id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj status=204 body=""
2023/10/01 11:50:32.398457 level=DEBUG event=HookInvocationStart type=post-finish id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj
DEBUG 2023-10-01 11:50:32,416 spexa.apps.upload.view tusd_hooks 42347 140679258297920 post-finish
INFO 2023-10-01 11:50:32,431 spexa.apps.upload.tusd handle 42347 140679258297920 post finish handle called :D
2023/10/01 11:50:32.474836 level=DEBUG event=HookInvocationFinish type=post-finish id=f3fd5d36d707149a54912909c0520479+2~uxFHk2KuyoSJ8ONjACjV_FdOZTo3HWj
DEBUG 2023-10-01 11:51:20,172 spexa.apps.upload.tests test_upload_concatenation 42347 140679901499392 temp chunk size: 10000
2023/10/01 11:51:21.280388 level=INFO event=RequestIncoming method=POST path="" requestId=""
2023/10/01 11:51:21.280525 level=DEBUG event=HookInvocationStart type=pre-create id=""
DEBUG 2023-10-01 11:51:21,296 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-create
2023/10/01 11:51:21.327202 level=DEBUG event=HookInvocationFinish type=pre-create id=""
2023/10/01 11:51:22.272520 level=INFO event=UploadCreated method=POST path="" requestId="" id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX size=10000 url=http://0.0.0.0:1080/files/f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX
2023/10/01 11:51:22.272554 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX status=201 body=""
2023/10/01 11:51:22.301556 level=INFO event=RequestIncoming method=PATCH path=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX requestId=""
2023/10/01 11:51:22.638026 level=INFO event=ChunkWriteStart method=PATCH path=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX requestId="" id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX maxSize=10000 offset=0
2023/10/01 11:51:23.041932 level=INFO event=ChunkWriteComplete method=PATCH path=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX requestId="" id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX bytesWritten=10000
2023/10/01 11:51:23.419932 level=DEBUG event=HookInvocationStart type=pre-finish id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX
DEBUG 2023-10-01 11:51:23,434 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-finish
2023/10/01 11:51:23.437293 level=DEBUG event=HookInvocationFinish type=pre-finish id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX
2023/10/01 11:51:23.437350 level=INFO event=UploadFinished method=PATCH path=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX requestId="" id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX size=10000
2023/10/01 11:51:23.437396 level=INFO event=ResponseOutgoing method=PATCH path=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX requestId="" id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX status=204 body=""
2023/10/01 11:51:23.437527 level=DEBUG event=HookInvocationStart type=post-finish id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX
DEBUG 2023-10-01 11:51:23,456 spexa.apps.upload.view tusd_hooks 42347 140679258297920 post-finish
INFO 2023-10-01 11:51:23,457 spexa.apps.upload.tusd handle 42347 140679258297920 post finish handle called :D
DEBUG 2023-10-01 11:51:23,460 spexa.apps.upload.tests test_upload_concatenation 42347 140679901499392 temp chunk size: 10000
2023/10/01 11:51:23.502811 level=DEBUG event=HookInvocationFinish type=post-finish id=f2712407e8eaed9be333357426f575df+2~4ZYGFqifXy8kVgmhGRho8G_q2y-YAZX
2023/10/01 11:51:24.593893 level=INFO event=RequestIncoming method=POST path="" requestId=""
2023/10/01 11:51:24.593972 level=DEBUG event=HookInvocationStart type=pre-create id=""
DEBUG 2023-10-01 11:51:24,602 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-create
2023/10/01 11:51:24.620170 level=DEBUG event=HookInvocationFinish type=pre-create id=""
2023/10/01 11:51:25.376492 level=INFO event=UploadCreated method=POST path="" requestId="" id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn size=10000 url=http://0.0.0.0:1080/files/6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn
2023/10/01 11:51:25.376559 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn status=201 body=""
2023/10/01 11:51:25.403857 level=INFO event=RequestIncoming method=PATCH path=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn requestId=""
2023/10/01 11:51:25.931704 level=INFO event=ChunkWriteStart method=PATCH path=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn requestId="" id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn maxSize=10000 offset=0
2023/10/01 11:51:26.356957 level=INFO event=ChunkWriteComplete method=PATCH path=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn requestId="" id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn bytesWritten=10000
2023/10/01 11:51:26.703806 level=DEBUG event=HookInvocationStart type=pre-finish id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn
DEBUG 2023-10-01 11:51:26,717 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-finish
2023/10/01 11:51:26.721644 level=DEBUG event=HookInvocationFinish type=pre-finish id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn
2023/10/01 11:51:26.721697 level=INFO event=UploadFinished method=PATCH path=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn requestId="" id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn size=10000
2023/10/01 11:51:26.721750 level=INFO event=ResponseOutgoing method=PATCH path=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn requestId="" id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn status=204 body=""
2023/10/01 11:51:26.721878 level=DEBUG event=HookInvocationStart type=post-finish id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn
DEBUG 2023-10-01 11:51:26,738 spexa.apps.upload.view tusd_hooks 42347 140679258297920 post-finish
INFO 2023-10-01 11:51:26,739 spexa.apps.upload.tusd handle 42347 140679258297920 post finish handle called :D
DEBUG 2023-10-01 11:51:26,742 spexa.apps.upload.tests test_upload_concatenation 42347 140679901499392 temp chunk size: 8894
2023/10/01 11:51:26.786853 level=DEBUG event=HookInvocationFinish type=post-finish id=6e2643e8718027c1bf81de10015b74e7+2~0VNyKPwdWn31_jDJW0zsI2YZMfXKdDn
2023/10/01 11:51:28.196960 level=INFO event=RequestIncoming method=POST path="" requestId=""
2023/10/01 11:51:28.197015 level=DEBUG event=HookInvocationStart type=pre-create id=""
DEBUG 2023-10-01 11:51:28,201 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-create
2023/10/01 11:51:28.215122 level=DEBUG event=HookInvocationFinish type=pre-create id=""
2023/10/01 11:51:28.937440 level=INFO event=UploadCreated method=POST path="" requestId="" id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu size=8894 url=http://0.0.0.0:1080/files/cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu
2023/10/01 11:51:28.937493 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu status=201 body=""
2023/10/01 11:51:28.971567 level=INFO event=RequestIncoming method=PATCH path=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu requestId=""
2023/10/01 11:51:29.325542 level=INFO event=ChunkWriteStart method=PATCH path=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu requestId="" id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu maxSize=8894 offset=0
2023/10/01 11:51:30.539818 level=INFO event=ChunkWriteComplete method=PATCH path=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu requestId="" id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu bytesWritten=8894
2023/10/01 11:51:31.649837 level=DEBUG event=HookInvocationStart type=pre-finish id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu
DEBUG 2023-10-01 11:51:31,663 spexa.apps.upload.view tusd_hooks 42347 140679258297920 pre-finish
2023/10/01 11:51:31.666865 level=DEBUG event=HookInvocationFinish type=pre-finish id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu
2023/10/01 11:51:31.666916 level=INFO event=UploadFinished method=PATCH path=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu requestId="" id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu size=8894
2023/10/01 11:51:31.666970 level=INFO event=ResponseOutgoing method=PATCH path=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu requestId="" id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu status=204 body=""
2023/10/01 11:51:31.667151 level=DEBUG event=HookInvocationStart type=post-finish id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu
DEBUG 2023-10-01 11:51:31,675 spexa.apps.upload.tests test_upload_concatenation 42347 140679901499392 total temp chunk size: 38894
DEBUG 2023-10-01 11:51:31,689 spexa.apps.upload.view tusd_hooks 42347 140679258297920 post-finish
INFO 2023-10-01 11:51:31,690 spexa.apps.upload.tusd handle 42347 140679258297920 post finish handle called :D
2023/10/01 11:51:31.738785 level=DEBUG event=HookInvocationFinish type=post-finish id=cadf7454fcfb9c3f186f53998b7e038c+2~jKUcsx9sjEfXl_MA5pD3rQ_99FtpWCu
2023/10/01 11:52:43.844428 level=INFO event=RequestIncoming method=POST path="" requestId=""
2023/10/01 11:52:46.232313 level=ERROR event=InternalServerError method=POST path="" requestId="" message="operation error S3: ListParts, https response error StatusCode: 404, RequestID: tx000004d1ec4f709b0aa19-0065195d8d-8fc3f44f-ir-thr-at1, HostID: 8fc3f44f-ir-thr-at1-ir-thr, api error NoSuchKey: UnknownError"
2023/10/01 11:52:46.232415 level=INFO event=ResponseOutgoing method=POST path="" requestId="" status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: ListParts, https response error StatusCode: 404, RequestID: tx000004d1ec4f709b0aa19-0065195d8d-8fc3f44f-ir-thr-at1, HostID: 8fc3f44f-ir-thr-at1-ir-thr, api error NoSuchKey: UnknownError\n"

To Reproduce Steps to reproduce the behavior:

  1. Execute command '...'
  2. Start upload '....'
  3. See error

Expected behavior upload chunks and finalize concat chunks as a single file.

Setup details Please provide following details, if applicable to your situation:

dargmuesli commented 9 months ago

I have the same problem. For me everything works using MinIO locally, but when targeting Scaleway's object storage I get the same error.

Acconut commented 8 months ago

This could be a regression and I will investigate. Does this error occur for every concatenation request repeatedly or just occasionally?

mosi-kha commented 8 months ago

@Acconut Thank you for following up, it occurs for every concatenation request.

Acconut commented 8 months ago

I cannot reproduce this when uploading to AWS S3. What storage provider are you using? It does not look like AWS. What does your client code look like?

dargmuesli commented 8 months ago

For me, it's Scaleway:

Acconut commented 8 months ago

It is possible that AWS S3 returns a different error than Scaleway (or @mosi-kha's S3 endpoint) when tusd tries to detect if the multipart upload still exists. The error message does look a bit odd:

api error NoSuchKey: UnknownError

tusd ignores some errors on purpose and it is possible that we need to include the errors from your endpoint in there as well. In the end, this boils down to an incompatibility between the S3 implementations.

dargmuesli commented 8 months ago

Can I help with extending support for those errors somehow? :bulb:

Acconut commented 8 months ago

What is the exact error that you are getting, @dargmuesli? Is its message exactly the same as the error reported by @mosi-kha?

You can also try on your own to adjust the error handling in https://github.com/tus/tusd/blob/5d0f1210b1fa6e24d4c3bb6c73a1ad8e8fc5956d/pkg/s3store/s3store.go#L697

A brief background: when tusd concatenates a file, is loads the state of all partial uploads. That includes listing the parts in its corresponding multipart upload to see if the upload is complete or not. If the multipart upload is not found, this is an indication that the upload has been completed. I assume that your S3 endpoint returns a slightly different error, so that our error handling does not recognize this.

dargmuesli commented 8 months ago

I receive the following errors when trying to delete an upload on Scaleway:

2023/10/31 03:53:24.005699 level=INFO event=ResponseOutgoing method=DELETE path=823c95a07bfae1b851df10b616de3856+MGNiM2U1M2YtMTczMS00YjRiLWIyY2EtMzQzNWRkNWI0NTEw requestId="" id=823c95a07bfae1b851df10b616de3856+MGNiM2U1M2YtMTczMS00YjRiLWIyY2EtMzQzNWRkNWI0NTEw status=500 body="ERR_INTERNAL_SERVER_ERROR: Multiple errors occurred:\n\toperation error S3: AbortMultipartUpload, https response error StatusCode: 404, RequestID: txb7d0e76680e24406accd5-0065407a33, HostID: txb7d0e76680e24406accd5-0065407a33, api error NoSuchKey: The specified key does not exist.\n\n"
2023/10/31 03:53:24.004944 level=ERROR event=InternalServerError method=DELETE path=823c95a07bfae1b851df10b616de3856+MGNiM2U1M2YtMTczMS00YjRiLWIyY2EtMzQzNWRkNWI0NTEw requestId="" id=823c95a07bfae1b851df10b616de3856+MGNiM2U1M2YtMTczMS00YjRiLWIyY2EtMzQzNWRkNWI0NTEw message="Multiple errors occurred:\n\toperation error S3: AbortMultipartUpload, https response error StatusCode: 404, RequestID: txb7d0e76680e24406accd5-0065407a33, HostID: txb7d0e76680e24406accd5-0065407a33, api error NoSuchKey: The specified key does not exist.\n"
2023/10/31 03:53:23.719716 level=INFO event=RequestIncoming method=DELETE path=823c95a07bfae1b851df10b616de3856+MGNiM2U1M2YtMTczMS00YjRiLWIyY2EtMzQzNWRkNWI0NTEw requestId=""

The upload is deleted successfull though!

Acconut commented 8 months ago

We probably need something like https://github.com/tus/tusd/pull/1019 for the upload termination in https://github.com/tus/tusd/blob/480420942cd831d7261e1c1bd08de88e494cadf7/pkg/s3store/s3store.go#L764 as well.

Maybe it would be a good time to refactor that error logic into a common isAwsNotFoundErr method that checks for Not Found, NoSuchKey, and Forbidden errors. Then this could be used all throughout the S3 store

dargmuesli commented 6 months ago

Hmm, it seems to be working for me now :thinking:

Acconut commented 6 months ago

Alright, maybe their updated their API in the meantime. We depend on help from contributors like you to debug and fix these issues as we don't use S3-compatible service like Scaleway :) I will close this issue then, but feel free to continue here if the problem comes back.

dargmuesli commented 6 months ago

I thought about how I could adjust the error handling, but I'm not experienced in Go enough to make the necessary changes and run the necessary build steps to debug this issue in a reasonable amount of time given the non-urgent priority I personally categorized this issue as :see_no_evil:

Acconut commented 6 months ago

Are you still running into this issue? If not, then there is no need to adjust the code and we can close this issue.

dargmuesli commented 6 months ago

I don't, but maybe it's still an issue for @mosi-kha?

Acconut commented 6 months ago

Alright, thanks for the response. I will close this for now. @mosi-kha, if you are still running into these issues let us know and I will reopen it.

mosi-kha commented 6 months ago

@Acconut @dargmuesli I tested it now with version 2.2.2 and it seems to be fixed Thank you for following up