search

tus / tusd

Reference server implementation in Go of tus: the open protocol for resumable file uploads
https://tus.github.io/tusd
MIT License
3.08k stars 479 forks source link

Context canceled on S3 upload when upload speed is throttled #1172

Open akkie opened 2 months ago

akkie commented 2 months ago

Describe the bug I locally test tusd with Minio. To test my frontend I throttle the upload speed in the browser. If I upload as example 30 image files, then after a certain time I get a lot of operation error S3: CompleteMultipartUpload, context canceled messages in the tusd log. Maybe it's a timeout on the writing side to S3?

2024-08-21 21:06:08 2024/08/21 19:06:08.197807 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J5V4TPTSZAWP88RRFQJT94E9+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmNhOWIxYzY2LWRhNTEtNGYyYS1hMGZjLWJmNWFlMWRiYmU5M3gxNzI0MjY2NjY2ODU0MTY5NzEw message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-21 21:06:08 2024/08/21 19:06:08.197817 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J5V4TPTSZAWP88RRFQJT94E9+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmNhOWIxYzY2LWRhNTEtNGYyYS1hMGZjLWJmNWFlMWRiYmU5M3gxNzI0MjY2NjY2ODU0MTY5NzEw status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"

There is no such problem if I do not throttle. But this might be still an issue if a client uploads data with limited bandwidth.

To Reproduce Steps to reproduce the behavior:

  1. Configure tusd to work with Minio
  2. Throttle upload speed
  3. Upload multiple files

Expected behavior There should not be an error if the bandwidth is limited.

Setup details Please provide following details, if applicable to your situation:

Acconut commented 2 months ago

Can you give me more details to reproduce this? Which browser did you use and how did you throttle it? What tus-js-client configuration did you use? What are the full logs from tusd?

akkie commented 2 months ago

Sure, I currently use Arc which is Chromium based. If you go to the network setting in the developer tools you have the possibility to throttle the bandwidth.

Sorry, it's german. But I think you get the position of the dropdown.

Bildschirmfoto 2024-08-23 um 22 18 05

I have selected slow 4G and I upload 43 files with a total of 271,4 MB.

My tusd client configuration is:

uploadLengthDeferred: false,
uploadDataDuringCreation: true,
removeFingerprintOnSuccess: true,
retryDelays: [0, 3000, 5000, 10000],
...some metadata

Here are some more logs:

2024-08-23 21:17:14 2024/08/23 19:17:14.660173 level=INFO event=ChunkWriteStart method=POST path="" requestId="" id=01J60AQS4R0CDBB3Q3P91QE0TY+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmFhZmExZjdlLThmZTEtNDZiOC1hNzgzLTM0Yzc4YzkyNTQ5YngxNzI0NDQwNjM0NTIzODcxMTMy maxSize=7979771 offset=0
2024-08-23 22:06:49 2024/08/23 20:06:49.747832 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRRZJCHJ7CC1JHYH1G2N+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmMyYjhhOTg5LTY1MjktNDI5Ny05M2RhLWU2YmExNDUyNzBlNHgxNzI0NDQwNjM0MTQ2Mjc2NTA3 bytesWritten=432029
2024-08-23 22:06:49 2024/08/23 20:06:49.748012 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRRZJCHJ7CC1JHYH1G2N+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmMyYjhhOTg5LTY1MjktNDI5Ny05M2RhLWU2YmExNDUyNzBlNHgxNzI0NDQwNjM0MTQ2Mjc2NTA3 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748050 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRS2Z9K983CWZMSQDWXS+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjk0OGViMjQyLThmZWUtNDlmYy05MTRhLWQxOGM5NzBjNmYyOXgxNzI0NDQwNjM0MTQ4NzM5MDkw bytesWritten=858108
2024-08-23 22:06:49 2024/08/23 20:06:49.748130 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRS2Z9K983CWZMSQDWXS+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjk0OGViMjQyLThmZWUtNDlmYy05MTRhLWQxOGM5NzBjNmYyOXgxNzI0NDQwNjM0MTQ4NzM5MDkw message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748142 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRRZJCHJ7CC1JHYH1G2N+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmMyYjhhOTg5LTY1MjktNDI5Ny05M2RhLWU2YmExNDUyNzBlNHgxNzI0NDQwNjM0MTQ2Mjc2NTA3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748155 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRS2Z9K983CWZMSQDWXS+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjk0OGViMjQyLThmZWUtNDlmYy05MTRhLWQxOGM5NzBjNmYyOXgxNzI0NDQwNjM0MTQ4NzM5MDkw status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748178 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRTS7T528X57HACBB70T+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmVmZTFkYzRkLWYxZDUtNGMwNi1iMzA2LTdiOTNhZmUyYzQ1MXgxNzI0NDQwNjM0MjA4NzkxOTI0 bytesWritten=958180
2024-08-23 22:06:49 2024/08/23 20:06:49.748238 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRTS7T528X57HACBB70T+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmVmZTFkYzRkLWYxZDUtNGMwNi1iMzA2LTdiOTNhZmUyYzQ1MXgxNzI0NDQwNjM0MjA4NzkxOTI0 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748270 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRTS7T528X57HACBB70T+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmVmZTFkYzRkLWYxZDUtNGMwNi1iMzA2LTdiOTNhZmUyYzQ1MXgxNzI0NDQwNjM0MjA4NzkxOTI0 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748360 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRRMNTYJFWG14N3YFCS4+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjU5MGFjNWQ5LTI4MDEtNDE4Zi1iZGFkLWI3MTBhYWZiNTVjOHgxNzI0NDQwNjM0MTM3MzAyMjE1 bytesWritten=1146961
2024-08-23 22:06:49 2024/08/23 20:06:49.748429 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRRMNTYJFWG14N3YFCS4+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjU5MGFjNWQ5LTI4MDEtNDE4Zi1iZGFkLWI3MTBhYWZiNTVjOHgxNzI0NDQwNjM0MTM3MzAyMjE1 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748440 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRRMNTYJFWG14N3YFCS4+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjU5MGFjNWQ5LTI4MDEtNDE4Zi1iZGFkLWI3MTBhYWZiNTVjOHgxNzI0NDQwNjM0MTM3MzAyMjE1 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748478 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRRJ6X5JWVKWN5GWN8RD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjdlOThhZWZiLTc2MWYtNDRjOC1iYWUyLTllZDBlZGEyNDYwMXgxNzI0NDQwNjM0MTM1MDczMTMy bytesWritten=1267932
2024-08-23 22:06:49 2024/08/23 20:06:49.748538 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRRJ6X5JWVKWN5GWN8RD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjdlOThhZWZiLTc2MWYtNDRjOC1iYWUyLTllZDBlZGEyNDYwMXgxNzI0NDQwNjM0MTM1MDczMTMy message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748557 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRRJ6X5JWVKWN5GWN8RD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjdlOThhZWZiLTc2MWYtNDRjOC1iYWUyLTllZDBlZGEyNDYwMXgxNzI0NDQwNjM0MTM1MDczMTMy status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748596 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRR5BQJDFT8PX3TCQ2YK+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmEzYzNkNTNkLTYyYjYtNGRiMC05MjQ3LWY0Mzg4NWJhNmQ2MHgxNzI0NDQwNjM0MTIxMjc1MTc0 bytesWritten=1401949
2024-08-23 22:06:49 2024/08/23 20:06:49.748661 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRR5BQJDFT8PX3TCQ2YK+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmEzYzNkNTNkLTYyYjYtNGRiMC05MjQ3LWY0Mzg4NWJhNmQ2MHgxNzI0NDQwNjM0MTIxMjc1MTc0 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748674 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRR5BQJDFT8PX3TCQ2YK+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmEzYzNkNTNkLTYyYjYtNGRiMC05MjQ3LWY0Mzg4NWJhNmQ2MHgxNzI0NDQwNjM0MTIxMjc1MTc0 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748712 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRSJYB5Q1BYAN7810Y8Z+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjg2YzdiNDU5LWY1YTQtNDQ2ZS04MmYwLTAzMzcwMDdmNGNkMngxNzI0NDQwNjM0MTc0MTM0NzU3 bytesWritten=1418989
2024-08-23 22:06:49 2024/08/23 20:06:49.748766 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRSJYB5Q1BYAN7810Y8Z+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjg2YzdiNDU5LWY1YTQtNDQ2ZS04MmYwLTAzMzcwMDdmNGNkMngxNzI0NDQwNjM0MTc0MTM0NzU3 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748776 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRSJYB5Q1BYAN7810Y8Z+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjg2YzdiNDU5LWY1YTQtNDQ2ZS04MmYwLTAzMzcwMDdmNGNkMngxNzI0NDQwNjM0MTc0MTM0NzU3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748815 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRRP1FZS27CHPDTAHJAA+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmRmOTM0ZmJiLTg0NWQtNDJkZC1iZmRkLTljYjFiNmMzYjUyZXgxNzI0NDQwNjM0MTQwMDE1MzQw bytesWritten=1620977
2024-08-23 22:06:49 2024/08/23 20:06:49.748892 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRRP1FZS27CHPDTAHJAA+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmRmOTM0ZmJiLTg0NWQtNDJkZC1iZmRkLTljYjFiNmMzYjUyZXgxNzI0NDQwNjM0MTQwMDE1MzQw message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.748904 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRRP1FZS27CHPDTAHJAA+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmRmOTM0ZmJiLTg0NWQtNDJkZC1iZmRkLTljYjFiNmMzYjUyZXgxNzI0NDQwNjM0MTQwMDE1MzQw status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.748985 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRQSQZRQ084Z1Q9S5D00+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjkxMTkwMzJiLTkyNjEtNDAwNi1iYmZlLTgxYjg3NTI3NTEwZHgxNzI0NDQwNjM0MTA4MjY4ODQw bytesWritten=2035226
2024-08-23 22:06:49 2024/08/23 20:06:49.749117 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRQSQZRQ084Z1Q9S5D00+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjkxMTkwMzJiLTkyNjEtNDAwNi1iYmZlLTgxYjg3NTI3NTEwZHgxNzI0NDQwNjM0MTA4MjY4ODQw message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.749127 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRQSQZRQ084Z1Q9S5D00+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjkxMTkwMzJiLTkyNjEtNDAwNi1iYmZlLTgxYjg3NTI3NTEwZHgxNzI0NDQwNjM0MTA4MjY4ODQw status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.749194 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZ4DQSFH70KNCY783VQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjkxNzBjNmEwLTNlNTctNDMwNi05YTk0LWFhNzY0YWY0ODZlNXgxNzI0NDQwNjM0MzQ1OTU3NTkx bytesWritten=2526687
2024-08-23 22:06:49 2024/08/23 20:06:49.749327 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZ4DQSFH70KNCY783VQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjkxNzBjNmEwLTNlNTctNDMwNi05YTk0LWFhNzY0YWY0ODZlNXgxNzI0NDQwNjM0MzQ1OTU3NTkx message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.749352 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZ4DQSFH70KNCY783VQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjkxNzBjNmEwLTNlNTctNDMwNi05YTk0LWFhNzY0YWY0ODZlNXgxNzI0NDQwNjM0MzQ1OTU3NTkx status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.749413 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZ4EBE0ZCEGBDDTF569+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmJiNWQ5N2M0LWFmZDMtNDMyNS1hYTE5LTc3MmE4MGE0YzA5MXgxNzI0NDQwNjM0MzQzODEzMjk5 bytesWritten=2990674
2024-08-23 22:06:49 2024/08/23 20:06:49.749520 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZ4EBE0ZCEGBDDTF569+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmJiNWQ5N2M0LWFmZDMtNDMyNS1hYTE5LTc3MmE4MGE0YzA5MXgxNzI0NDQwNjM0MzQzODEzMjk5 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.749551 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZ4EBE0ZCEGBDDTF569+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmJiNWQ5N2M0LWFmZDMtNDMyNS1hYTE5LTc3MmE4MGE0YzA5MXgxNzI0NDQwNjM0MzQzODEzMjk5 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.749616 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZCTSSP9SQW2GPYJ48B+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjA3ZTM1ZDc4LWU4NzQtNDIyNi1iMTBjLTljMjkxMDU3ZDlmMXgxNzI0NDQwNjM0MzUwNzYxNTkx bytesWritten=3200799
2024-08-23 22:06:49 2024/08/23 20:06:49.749677 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZCTSSP9SQW2GPYJ48B+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjA3ZTM1ZDc4LWU4NzQtNDIyNi1iMTBjLTljMjkxMDU3ZDlmMXgxNzI0NDQwNjM0MzUwNzYxNTkx message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.749688 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZCTSSP9SQW2GPYJ48B+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjA3ZTM1ZDc4LWU4NzQtNDIyNi1iMTBjLTljMjkxMDU3ZDlmMXgxNzI0NDQwNjM0MzUwNzYxNTkx status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.749732 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZY0SE17MYMCGZ102C9+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjY2ODQwOWI5LTA5YTQtNDI5Mi1iNmQxLTEyZmE0M2FjMjM1MHgxNzI0NDQwNjM0MzczNTQxNTA3 bytesWritten=0
2024-08-23 22:06:49 2024/08/23 20:06:49.749755 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZY0SE17MYMCGZ102C9+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjY2ODQwOWI5LTA5YTQtNDI5Mi1iNmQxLTEyZmE0M2FjMjM1MHgxNzI0NDQwNjM0MzczNTQxNTA3 message="operation error S3: UploadPart, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.749766 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZY0SE17MYMCGZ102C9+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjY2ODQwOWI5LTA5YTQtNDI5Mi1iNmQxLTEyZmE0M2FjMjM1MHgxNzI0NDQwNjM0MzczNTQxNTA3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: UploadPart, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.749814 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRY6SX5Y1R6KGHGQ8RAJ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmY1ODlhYmVmLWRiMmEtNGMzMC1iZGVjLWNhOWUwNDZjZGI4M3gxNzI0NDQwNjM0MzEzNjQxOTI0 bytesWritten=3693761
2024-08-23 22:06:49 2024/08/23 20:06:49.749898 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRY6SX5Y1R6KGHGQ8RAJ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmY1ODlhYmVmLWRiMmEtNGMzMC1iZGVjLWNhOWUwNDZjZGI4M3gxNzI0NDQwNjM0MzEzNjQxOTI0 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.749920 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRY6SX5Y1R6KGHGQ8RAJ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmY1ODlhYmVmLWRiMmEtNGMzMC1iZGVjLWNhOWUwNDZjZGI4M3gxNzI0NDQwNjM0MzEzNjQxOTI0 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.749971 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZXA7Q30YMMHQ1CMWNJ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjNmNTQ1NmU5LTNkMjctNDgwMS1hYTNjLWIwZjBkZTVmMGQ3NHgxNzI0NDQwNjM0MzczNjI0MzQx bytesWritten=4173813
2024-08-23 22:06:49 2024/08/23 20:06:49.750047 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZXA7Q30YMMHQ1CMWNJ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjNmNTQ1NmU5LTNkMjctNDgwMS1hYTNjLWIwZjBkZTVmMGQ3NHgxNzI0NDQwNjM0MzczNjI0MzQx message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750057 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZXA7Q30YMMHQ1CMWNJ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjNmNTQ1NmU5LTNkMjctNDgwMS1hYTNjLWIwZjBkZTVmMGQ3NHgxNzI0NDQwNjM0MzczNjI0MzQx status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750092 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZDRRNBSX30V148J05Q+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjViZGIxM2E0LTRkNzYtNDNiNy1iYjMzLWVmMzAwMTBlZmFhMHgxNzI0NDQwNjM0MzUzODQ3OTI0 bytesWritten=4454746
2024-08-23 22:06:49 2024/08/23 20:06:49.750180 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZDRRNBSX30V148J05Q+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjViZGIxM2E0LTRkNzYtNDNiNy1iYjMzLWVmMzAwMTBlZmFhMHgxNzI0NDQwNjM0MzUzODQ3OTI0 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750201 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZDRRNBSX30V148J05Q+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjViZGIxM2E0LTRkNzYtNDNiNy1iYjMzLWVmMzAwMTBlZmFhMHgxNzI0NDQwNjM0MzUzODQ3OTI0 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750252 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZ3AW8Q9YEMXYYEDJ9Q+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjI1OWY4ZjVlLTNhZjItNDQ2OC04NzI2LWViMmRiMmUyMjFmNHgxNzI0NDQwNjM0MzQ0MTk1Mzgy bytesWritten=4946174
2024-08-23 22:06:49 2024/08/23 20:06:49.750314 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZ3AW8Q9YEMXYYEDJ9Q+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjI1OWY4ZjVlLTNhZjItNDQ2OC04NzI2LWViMmRiMmUyMjFmNHgxNzI0NDQwNjM0MzQ0MTk1Mzgy message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750333 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZ3AW8Q9YEMXYYEDJ9Q+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjI1OWY4ZjVlLTNhZjItNDQ2OC04NzI2LWViMmRiMmUyMjFmNHgxNzI0NDQwNjM0MzQ0MTk1Mzgy status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750363 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZG12B95VDY1V8FKSD6+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjIzY2Y2YzI2LWJhMWYtNDcyZS1iM2M1LWVmOTNhZWY5OGYwMngxNzI0NDQwNjM0MzU4MTMxNDY2 bytesWritten=5027201
2024-08-23 22:06:49 2024/08/23 20:06:49.750419 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZG12B95VDY1V8FKSD6+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjIzY2Y2YzI2LWJhMWYtNDcyZS1iM2M1LWVmOTNhZWY5OGYwMngxNzI0NDQwNjM0MzU4MTMxNDY2 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750435 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZG12B95VDY1V8FKSD6+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjIzY2Y2YzI2LWJhMWYtNDcyZS1iM2M1LWVmOTNhZWY5OGYwMngxNzI0NDQwNjM0MzU4MTMxNDY2 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750460 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZNJGWJCMG6WF7VZPCA+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjVmZGUxOWMzLWQ4ODktNDAwYy1hMTcxLTAxM2VkZWI0NWY0N3gxNzI0NDQwNjM0MzYwMjYxNjMy bytesWritten=5329688
2024-08-23 22:06:49 2024/08/23 20:06:49.750497 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZNJGWJCMG6WF7VZPCA+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjVmZGUxOWMzLWQ4ODktNDAwYy1hMTcxLTAxM2VkZWI0NWY0N3gxNzI0NDQwNjM0MzYwMjYxNjMy message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750514 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZNJGWJCMG6WF7VZPCA+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjVmZGUxOWMzLWQ4ODktNDAwYy1hMTcxLTAxM2VkZWI0NWY0N3gxNzI0NDQwNjM0MzYwMjYxNjMy status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750536 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS2V4BH59Y7D22X97WRK+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjc0MTRhNmYyLTk4ZWItNDQ1Ni1iMDJlLWQ1MTZkYmIzMTMyMXgxNzI0NDQwNjM0NDY0NDYyNjc0 bytesWritten=5443240
2024-08-23 22:06:49 2024/08/23 20:06:49.750596 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS2V4BH59Y7D22X97WRK+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjc0MTRhNmYyLTk4ZWItNDQ1Ni1iMDJlLWQ1MTZkYmIzMTMyMXgxNzI0NDQwNjM0NDY0NDYyNjc0 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750607 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS2V4BH59Y7D22X97WRK+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjc0MTRhNmYyLTk4ZWItNDQ1Ni1iMDJlLWQ1MTZkYmIzMTMyMXgxNzI0NDQwNjM0NDY0NDYyNjc0 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750635 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZCV67D87TTS8ECFPAV+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjk0YmJjNDQ4LTRkM2QtNDU2Ni05MzY0LTVlMjI3MTM2YWQyOXgxNzI0NDQwNjM0MzU0NTg5NzU3 bytesWritten=5591095
2024-08-23 22:06:49 2024/08/23 20:06:49.750681 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZCV67D87TTS8ECFPAV+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjk0YmJjNDQ4LTRkM2QtNDU2Ni05MzY0LTVlMjI3MTM2YWQyOXgxNzI0NDQwNjM0MzU0NTg5NzU3 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750689 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZCV67D87TTS8ECFPAV+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjk0YmJjNDQ4LTRkM2QtNDU2Ni05MzY0LTVlMjI3MTM2YWQyOXgxNzI0NDQwNjM0MzU0NTg5NzU3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750708 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS05AZHCGJRV19CJTE5X+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjM4OWY1YTA3LTc0OTAtNGFlNS04MjhiLTQwZjZhNGY2YzhkYXgxNzI0NDQwNjM0Mzk2Njc4NTA3 bytesWritten=5636635
2024-08-23 22:06:49 2024/08/23 20:06:49.750777 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS05AZHCGJRV19CJTE5X+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjM4OWY1YTA3LTc0OTAtNGFlNS04MjhiLTQwZjZhNGY2YzhkYXgxNzI0NDQwNjM0Mzk2Njc4NTA3 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750792 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS05AZHCGJRV19CJTE5X+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjM4OWY1YTA3LTc0OTAtNGFlNS04MjhiLTQwZjZhNGY2YzhkYXgxNzI0NDQwNjM0Mzk2Njc4NTA3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750826 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS00V05H0QGHYDSZFQ9X+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjM1OWFiNzI4LWZhYTMtNDNlYy04NDdkLTBkMDRiZDIyZThiYngxNzI0NDQwNjM0MzcwNzAyNTkx bytesWritten=6148748
2024-08-23 22:06:49 2024/08/23 20:06:49.750905 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS00V05H0QGHYDSZFQ9X+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjM1OWFiNzI4LWZhYTMtNDNlYy04NDdkLTBkMDRiZDIyZThiYngxNzI0NDQwNjM0MzcwNzAyNTkx message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750913 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS00V05H0QGHYDSZFQ9X+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjM1OWFiNzI4LWZhYTMtNDNlYy04NDdkLTBkMDRiZDIyZThiYngxNzI0NDQwNjM0MzcwNzAyNTkx status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.750930 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS1MX8DNXKMMRZMZPQRD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjlmZWRkOWU2LTBmYTgtNDYxOC04N2FjLTk4NjBiMjVjNzFkZHgxNzI0NDQwNjM0NDIxNzU2MjU3 bytesWritten=6327729
2024-08-23 22:06:49 2024/08/23 20:06:49.750965 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS1MX8DNXKMMRZMZPQRD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjlmZWRkOWU2LTBmYTgtNDYxOC04N2FjLTk4NjBiMjVjNzFkZHgxNzI0NDQwNjM0NDIxNzU2MjU3 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.750978 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS1MX8DNXKMMRZMZPQRD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjlmZWRkOWU2LTBmYTgtNDYxOC04N2FjLTk4NjBiMjVjNzFkZHgxNzI0NDQwNjM0NDIxNzU2MjU3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751015 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS2Y1VFSQJQKPQ5JDAZQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjIzNjM5YTAzLTM4ZTAtNGQyYi05NzA0LTZkMzZkOTkyZjNmOXgxNzI0NDQwNjM0NDY1NTg4NzU3 bytesWritten=7338053
2024-08-23 22:06:49 2024/08/23 20:06:49.751060 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS2Y1VFSQJQKPQ5JDAZQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjIzNjM5YTAzLTM4ZTAtNGQyYi05NzA0LTZkMzZkOTkyZjNmOXgxNzI0NDQwNjM0NDY1NTg4NzU3 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751065 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS2Y1VFSQJQKPQ5JDAZQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjIzNjM5YTAzLTM4ZTAtNGQyYi05NzA0LTZkMzZkOTkyZjNmOXgxNzI0NDQwNjM0NDY1NTg4NzU3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751082 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS4WHZXQKS0MFGYT5MXG+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjY1MTI5ZGQ5LWU1OTAtNDRiNS04MTQwLWRlYWZlNjQ4OTZiZXgxNzI0NDQwNjM0NTI1NzM2NTkx bytesWritten=8066909
2024-08-23 22:06:49 2024/08/23 20:06:49.751115 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS4WHZXQKS0MFGYT5MXG+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjY1MTI5ZGQ5LWU1OTAtNDRiNS04MTQwLWRlYWZlNjQ4OTZiZXgxNzI0NDQwNjM0NTI1NzM2NTkx message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751120 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS4WHZXQKS0MFGYT5MXG+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjY1MTI5ZGQ5LWU1OTAtNDRiNS04MTQwLWRlYWZlNjQ4OTZiZXgxNzI0NDQwNjM0NTI1NzM2NTkx status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751141 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS1D75X6BR7NDN0VR0VD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjQ4NGE2NjJlLTdkYWEtNDk1YS1iNjRiLTJjODZjYjMyNTY5M3gxNzI0NDQwNjM0NDE1MDg2Njc0 bytesWritten=8866867
2024-08-23 22:06:49 2024/08/23 20:06:49.751185 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS1D75X6BR7NDN0VR0VD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjQ4NGE2NjJlLTdkYWEtNDk1YS1iNjRiLTJjODZjYjMyNTY5M3gxNzI0NDQwNjM0NDE1MDg2Njc0 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751198 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS1D75X6BR7NDN0VR0VD+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjQ4NGE2NjJlLTdkYWEtNDk1YS1iNjRiLTJjODZjYjMyNTY5M3gxNzI0NDQwNjM0NDE1MDg2Njc0 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751232 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQRZQDTD2YRXSNEKSZN82+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjZiNzgyYThmLTEwNzgtNDNkYi1iY2IzLTZkNjM0ZWM0ODhlN3gxNzI0NDQwNjM0MzYzNDk4NzU3 bytesWritten=0
2024-08-23 22:06:49 2024/08/23 20:06:49.751235 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQRZQDTD2YRXSNEKSZN82+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjZiNzgyYThmLTEwNzgtNDNkYi1iY2IzLTZkNjM0ZWM0ODhlN3gxNzI0NDQwNjM0MzYzNDk4NzU3 message="operation error S3: UploadPart, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751248 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQRZQDTD2YRXSNEKSZN82+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjZiNzgyYThmLTEwNzgtNDNkYi1iY2IzLTZkNjM0ZWM0ODhlN3gxNzI0NDQwNjM0MzYzNDk4NzU3 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: UploadPart, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751274 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS36PBEWZEB1Q5ACJCSQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmNhYzkxZTIzLWU1ZDQtNDM4MC04OTgzLTFkZDZmMGZkYTNiM3gxNzI0NDQwNjM0NDcyMzk4OTY2 bytesWritten=9173680
2024-08-23 22:06:49 2024/08/23 20:06:49.751345 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS36PBEWZEB1Q5ACJCSQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmNhYzkxZTIzLWU1ZDQtNDM4MC04OTgzLTFkZDZmMGZkYTNiM3gxNzI0NDQwNjM0NDcyMzk4OTY2 message="operation error S3: CompleteMultipartUpload, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751365 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS36PBEWZEB1Q5ACJCSQ+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmNhYzkxZTIzLWU1ZDQtNDM4MC04OTgzLTFkZDZmMGZkYTNiM3gxNzI0NDQwNjM0NDcyMzk4OTY2 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: CompleteMultipartUpload, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751392 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS0Z1P7ZRE3R519B8873+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjgyNWM5ZmExLTUwMGQtNGE1NC05MjYwLWVkYmNlNzlmNWE4OXgxNzI0NDQwNjM0NDAxMDkzODgy bytesWritten=0
2024-08-23 22:06:49 2024/08/23 20:06:49.751395 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS0Z1P7ZRE3R519B8873+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjgyNWM5ZmExLTUwMGQtNGE1NC05MjYwLWVkYmNlNzlmNWE4OXgxNzI0NDQwNjM0NDAxMDkzODgy message="operation error S3: UploadPart, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751399 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS0Z1P7ZRE3R519B8873+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjgyNWM5ZmExLTUwMGQtNGE1NC05MjYwLWVkYmNlNzlmNWE4OXgxNzI0NDQwNjM0NDAxMDkzODgy status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: UploadPart, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751413 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS4ZJ9YF4Q1XEW05NXEF+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmNjYmQzOTk1LTBlZWMtNGUxOC05NDJlLWNjZjExMDQxY2VkYngxNzI0NDQwNjM0NTI4NTI2NTkx bytesWritten=9897271
2024-08-23 22:06:49 2024/08/23 20:06:49.751657 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS17W4CS4FPRD4V1Y4Q0+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjE4M2VmODEzLTk1ZTAtNGI3MC1iOTY4LTk5OWY2NzVjZjk2ZHgxNzI0NDQwNjM0NDEwNjg0Njc0 bytesWritten=0
2024-08-23 22:06:49 2024/08/23 20:06:49.751661 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS17W4CS4FPRD4V1Y4Q0+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjE4M2VmODEzLTk1ZTAtNGI3MC1iOTY4LTk5OWY2NzVjZjk2ZHgxNzI0NDQwNjM0NDEwNjg0Njc0 message="operation error S3: UploadPart, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751665 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS17W4CS4FPRD4V1Y4Q0+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjE4M2VmODEzLTk1ZTAtNGI3MC1iOTY4LTk5OWY2NzVjZjk2ZHgxNzI0NDQwNjM0NDEwNjg0Njc0 status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: UploadPart, context canceled\n"
2024-08-23 22:06:49 2024/08/23 20:06:49.751689 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS3Y7WE2GP9E8F0GVY81+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLjYxYjZhM2MwLTczZmUtNGM5NS1iN2QyLWRjYmYyOWIzMjMyYXgxNzI0NDQwNjM0NDk2NzE2NzU3 bytesWritten=10383332
2024-08-23 22:06:49 2024/08/23 20:06:49.751867 level=INFO event=ChunkWriteComplete method=POST path="" requestId="" id=01J60AQS4R0CDBB3Q3P91QE0TY+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmFhZmExZjdlLThmZTEtNDZiOC1hNzgzLTM0Yzc4YzkyNTQ5YngxNzI0NDQwNjM0NTIzODcxMTMy bytesWritten=0
2024-08-23 22:06:49 2024/08/23 20:06:49.751871 level=ERROR event=InternalServerError method=POST path="" requestId="" id=01J60AQS4R0CDBB3Q3P91QE0TY+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmFhZmExZjdlLThmZTEtNDZiOC1hNzgzLTM0Yzc4YzkyNTQ5YngxNzI0NDQwNjM0NTIzODcxMTMy message="operation error S3: UploadPart, context canceled"
2024-08-23 22:06:49 2024/08/23 20:06:49.751876 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=01J60AQS4R0CDBB3Q3P91QE0TY+OWMyODgxOGQtZmUwYS00MmU2LWI2ZTMtNWY5ZmU2NTEzN2ZhLmFhZmExZjdlLThmZTEtNDZiOC1hNzgzLTM0Yzc4YzkyNTQ5YngxNzI0NDQwNjM0NTIzODcxMTMy status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: UploadPart, context canceled\n"

The browser shows the following logs:

Bildschirmfoto 2024-08-23 um 22 26 29

I use caddy in front of tusd with the following settings:

@upload host upload.my.domain
    handle @upload {
        # tusd provides a GET method for debugging, we disable that
        # https://github.com/tus/tusd/blob/main/docs/faq.md#how-can-i-prevent-users-from-downloading-the-uploaded-files
        @isGet method GET
        respond @isGet 405

        reverse_proxy http://tusd:9500 {
            # When running tusd behind a reverse proxy, buffering should be completely disabled:
            # https://github.com/tus/tusd/blob/main/docs/faq.md#can-i-run-tusd-behind-a-reverse-proxy
            #
            # By default, the proxy partially buffers the response for wire efficiency:
            # https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#streaming
            #
            # This setting disabled buffering completely
            flush_interval -1
        }
    }
Acconut commented 2 months ago

Thanks for the additional details. I will look into this in two weeks. Does this also happen if you upload directly to tusd without using Caddy as a reverse proxy?

akkie commented 2 months ago

It needs a bit tweaking on my side to test that only with tusd, because my hook endpoints use authentication and that works only with my local domain.

But I discovered another interesting case. Maybe this is expected, but I think this could lead to disk space issues on the tusd instance.

During a throttled upload I have looked to my upload bucket and I have seen that only the info file gets stored and after looking at the docker metrics I have seen that tusd writes the data to its own storage. With a not throttled upload it sends it directly to minio over the network.

In the first screenshot you can see that I first tested the upload with some small files and later then with a larger video (ca. 145mb). This is where I discovered the issue that there is no file in minio during the upload. Then I restarted tusd, to clear the stats. I disabled throttling and uploaded the video. You can see that there was no write to the disk but the networking IO shows that it has sent the file to minio. It's only a blue line, but if I hover over it, then you can see that it also has sent the data.

Bildschirmfoto 2024-08-24 um 23 16 39

Hovering over the data

Bildschirmfoto 2024-08-25 um 00 04 09

In the second screenshot you can see that with enabled throttling it writes the data to the disk and then sends it in three chunks over the network. But this is still strange, I can see now data in minio during upload.

Bildschirmfoto 2024-08-24 um 23 33 57
Acconut commented 2 months ago

But I discovered another interesting case. Maybe this is expected, but I think this could lead to disk space issues on the tusd instance.

Tusd buffers data on disk before uploading it to S3. This happens in chunks, so the entire file isn't stored on disk. We have to do this because S3 expects us to known the size of the chunk we are uploading. See https://github.com/tus/tusd/issues/1084 for more details and a potential alternative/improvement.

Acconut commented 2 months ago

But this is still strange, I can see now data in minio during upload.

S3 uploads are performing using S3 nultipart uploads, which don't shown up in the file explorer and ListObjects calls until the upload is done.

akkie commented 2 months ago

Thanks for you explanation and the insides. Very helpful. Do you also have an explanation why it has no writes on the disk, if throttling is disabled as you can see in the first screenshot?

akkie commented 2 months ago

Tested it without Caddy today and I couldn't reproduce it. So it seems it's an issue when using caddy.

spaiz commented 2 months ago

I think I'm getting the same when working with telnyx.com and backblaze.com It is happening when I try to upload ~1GB file (for 100-300mb everything is OK)

I'm using embedded golang package, and testing it from local host. And this is what I get :/ Something that I notices, PATCH endpoint fails with status 500 after ~10 seconds.

For frontend I use uppy.

Logs from telnyx.com

SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
SDK 2024/09/01 23:27:29 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , canceled, context canceled
2024/09/01 23:27:29 INFO ChunkWriteComplete method=PATCH path=/b2b325c9c55414ce8837d864c1acc6c5+2~gk44tFzF-niTC3yFuS8HPTGQ_vn1jqI requestId="" id=b2b325c9c55414ce8837d864c1acc6c5+2~gk44tFzF-niTC3yFuS8HPTGQ_vn1jqI bytesWritten=0
2024/09/01 23:27:29 ERROR InternalServerError method=PATCH path=/b2b325c9c55414ce8837d864c1acc6c5+2~gk44tFzF-niTC3yFuS8HPTGQ_vn1jqI requestId="" id=b2b325c9c55414ce8837d864c1acc6c5+2~gk44tFzF-niTC3yFuS8HPTGQ_vn1jqI message="operation error S3: UploadPart, context canceled"
2024/09/01 23:27:29 INFO ResponseOutgoing method=PATCH path=/b2b325c9c55414ce8837d864c1acc6c5+2~gk44tFzF-niTC3yFuS8HPTGQ_vn1jqI requestId="" id=b2b325c9c55414ce8837d864c1acc6c5+2~gk44tFzF-niTC3yFuS8HPTGQ_vn1jqI status=500 body="ERR_INTERNAL_SERVER_ERROR: operation error S3: UploadPart, context canceled\n"
[GIN] 2024/09/01 - 23:27:29 | 500 | 10.051479083s |             ::1 | PATCH    "/uploads/b2b325c9c55414ce8837d864c1acc6c5+2~gk44tFzF-niTC3yFuS8HPTGQ_vn1jqI"

Logs from backblaze.com

Screenshot 2024-09-02 at 10 49 47

I tried to find any way to configure some timeout maybe, but didn't find anything.

My setup is

func NewTelnyxStore(ctx context.Context) (s3store.S3Store, error) {
    telnyxAPIKey := os.Getenv("TELNYX_API_KEY")
    region, _ := os.LookupEnv("AWS_REGION")
    endpoint := fmt.Sprintf("https://%s.telnyxcloudstorage.com", region)

    s3Client, err := connectS3(ctx, telnyxAPIKey, telnyxAPIKey, region, endpoint)
    bucket, _ := os.LookupEnv("STORAGE_BUCKET")
    store := s3store.New(bucket, s3Client)
    return store, err
}

func NewBackblazeStore(ctx context.Context) (s3store.S3Store, error) {
    region := "us-east-005"
    endpoint := fmt.Sprintf("https://s3.%s.backblazeb2.com", region)

    keyId := "***"
    appKey := "****"

    s3Client, err := connectS3(ctx, keyId, appKey, region, endpoint)
    bucket, _ := os.LookupEnv("STORAGE_BUCKET")
    store := s3store.New(bucket, s3Client)
    return store, err
}

func connectS3(ctx context.Context, apiKey string, secretKey string, region string, endpoint string) (*s3.Client, error) {

    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion(region),
        config.WithCredentialsProvider(aws.CredentialsProviderFunc(
            func(context.Context) (aws.Credentials, error) {
                return aws.Credentials{
                    AccessKeyID:     apiKey,
                    SecretAccessKey: secretKey,
                }, nil
            })),
        config.WithS3UseARNRegion(true),
        config.WithS3DisableExpressAuth(true),
        config.WithS3DisableMultiRegionAccessPoints(true),
    )

    if err != nil {
        return nil, err
    }

    cfg.BaseEndpoint = aws.String(endpoint)

    httpClient := awshttp.NewBuildableClient().WithDialerOptions(func(d *net.Dialer) {
        //d.KeepAlive = -1
        d.Timeout = time.Minute * 10
    })

    client := s3.NewFromConfig(cfg, func(o *s3.Options) {
        o.UsePathStyle = true
        o.ClientLogMode |= aws.LogResponse | aws.LogResponseEventMessage | aws.LogRequestEventMessage | aws.LogRequest | aws.LogDeprecatedUsage | aws.LogRetries | aws.LogResponseWithBody
        o.HTTPClient = httpClient
    })

    return client, nil
}
/*
    storageDriver := os.Getenv("STORAGE_DRIVER")
    store, err := infra.NewTelnyxStore(context.Background())
    if err != nil {
        panic(err)
    }
*/
        store, err := infra.NewBackblazeStore(context.Background())
    if err != nil {
        panic(err)
    }

    locker := filelocker.New("./files/uploads")
    composer := tusd.NewStoreComposer()
    store.UseIn(composer)
    locker.UseIn(composer)

    cc := tusd.Config{
        BasePath:              "/uploads/",
        StoreComposer:         composer,
        NotifyCompleteUploads: true,
    }

    handler, err := tusd.NewUnroutedHandler(cc)
spaiz commented 2 months ago

I solved the issue by setting

        NetworkTimeout:     time.Minute * 5,

@akkie I'm sure there should be a similar command line param.