search

cvisionai / tator

Video analytics web platform
https://tator.io
GNU Affero General Public License v3.0
100 stars 34 forks source link

Incompatibility with Min.IO when transaction size is 64Kb #1593

Open bctcvai opened 5 months ago

bctcvai commented 5 months ago

The multibucket API wrapper in store.py appears to have an API in compatibility with min.io servers.

Steps to reproduce:

1.) Configure a min.io project specific bucket. 2.) Attempt to upload a file in 64Kb chunks 3.) Observe the upload fails.

In the REST logs:

INFO 2024-03-23 12:14:05,716 store 9802 140398682389568 "Multipart failed: An error occurred (EntityTooSmall) when calling the CompleteMultipartUpload operation: Your proposed upload is smaller than the minimum allowed object size."
INFO 2024-03-23 12:14:05,716 _base_views 9802 140398682389568 "Handling Exception!"
INFO 2024-03-23 12:14:05,717 _base_views 9802 140398682389568 "<class 'Exception'>"
ERROR 2024-03-23 12:14:05,717 _base_views 9802 140398682389568 "Exception in request: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/tator_online/main/rest/_base_views.py", line 63, in post
    response_data = self._post(params)
  File "/tator_online/main/rest/upload_completion.py", line 35, in _post
    raise Exception(f"Upload completion for {key} failed!")
Exception: Upload completion for XX/YYY/ZZZZZZZZZ/720_ftuxqKvUKv.mp4 failed!

Client-side instrumentation, it can be observed there are 45 parts making up the upload, each with a valid ETag assigned.

bctcvai commented 5 months ago

We should add checks for valid chunk size, as to fail more gracefully.

Root cause of issue was default timeout of 30 seconds was added/documented but didn't change/break API so it was a silent but deadly change to cellular use cases.