Uploading large objects using multipart

sk4mi commented 10 months ago

Is your feature request related to a problem? Please describe. I'm trying to upload large files through s3-proxy, but I get EntityTooLarge: Your proposed upload exceeds the maximum allowed size error.

Describe the solution you'd like After some research, I found this aws s3 doc which describes how to perform multipart upload. Maybe it works without change, but i don't manage to upload file using curl:

curl -XPUT -H "Content-Type: multipart/form-data" --form file='@gentoo_root.img'  -L https
://s3-proxy.local/ -v
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://REDACTED
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 6442451163]
* [HTTP/2] [1] [content-type: multipart/form-data; boundary=------------------------FL8SuDEZRyPHrR90fgYD9T]
> PUT / HTTP/2
> Host: s3-proxy.local
> User-Agent: curl/8.4.0
> Accept: */*
> Content-Length: 6442451163
> Content-Type: multipart/form-data; boundary=------------------------FL8SuDEZRyPHrR90fgYD9T
< HTTP/2 500
< date: Thu, 19 Oct 2023 13:39:06 GMT
< content-type: text/html; charset=utf-8
< content-length: 300
< cache-control: no-cache, no-store, no-transform, must-revalidate, private, max-age=0
< expires: Thu, 01 Jan 1970 00:00:00 UTC
< pragma: no-cache
< strict-transport-security: max-age=15724800; includeSubDomains

<!DOCTYPE html>
<html>
  <body>
    <h1>Internal Server Error</h1>
    <p>EntityTooLarge: Your proposed upload exceeds the maximum allowed size
    status code: 400, **********
  </body>

I don't know if it's my utilization who is wrong, or if there some missing feature.

Thanks in advance

oxyno-zeta commented 10 months ago

Hello @sk4mi ,

Thanks for your issue.

With you, I learned that PutObject is limited to 5Gb file size (cf https://www.learnaws.org/2022/08/22/boto3-s3-upload_file-vs-put-object/#:~:text=No%20support%20for%20multipart%20uploads,files%20larger%20than%205%20GB. ) The current implementation must be changed to a s3manager which handles the multipart upload... I will try to work on this soon.

FYI: I've redacted your message because you leaked your instance URL

Regards,

Oxyno-zeta

github-actions[bot] commented 9 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days

oxyno-zeta / s3-proxy

Uploading large objects using multipart #391