search

davidchalifoux / kaput-cli

The unofficial CLI client for Put.io - Kaput lets you manage your Put.io account from the comfort of your terminal!
https://kaput.sh/
MIT License
68 stars 7 forks source link

Issues with large files not uploading correctly #57

Open sammylupt opened 1 month ago

sammylupt commented 1 month ago

Hi,

First off, thank you so much for your work on this crate! I have been looking for something like this for a while, and I really appreciate it.

I ran into an issue where I am unable to upload large files. (The file in question is 17 gb). After running the relevant line from my shell, the curl status is immediately printed (with "0%") and Kaput immediately prints out "upload finished!"

$ kaput files upload small.mp4  # this works fine                                  
Uploading: small.mp4

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9230k  100   480  100 9229k     72  1394k  0:00:06  0:00:06 --:--:-- 1232k

Upload finished!

$ kaput files upload large.mp4 # this does not work
Uploading: large.mp4

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 17.8G  100   176    0     0    520      0 --:--:-- --:--:-- --:--:--   519

Upload finished!

I did some googling and it seems that Curl can act up when uploading something over 4gb, unless provided with certain headers

Using kaput debug, I grabbed my API key and tried to manually upload the file to put.

When I used the --data-binary option, curl errored with "curl: option --data-binary: out of memory"

When I used the -X POST -T option, put (ngnix) immediately responded with a 413 Request Entity Too Large. I couldn't find any information in Put's API documents about a maximum file size, but apparently there is one 😄.

I imagine that the fix to this behavior would be to run the curl command with -X POST -T, wait for the output, and only call println!("Upload finished!") if the file was uploaded successfully.

Once again thank you so much for building this tool!

davidchalifoux commented 1 month ago

Thanks for reporting this and letting me know what you've already tried!

I'll look into this when I can.

@altaywtf are you able to confirm the file size limit for uploads?

berkanteber commented 1 month ago

Hello, for regular uploads the size limit is 1M. This endpoint is mainly for small files, like torrents or subtitles.

For larger files, we use tus. It's not currently documented but the protocol is open and you can track requests in our web app.

You can also look at these to see how it's implemented: