Closed ghost closed 3 years ago
jkawamoto
commented
3 years ago The library this app uses needs to copy the uploading file into RAM at least two times to build a request. That means it'll consume at least 10GB RAM to upload a 5GB file.
Btw, can you upload a 5GB file? The API document looks like the max file size is 1GB:
ghost
commented
3 years ago yes, you can upload 5GB via API. 15GB if you have pro account, etc.
i emailed pixeldrain support and Wim gave me a go app for uploading: https://pixeldrain.com/u/V8UGqPb4 pd "file/location/here"
does not consume ram which is an issue with your app i have as I upload large file simultaneously and i run out of ram.
Wim's app does not copy files into ram it seems, no usage at all. bottleneck is going to be your upload speed to PD not disks so i don't see need to copy to ram. maybe you can rework that. =)
btw if you email PD suport wim will provide source code for the go app i linked above if you are interested.
jkawamoto
commented
3 years ago I see. I've just opened a PR https://github.com/go-openapi/runtime/pull/214. It'll reduce the RAM usage.
ghost
commented
3 years ago I see, thank you :)
Fornax96
commented
2 years ago Here's the source code for my uploader: https://pixeldrain.com/u/VmcG7r4f
It spawns a separate goroutine to create the multipart file in the background.
jkawamoto
commented
2 years ago Thanks for the source code. It looks like go-openapi/runtime, which is used in this app, also uses a goroutine to prepare the multipart message:
It wan't optimal to compute a content type but it's fixed now.
why does it take 14GB+ usage of ram to upload a 5GB file?