imroc / req

Simple Go HTTP client with Black Magic
https://req.cool
MIT License
4.24k stars 346 forks source link

Multipart form upload use huge amount of memory #231

Open n0v3xx opened 1 year ago

n0v3xx commented 1 year ago

Hi, i think there is a bug or something in your implementation, because if i want to upload a file here against this API: https://rapidgator.net/article/api/file#upload

v3 - bad memory usage way: I tried all upload ways, SetFileReader, SetFiles and so on

ursp2 := &UploadResponse{}
resp, err := r.Client.Client.R().
    SetHeader("User-Agent", Useragent).
    SetFileReader("file", fileName, rr).
    //SetFiles(map[string]string{
    //  "file": filePath,
    //}).
    SetSuccessResult(ursp2).
    SetErrorResult(&errMsg).
    Post(ursp.Response.Upload.URL)
if err != nil {
    mursp.Error = err
    r.Logger.ErrorLogger.Println(err.Error())
    fmt.Println(err)
    //c <- mursp
    return
}

The upload process load the hole file in memory and that use tons of RAM. I dont know why but i implement a plain multipart file upload like this here, that works fine, without this memory bug.

Could you try to improve your multipart form request?

Your old v0.3.2 works. No extrem RAM usage

imroc commented 1 year ago

I don't think it's possible, SetFileReader and SetFiles will not load the hole file in memory, just copy from reader to server, I'll check it out in a few days when I'm free.

n0v3xx commented 1 year ago

I dont know why but the hole file is loaded to RAM but only for this filehoster. I tested other ones it works with your client. And i noticed that my "working solution" also not work... i dont understand why that happens. If you want i can send you test credentials for this filehoster and the code i use. If you want i send it to you via mail.

n0v3xx commented 1 year ago

I have done some more tests. Your old version v0.3.2 works like a charm, no RAM consumption at all. Your new version does not work. It loads the entire file into RAM. I don't know what you changed, but this is not good.