Weird problems with multiple file uploads

[cgrard]

In a nutshell, I'm experiencing issues with file uploads, at a certain point some files are received with a 0 bytes size but it seems erratic and random.

My use case is : the user selects multiple files and send them over so I can work with them server side, pretty basic. That being said, there could be many files (like a thousand) and the overall size may be around 2GB so I set the MaxMemory accordingly (the server it is running on has plenty), the issue I'm facing is that it does not work in a consistent way and it ends up with files randomly having a null size and I'm really having a hard time understanding the cause so hopefully you guys will be able to point out my mistake or point me in the right direction.

Here is a sample code of what I'm doing:

package controllers

import (
    "fmt"

    "github.com/astaxie/beego"
)

type MainController struct {
    beego.Controller
}

func (c *MainController) Get() {
    c.TplName = "index.html"
    c.Render()
}

func (c *MainController) Post() {
    var err error

    beego.BConfig.MaxMemory = 3221225472 //3GB
        beego.BConfig.MaxUploadSize = 3221225472 //3GB

    c.Ctx.Request.ParseMultipartForm(beego.BConfig.MaxMemory)
    mf := c.Ctx.Request.MultipartForm

    //files := mf.File["upload[]"]
        files, err := c.GetFiles("upload[]")

    if err != nil {
        fmt.Println("GetFiles error: " + err.Error())
    }

    for _, file := range files {
        _, err := file.Open()

        if err != nil {
            fmt.Println("File open error: " + err.Error())
        }

        if file.Size == 0 {
            // not good
            fmt.Println(file.Filename)
            fmt.Println(file.Header)
            fmt.Println(file.Size)
        }
    }
    c.Redirect("/", 200)
}

Go is 1.18.3 (but it works the same on 1.18.4) and bee is 2.0.4 on Linux (Ubuntu 20.04 and CentOS 8) amd64 processor

[flycash]

Did you see any error information? From your demo, it looks like you are using http.Request directly and it's out of Beego scope. Did you call Beego API to parse the form or store the files, for example, Controller.SaveToFile?

I have a suggestion that in your case you'd better use OSS, and those cloud service provider like Azure providing OSS. Here are my points:

• In modern web application, we don't want to support upload files because it has significant impact on web application
• OSS provides fault tolerance, authorization...

[github-actions[bot]]

This issue is inactive for a long time.