beego / beego

beego is an open-source, high-performance web framework for the Go programming language.
Other
31.57k stars 5.62k forks source link

Weird problems with multiple file uploads #5030

Closed cgrard closed 2 years ago

cgrard commented 2 years ago

In a nutshell, I'm experiencing issues with file uploads, at a certain point some files are received with a 0 bytes size but it seems erratic and random.

My use case is : the user selects multiple files and send them over so I can work with them server side, pretty basic. That being said, there could be many files (like a thousand) and the overall size may be around 2GB so I set the MaxMemory accordingly (the server it is running on has plenty), the issue I'm facing is that it does not work in a consistent way and it ends up with files randomly having a null size and I'm really having a hard time understanding the cause so hopefully you guys will be able to point out my mistake or point me in the right direction.

Here is a sample code of what I'm doing:

package controllers

import (
    "fmt"

    "github.com/astaxie/beego"
)

type MainController struct {
    beego.Controller
}

func (c *MainController) Get() {
    c.TplName = "index.html"
    c.Render()
}

func (c *MainController) Post() {
    var err error

    beego.BConfig.MaxMemory = 3221225472 //3GB
        beego.BConfig.MaxUploadSize = 3221225472 //3GB

    c.Ctx.Request.ParseMultipartForm(beego.BConfig.MaxMemory)
    mf := c.Ctx.Request.MultipartForm

    //files := mf.File["upload[]"]
        files, err := c.GetFiles("upload[]")

    if err != nil {
        fmt.Println("GetFiles error: " + err.Error())
    }

    for _, file := range files {
        _, err := file.Open()

        if err != nil {
            fmt.Println("File open error: " + err.Error())
        }

        if file.Size == 0 {
            // not good
            fmt.Println(file.Filename)
            fmt.Println(file.Header)
            fmt.Println(file.Size)
        }
    }
    c.Redirect("/", 200)
}

Go is 1.18.3 (but it works the same on 1.18.4) and bee is 2.0.4 on Linux (Ubuntu 20.04 and CentOS 8) amd64 processor

flycash commented 2 years ago

Did you see any error information? From your demo, it looks like you are using http.Request directly and it's out of Beego scope. Did you call Beego API to parse the form or store the files, for example, Controller.SaveToFile?

I have a suggestion that in your case you'd better use OSS, and those cloud service provider like Azure providing OSS. Here are my points:

github-actions[bot] commented 2 years ago

This issue is inactive for a long time.