🐛 Fiber Panic when trying to stream multipart/form-data body with RequestBodyStream

[alfaex]

Fiber version v2.30.0

Issue description While studying stream upload multipart/form-data, for multiple large files, (2GB) in a small container with low memory.

I managed to do this with net/http with a docker container running with limited memory (32MB) and it worked, idle was using 5MB and while uploading 3GB files went to 10mb (limited chuncks to 1MB)

On Fiber it kept cashing the container because further reading about fasthttp it's stated that the whole body is read at once. link But at the end it's pointed to a solution StreamRequestBody

Everytime that I build the binary and use it in docker it crashes because it tried to use more than 32MB

The last thing I tried is to copy what net/http MultipartReader does using the c.Context().RequestBodyStream()

Code snippet

Command to run the container

$ go build main.go && docker run --rm -it -p 3000:3000  -m 32m --memory-swap 32m -v $PWD:/fiber ubuntu /fiber/main

To check the memory I used this command.

$ docker stats

Command to send the file

$ curl -v -X POST --form file=@bnn.mkv http://localhost:3000/upload # 2GB file

net/http example

package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
)

func main() {

    uploadHandler := func(w http.ResponseWriter, req *http.Request) {
        reader, err := req.MultipartReader()

        if err != nil {
            fmt.Println("not multipart mime reader:", err)
        }

        for {
            part, err := reader.NextPart()
            if err != nil {
                if err == io.EOF {
                    fmt.Println("EOF")
                    break
                } else {
                    fmt.Println("Other type of err", err)
                    return
                }
            }
            fmt.Println("FILENAME", part.FormName(), part.FileName(), part.Header.Get("Content-Type"))

            saving, err := os.Create(part.FileName())
            if err != nil {
                fmt.Println("not able to create file", err)
            }
            defer saving.Close()

            temp := bufio.NewWriter(saving)
            buffer := make([]byte, 1024*1024)
            for {
                read, err := part.Read(buffer)
                if err == io.EOF {
                    fmt.Println("EOF", err)
                    break
                }
                if err != nil {
                    fmt.Println("Other type of error while saving", err)
                }
                temp.Write(buffer[:read])
            }
            temp.Flush()
        }
        fmt.Fprintf(w, "body request response")
    } // helloHandler

    http.HandleFunc("/upload", uploadHandler)
    log.Fatal(http.ListenAndServe(":3000", nil))
}

Fiber example

Command to run the container

$ go build main.go && docker run --rm -it -p 3000:3000  -m 32m --memory-swap 32m -v $PWD:/fiber ubuntu /fiber/main

When trying to send the file like the previous one the container crashes

Command to send the file

$ curl -v -X POST --form file=@bnn.mkv http://localhost:3000/upload # 2GB file

The error reported from fiber (on docker it don't show the error, only when running go run main.go)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x51b412]

package main

import (
    "fmt"
    "io"
    "mime"
    "mime/multipart"

    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/cors"
)

func main() {

    app := fiber.New()
    app.Server().StreamRequestBody = true
    app.Use(cors.New())

    app.Post("/upload", func(c *fiber.Ctx) error {
        // https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/net/http/request.go;l=467
        v := c.Get("Content-Type")
        if v == "" {
            return nil
        }
        d, params, err := mime.ParseMediaType(v)
        if err != nil || !(d == "multipart/form-data" || d == "multipart/mixed") {
            return nil
        }
        boundary, ok := params["boundary"]
        if !ok {
            return nil
        }
        reader := multipart.NewReader(c.Context().RequestBodyStream(), boundary)
        for {
            part, err := reader.NextPart()
            if err != nil {
                if err == io.EOF {
                    fmt.Println("EOF")
                    break
                } else {
                    fmt.Println("Other type of error", err)
                    return nil
                }
            }
            fmt.Println("FILENAME", part.FormName(), part.FileName(), part.Header.Get("Content-Type"))
        }
        return nil
    })

    app.Listen(":3000")
}

Thanks ❤ Fiber ❤🏆🔥

[welcome[bot]]

Thanks for opening your first issue here! 🎉 Be sure to follow the issue template! If you need help or want to chat with us, join us on Discord https://gofiber.io/discord

[ReneWerner87]

thanks for the report

i think we need a little time for this, as currently many of us are busy with other tasks

but we will take it into account and look at it, just want to say that it may take longer

[bigflood]

@alfaex

you should set DisablePreParseMultipartForm to true

by default, server reads the entire multipart message before calling the handler.

https://github.com/gofiber/fiber/blob/a7032b7a175a5f909413dc43600a8e505dca322e/app_test.go#L1558

Pre-read multipart form data of known length https://github.com/valyala/fasthttp/blob/7a5afddf5b805a022f8e81281c772c11600da2f4/http.go#L1214

[alfaex]

@bigflood it worked amazingly well

Problem solved.

Thanks.

[alexanderbkl]

@alfaex

you should set DisablePreParseMultipartForm to true

by default, server reads the entire multipart message before calling the handler.

https://github.com/gofiber/fiber/blob/a7032b7a175a5f909413dc43600a8e505dca322e/app_test.go#L1558

Pre-read multipart form data of known length https://github.com/valyala/fasthttp/blob/7a5afddf5b805a022f8e81281c772c11600da2f4/http.go#L1214

I had problems uploading files that are other than image or text (pdf, exe, dlls...) but this fixed it all and works wonders. Thank you very much!!!