LoadAndSave around http.FileServer allocates a lot of memory

[wansing]

I use LoadAndSave on a http.Handler which contains an http.FileServer. When a large file is downloaded, the application starts to consume a lot of memory. The memory usage seems to converge towards 5 × filesize. Example:

package main

import (
    "net/http"
    "github.com/alexedwards/scs/v2"
)

func main() {
    var sessionManager = scs.New()
    http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("static"))))
    http.ListenAndServe(":8080", sessionManager.LoadAndSave(http.DefaultServeMux))
}

Create the static dir with a large file and run the code:

$ mkdir static
$ dd if=/dev/zero of=static/myfile bs=1M count=100
$ go run main.go

Download the large file a few times:

$ wget -O /dev/null "http://127.0.0.1:8080/static/myfile"

With the 100 MB file, RES memory allocation reaches 470 MB on my machine. With a 200 MB file, it's 940 MB. Without LoadAndSave, RES memory usage is around 7 MB.

One workaround is to wrap LoadAndSave around selected handlers only. But we couldn't restrict file access based on the session then.

[jum]

The issue here is due to the concept of the cookie subsystem to be able to modify cookie state until all the handler execution is completed, the complete output must be buffered and only after the handler is completed, the eventually modified cookie will be set in the headers of the response and the buffered output is send.

For your use case, you could use a simple middleware of your own that only loads the cookie state to make access decisions depending upon any cookies the client did sent. For static files you could probably do without modifying the cookie state in the client (counting something in your internal database would probably be OK).

[alexedwards]

Yes @jum is spot-on in their explanation.

You could create your own LoadOnly middleware and use that on the static files instead of LoadAndSave. Similar to this:

type MySessionManager struct {
    *SessionManager
}

func (s *MySessionManager) LoadOnly(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        var token string
        cookie, err := r.Cookie(s.Cookie.Name)
        if err == nil {
            token = cookie.Value
        }

        ctx, err := s.Load(r.Context(), token)
        if err != nil {
            s.ErrorFunc(w, r, err)
            return
        }

        next.ServeHTTP(w, r.WithContext(ctx))
}

That will load the session data into the request context, so you can use it to restrict access, but it won't buffer the response data so it should solve the issue you're describing.

(But beware that the session will be 'read-only' in the sense that any changes to the session data won't be persisted unless you manually call Commit).

[wansing]

Thank you very much for the explanation and the example!