Memory not released to OS

[hamochi]

Hi, I'm working on a resize server that creates a few thumbnails and uploads them to S3, and when I started to load test my app I noticed that the memory is not release back to the OS. I've tried this both inside a docker container and without docker on and Amazon EC2 m4.xlarge instance with 4 virtual cores and 16 BG of memory.

Here is a simplified version of what I'm basically doing..

I'm using Vips 8.7.0

My code and Docker file can be found here: https://github.com/hamochi/bimg-memory-issue

func main() {
    for i := 0; i < 5; i++ {
        DoSomeWork(i)
        time.Sleep(10 * time.Minute)
    }
    <-time.After(20 * time.Hour)

}

func DoSomeWork(name int) {
    fmt.Printf("starting cycle %v\n", name)
    wg := sync.WaitGroup{}
    for i := 1; i < 101; i++ {
        wg.Add(6)
        go resize(fmt.Sprintf("./img/%v.jpg", i), 2300, 1533, &wg)
        go resize(fmt.Sprintf("./img/%v.jpg", i), 1920, 1280, &wg)
        go resize(fmt.Sprintf("./img/%v.jpg", i), 1800, 1200, &wg)
        go resize(fmt.Sprintf("./img/%v.jpg", i), 1400, 933, &wg)
        go resize(fmt.Sprintf("./img/%v.jpg", i), 900, 600, &wg)
        go resize(fmt.Sprintf("./img/%v.jpg", i), 600, 400, &wg)
    }
    wg.Wait()
    fmt.Printf("done with cycle %v\n", name)
    debug.FreeOSMemory()
}

func resize(filename string, width int, height int, wg *sync.WaitGroup) {
    defer wg.Done()

    buffer, err := bimg.Read(filename)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
    }

    _, err = bimg.NewImage(buffer).Resize(width, height)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
    }
}

[ajbeach2]

Can you add the following to your running server? This may actually be a libvips issue, but you can run pprof on a running server:

import _ "net/http/pprof"
go func() {
    log.Println(http.ListenAndServe("localhost:6060", nil))
}()

EDIT: https://golang.org/pkg/net/http/pprof/

You can dump the heap profile with:

go tool pprof http://localhost:6060/debug/pprof/heap

[dgozick]

I had the same issue, set these variables in main before resizing

func main() {
    bimg.VipsCacheSetMax(0)
    bimg.VipsCacheSetMaxMem(0)
    ...
}

[fuji246]

I have the same issue, pprof seems only shows the heap used in go code.

[fuji246]

@hamochi , try this, the image is allocated should be unref, otherwise it will leak.

func resize(filename string, width int, height int, wg *sync.WaitGroup) {
    defer wg.Done()

    buffer, err := bimg.Read(filename)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
    }

    image, err = bimg.NewImage(buffer).Resize(width, height)
        defer C.g_object_unref(C.gpointer(image))
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
    }
}

[alikhil]

Try to set environment variable MALLOC_ARENA_MAX=2. It helped in my case. Source

[faesslerpascal]

@alikhil Thank you so much. This solved my memory issue.