search

davidbyttow / govips

A lightning fast image processing and resizing library for Go
MIT License
1.31k stars 203 forks source link

Memory overflow when convert large size jpeg to avif #444

Closed Hanmo123 closed 3 months ago

Hanmo123 commented 3 months ago

I tried to convert a image(from Hasselblad x2d, 100 million pixels, about 65MB, you can download here) to avif using codes below:

package main

import (
    "os"

    "github.com/davidbyttow/govips/v2/vips"
)

func main() {
    vips.Startup(nil)
    defer vips.Shutdown()

    image1, _ := vips.NewImageFromFile("thomas-x2d-xcd-25v-1.jpg")
    ep := vips.NewAvifExportParams()
    imageBytes, _, _ := image1.ExportAvif(ep)
    os.WriteFile("output.avif", imageBytes, 0644)
}

it takes more than 13GB memory and finally OOM: image

why i suppose it is abnormal:

import sharp from 'sharp';
sharp('thomas-x2d-xcd-25v-1.jpg').jpeg().toFile('output.avif')
2024/08/09 17:39:44 [VIPS.info] g_getenv( "PATH" ) == "/home/hanmo/.vscode-server/bin/f1e16e1e6214d7c44d078b1f0607b2388f29d729/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files (x86)/ZeroTier/One/:/mnt/c/Program Files/Microsoft VS Code/bin:/mnt/c/Users/13749/AppData/Local/Microsoft/WindowsApps"
2024/08/09 17:39:44 [VIPS.info] looking in "/home/hanmo/.vscode-server/bin/f1e16e1e6214d7c44d078b1f0607b2388f29d729/bin/remote-cli" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/usr/local/sbin" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/usr/local/bin" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/usr/sbin" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/usr/bin" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/sbin" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/bin" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/usr/games" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/usr/local/games" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/usr/lib/wsl/lib" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Windows/system32" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Windows" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Windows/System32/Wbem" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Windows/System32/WindowsPowerShell/v1.0/" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Windows/System32/OpenSSH/" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Program Files/dotnet/" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Program Files (x86)/ZeroTier/One/" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Program Files/Microsoft VS Code/bin" for "govips"
2024/08/09 17:39:44 [VIPS.info] looking in "/mnt/c/Users/13749/AppData/Local/Microsoft/WindowsApps" for "govips"
2024/08/09 17:39:44 [VIPS.info] trying for dir = "/data/image-worker/govips", name = "govips"
2024/08/09 17:39:44 [VIPS.info] canonicalised path = "/data/image-worker"
2024/08/09 17:39:44 [VIPS.info] VIPS_PREFIX = /usr
2024/08/09 17:39:44 [VIPS.info] VIPS_LIBDIR = /usr/lib/x86_64-linux-gnu
2024/08/09 17:39:44 [VIPS.info] prefix = /usr
2024/08/09 17:39:44 [VIPS.info] libdir = /usr/lib/x86_64-linux-gnu
2024/08/09 17:39:44 [VIPS.info] searching "/usr/lib/x86_64-linux-gnu/vips-plugins-8.12"
2024/08/09 17:39:44 [govips.info] vips 8.12.1-Wed Feb  2 14:43:28 UTC 2022 started with concurrency=22 cache_max_files=0 cache_max_mem=0 cache_max=0
2024/08/09 17:39:44 [govips.info] registered image type loader type=png
2024/08/09 17:39:44 [govips.info] registered image type loader type=svg
2024/08/09 17:39:44 [govips.info] registered image type loader type=heif
2024/08/09 17:39:44 [govips.info] registered image type loader type=heif
2024/08/09 17:39:44 [govips.info] registered image type loader type=magick
2024/08/09 17:39:44 [govips.info] registered image type loader type=jpeg
2024/08/09 17:39:44 [govips.info] registered image type loader type=pdf
2024/08/09 17:39:44 [govips.info] registered image type loader type=tiff
2024/08/09 17:39:44 [govips.info] registered image type loader type=webp
2024/08/09 17:39:44 [govips.info] registered image type loader type=jp2k
2024/08/09 17:39:44 [govips.info] registered image type loader type=gif
vips cache+: jpegload_buffer buffer=VIPS_TYPE_BLOB, data = 0xc000200000, length = 67527869 out=((VipsImage*) 0x1502020) flags=((VipsForeignFlags) VIPS_FOREIGN_SEQUENTIAL) fail=TRUE -
vips cache+: cast in=((VipsImage*) 0x1502020) out=((VipsImage*) 0x15021b0) format=((VipsBandFormat) VIPS_FORMAT_UCHAR) -
vips cache : copy in=((VipsImage*) 0x15021b0) out=((VipsImage*) 0x1502340) -
vips cache+: linecache in=((VipsImage*) 0x15027f0) out=((VipsImage*) 0x1502b10) tile-height=8 access=((VipsAccess) VIPS_ACCESS_SEQUENTIAL) -
vips cache+: sequential in=((VipsImage*) 0x15027f0) out=((VipsImage*) 0x1502980) tile-height=8 -
vips cache+: extract_area input=((VipsImage*) 0x1502980) out=((VipsImage*) 0x1502ca0) left=0 top=0 width=11657 height=8741 -
2024/08/09 17:39:44 [VIPS.info] vips__open_image_write: opening with O_TMPFILE
2024/08/09 17:39:44 [VIPS.info] vips__open_image_write: O_TMPFILE failed!
2024/08/09 17:39:44 [VIPS.info] vips__open_image_write: simple open
signal: killed
tonimelisma commented 3 months ago

Hey @Hanmo123. Thanks for the issue report. I believe it's probably not a memory leak but instead just has to do with how slow golang's garbage collector works. Try running short-running goroutines that handle the images for instance, or use the gc-functions in the golang standard library.