discord / lilliput

Resize images and animated GIFs in Go
https://discord.com/blog/how-discord-resizes-150-million-images-every-day-with-go-and-c
Other
1.96k stars 124 forks source link

Buffer too small #38

Closed theRoughCode closed 5 years ago

theRoughCode commented 6 years ago

Hi team,

I've been encountering this issue when using the library: buffer too small to hold image. Is there a way to know how big of a buffer is needed or a way to bypass passing in a buffer?

Thanks!

brian-armstrong-discord commented 6 years ago

Can you provide a code sample that yields this error?

To answer your question, I don't think there's a good way to know that. It would be dependent on the encoding used and amount of entropy in the image. There is intentionally no method that allocates a buffer because this would create a lot of garbage and slow down Go - buffer reuse is highly recommended.

theRoughCode commented 6 years ago

I totally understand why this was chosen, but it does generate the above error.

newBufferSize := 50 * 1024 * 1024
buffer := make([]byte, newBufferSize)
img.byteArray, err = imgOps.Transform(decoder, options, buffer)

So, currently I have it set to 50MB which is a pretty large size. This works for most images (and is probably a too big of a buffer for a lot of them). However, I've encountered a few edge cases where 50MB is not large enough.

I see that there is a check that ensures that the buffer len is greater than width * height * CV_ELEM_SIZE(pixelType). But CV_ELEM_SIZE is an opencv function. I was wondering if there was a way to know how big of a buffer I'd need to pass in.

brian-armstrong-discord commented 6 years ago

What is in options?

theRoughCode commented 6 years ago
options := &lilliput.ImageOptions{
    FileType:     "." + img.format,
    Width:        img.width,
    Height:       img.height,
    ResizeMethod: resizeMethod,
    EncodeOptions: map[int]int{
        lilliput.JpegQuality:    100,
        lilliput.PngCompression: 0,
    },
}
brian-armstrong-discord commented 6 years ago

Well, I have a feeling that those EncodeOptions are going to make files much larger than they need to be. PngCompression will not affect image quality - pngs are lossless. It'll just change how much cpu time is used. 0 will indeed make the files much larger.

Similarly, 100 JpegQuality would be pretty uncommon. Typical web-quality Jpegs will sit around 75. 85-90 might be more of a "high quality" level. Additional from there will greatly increase the file size for little improvement in quality.

If you do want to use these options, I'd recommend using a larger buffer. These settings will produce very large encoded images.

theRoughCode commented 6 years ago

I see, thanks for the heads-up. So, there is no way to know how big a buffer is needed?

brian-armstrong-discord commented 6 years ago

I'm not sure how you could realistically guess that since image compression is so complex. There might be a maximum possible size for a given resolution, but it's going to be extremely large.