search

webp-sh / webp_server_go

Go version of WebP Server. A tool that will serve your JPG/PNG/BMP/SVGs as WebP/AVIF format with compression, on-the-fly.
https://docs.webp.sh
GNU General Public License v3.0
1.79k stars 174 forks source link

Error while converting large images #234

Closed astrowq closed 1 year ago

astrowq commented 1 year ago

Describe the bug

The server cannot convert an image of large size (14.9 MB).

To Reproduce

Try to convert this figure with the docker deployment option: https://inspirehep.net/files/3ce7ad44b03d925850182b268e87487e.png.

It returns 502 error code; the server exits and restarts (with the restart: always policy on)

Expected behaviour

It should convert the image and return the converted one.

Screenshots and logs

Webp Server Go is Running on http://0.0.0.0:3333
time="2023-06-17 21:37:22" level=info msg="Remote Addr is https://inspirehep.net/files/3ce7ad44b03d925850182b268e87487e.png, fetching info..." func="[118:main.proxyHandler()]"
2023/06/17 21:37:22 [VIPS.info] vips__open_image_write: opening with O_TMPFILE
2023/06/17 21:37:22 [VIPS.info] vips__open_image_write: O_TMPFILE failed!
2023/06/17 21:37:22 [VIPS.info] vips__open_image_write: simple open

Environment (please complete the following information):

Additional context

The problem is with libvips, apparently. The image is fetched from the remote backend successfuly and can be found in the folder /opt/remote-raw.

I did a simple search but failed to find a solution.

Also note that some other large image files have no problem of being processed, e.g. 6 MB (https://inspirehep.net/files/3afe268881ca91f21167f92921656e48)

n0vad3v commented 1 year ago

Hmm, that's indeed some problem with libvips, I can reproduce this issue locally:

INFO[2023-06-18 06:22:49][118:main.proxyHandler()] Remote Addr is https://inspirehep.net/files/3ce7ad44b03d925850182b268e87487e.png, fetching info... 
INFO[2023-06-18 06:22:51][133:main.proxyHandler()] Remote file not found in remote-raw path, fetching... 
2023/06/18 06:23:28 [VIPS.info] vips__open_image_write: opening with O_TMPFILE
2023/06/18 06:23:28 [VIPS.info] vips__open_image_write: O_TMPFILE failed!
2023/06/18 06:23:28 [VIPS.info] vips__open_image_write: simple open
WARN[2023-06-18 06:23:34][210:main.webpEncoder()] Can't encode source image: vips2webp: unable to encode

Stack:
goroutine 53 [running]:
runtime/debug.Stack()
    /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/davidbyttow/govips/v2/vips.handleVipsError()
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:38 +0x57
github.com/davidbyttow/govips/v2/vips.handleSaveBufferError(0xc000086960?)
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:31 +0x25
github.com/davidbyttow/govips/v2/vips.vipsSaveToBuffer({0x7f6d900061b0, 0x0, 0x2, 0x0, 0x1, 0x50, 0x0, 0x0, 0x1, 0x0, ...})
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:465 +0xb7
github.com/davidbyttow/govips/v2/vips.vipsSaveWebPToBuffer(0x7f6d900061b0, {0x1, 0x50, 0x0, 0x0, 0x0, {0x0, 0x0}})
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:386 +0x1f8
github.com/davidbyttow/govips/v2/vips.(*ImageRef).ExportWebp(0xc00010e0a0, 0x61?)
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/image.go:924 +0xc5
main.webpEncoder({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, 0x50, {0x91b8f0?, 0x0?})
    /home/nova/Desktop/webp_server_go/encoder.go:202 +0x29a
main.convertImage({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, {0x877eae, 0x4}, {0x0?, 0x0?})
    /home/nova/Desktop/webp_server_go/encoder.go:98 +0x2db
main.convertFilter.func2()
    /home/nova/Desktop/webp_server_go/encoder.go:55 +0x5c
created by main.convertFilter
    /home/nova/Desktop/webp_server_go/encoder.go:54 +0x215
 to WebP 
ERRO[2023-06-18 06:23:34][57:main.convertFilter.func2()] vips2webp: unable to encode

Stack:
goroutine 53 [running]:
runtime/debug.Stack()
    /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/davidbyttow/govips/v2/vips.handleVipsError()
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:38 +0x57
github.com/davidbyttow/govips/v2/vips.handleSaveBufferError(0xc000086960?)
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:31 +0x25
github.com/davidbyttow/govips/v2/vips.vipsSaveToBuffer({0x7f6d900061b0, 0x0, 0x2, 0x0, 0x1, 0x50, 0x0, 0x0, 0x1, 0x0, ...})
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:465 +0xb7
github.com/davidbyttow/govips/v2/vips.vipsSaveWebPToBuffer(0x7f6d900061b0, {0x1, 0x50, 0x0, 0x0, 0x0, {0x0, 0x0}})
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:386 +0x1f8
github.com/davidbyttow/govips/v2/vips.(*ImageRef).ExportWebp(0xc00010e0a0, 0x61?)
    /home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/image.go:924 +0xc5
main.webpEncoder({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, 0x50, {0x91b8f0?, 0x0?})
    /home/nova/Desktop/webp_server_go/encoder.go:202 +0x29a
main.convertImage({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, {0x877eae, 0x4}, {0x0?, 0x0?})
    /home/nova/Desktop/webp_server_go/encoder.go:98 +0x2db
main.convertFilter.func2()
    /home/nova/Desktop/webp_server_go/encoder.go:55 +0x5c
created by main.convertFilter
    /home/nova/Desktop/webp_server_go/encoder.go:54 +0x215
BennyThink commented 1 year ago

By configuring ReductionEffort we can resolve this. The root cause should be your kernel/OS doesn't support O_TMPFILE.

level 2: 5.9s, 7.6MB level 4: 26s 6.9MB

This will be fixed in the coming version.

astrowq commented 1 year ago

Hi, I have tried the version 0.9.0. Unfortunately, the same error still exists for the image file with size of 14.9 MB.

I am using the official docker image. Any idea on how to resolve the point you mentioned that kernel/OS doesn't support O_TMPFILE?

Many thanks.

BennyThink commented 1 year ago

ohh ReductionEffort should be set to 4(default value) for better compatibility.

n0vad3v commented 1 year ago

@astrowq We've released 0.9.1 that could fix this problem, I've tested locally(using docker container) and it's working now .

astrowq commented 1 year ago

It is working on my server too, after upgrading to 0.9.1. Thank you very much for the quick fix!

n0vad3v commented 1 year ago

Hi @astrowq ,we've released https://github.com/webp-sh/webp_server_go/releases/tag/0.9.5 that will use ReductionEffort of 0 for most images for better speed, and will automatically fallback to ReductionEffort of 4 when facing some big images(if encode error happens).

From local tests the new version should work well, would you like to upgrade to this version and provide some feedback for us?

astrowq commented 1 year ago

Hi, it still works smoothly. Unfortunately, I don't monitor the performance metrics for it yet. I can't provide performance comparison.

Thanks for the nice work. Keep it going!

n0vad3v commented 1 year ago

@astrowq Nice, thanks for feedback!

n0vad3v commented 1 year ago

@astrowq We've released 0.9.7 that optimized even better on this, maybe you can try update to that.

BTW, since your image address seems are publicly available, do you have interest to try out our newly built service called WebP Cloud, a SaaS version of WebP Server Go that without the burden of maintaining server yourself.

I've create a demo with address: https://a4970ae.webp.ee (Quality: 80%), for example, given the original image at https://inspirehep.net/files/3ce7ad44b03d925850182b268e87487e.png, you can use https://a4970ae.webp.ee/files/3ce7ad44b03d925850182b268e87487e.png for optimized image, and https://a4970ae.webp.ee/files/3ce7ad44b03d925850182b268e87487e.png?width=400 for thumbnails. 😃

WebP Cloud Dashboard: https://dashboard.webp.se/ WebP Cloud Documentation: https://docs.webp.se/webp-cloud/

astrowq commented 1 year ago

Thanks for the offer, these images are a third party service we are using as part of our service. We are happy with our current setup.

I will upgrade to the newest version soon. Thank you very much.