Webp v0.9.3 is consuming a lot of memory again

[Ingiboy]

Webp used both for converting to webp and resizing pictures. ~83k requests per day. Щn version 0.8.4 was consistently below 0.5gb

docker-compose.yml

version: "3.3"

services:
  webp:
    container_name: webp
    image: webp:alpine
    build:
      context: webp
      dockerfile: Dockerfile
      args:
        - UID=1001
        - GID=1001
        - REPO=https://github.com/webp-sh/webp_server_go.git
        - TAG=0.9.3
    restart: unless-stopped
    user: 1001:1001
    cap_drop:
      - ALL
    environment:
      - MALLOC_ARENA_MAX=1
      - LD_PRELOAD=/usr/lib/libjemalloc.so.2
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./webp/config.json:/etc/config.json:ro
      - /var/tmp/webp:/opt
    ports:
     - 127.0.0.1:3335:3333

Dockerfile:

FROM alpine:3.18 as builder

ARG REPO
ARG TAG

RUN set -ex && \
    apk --update upgrade --no-cache && \
    apk --update add --no-cache git go vips-dev && \
    mkdir /build && \
    cd /build && \
    git clone ${REPO} . && \
    git checkout tags/${TAG} && \
    go mod download && \
    go build -ldflags="-s -w" -o webp-server .

FROM alpine:3.18

ARG UID
ARG GID

RUN set -ex && \
    addgroup -g ${GID} user && adduser -u ${UID} -G user -s /bin/sh -D user && \
    apk --update upgrade --no-cache && \
    apk --update add --no-cache vips ca-certificates jemalloc

COPY --from=builder /build/webp-server  /usr/bin/webp-server
COPY --from=builder /build/config.json /etc/config.json

WORKDIR /opt

CMD ["/usr/bin/webp-server", "--config", "/etc/config.json"]

config.json:

{
  "HOST": "0.0.0.0",
  "PORT": "3333",
  "QUALITY": "80",
  "MAX_JOB_COUNT": "4",
  "IMG_PATH": "http://nginx:8080",
  "EXHAUST_PATH": "./exhaust",
  "ALLOWED_TYPES": ["jpg","png","jpeg","bmp"],
  "ENABLE_EXTRA_PARAMS": true,
  "ENABLE_AVIF": false
}

[n0vad3v]

Thanks for reporting, comparing 0.8.4 to 0.9.3 https://github.com/webp-sh/webp_server_go/compare/0.8.4...0.9.3, there are two possible position that might lead to this problem.

• One is GIF support https://github.com/webp-sh/webp_server_go/commit/90680bef6cefd5eae1d6bc16740a7ba7d1154041 that changes LoadImageFromFile's param
• One is our refactor: https://github.com/webp-sh/webp_server_go/commit/23bbed8ce6e15e39e8f124a0f671a90b96798a79

Could you please have a test using our official image and to see if this problem persists? (To mitigate possible alpine problems) (BTW, what's the main reason for you to build image yourself?

@BennyThink I think we might need a way to help test mem usage on each version, maybe can be triggered by make mem_test, this command should:

• Generate lots of images
• Send lots of hybrid requests to server
• Render RAM usage chart

@Ingiboy How is your RAM usage chart in this issue genereated?

[Ingiboy]

Ram usage chart: docker stats webp --no-stream | grep webp | awk '{print $4}' >>log.txt every 5 min and make diagram in google sheets. On my server webp converts jpeg and png only, no gif/apng/bmp/etc.

I use alpine for resource saving and safety reasons. As I recall, some of the debian/ubuntu based images did not run with cap_drop: - ALL unlike alpine. I will test your image and report back. Thanks for the quick reply!

[Ingiboy]

The official image exceeded 1gb in less than an hour of work

[n0vad3v]

@Ingiboy Thanks for you feedback, we're looking at this issue now.

[n0vad3v]

I've done some testing on some of the versions, and found that this might be the problem that only occurs after 0.8.4, the test procedure as follows:

1. psrecord for monitoring CPU and RAM usage, example command psrecord 2783701 --plot plot.png
2. Some sample images totaling of 2.4G
3. Using find * -type f -print | xargs -I {} echo "curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/{}" under ./pics for generating test command, this command should output lots of curl command like below:

   curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05953.jpg
   curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05954.jpg
   curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05955.jpg
   curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05956.jpg
   curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05957.jpg
   curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05958.jpg
   ...

Here are the test results:

0.8.4 with jemalloc

0.9.1 with jemalloc

0.9.3 with jemalloc

0.9.4 with jemalloc

0.9.4 with jemalloc and ReductionEffort set to 0

From the images above we can see after 0.9, the RAM usage is only about 300MB bigger than 0.8.x.(1200MB vs 1600MB)

Also we found out that after 0.9.x, with the introduction of ReductionEffort set to 4 (to solve issue https://github.com/webp-sh/webp_server_go/issues/234), the convertion time has been slowed down to 3 times.

Future steps might be:

• Setup tests to cover all versions and to automate tests on RAM and CPU usage to prevent future versions have worse performance
• Change ReductionEffort to 0 by default, and if it failed to convert, retry using 4, for better performance
• Dig in more on the encode error when ReductionEffort is 0 on some files.

(And there might be a possibility that ReductionEffort set to 4 causes more RAM usage.

[n0vad3v]

@Ingiboy We've released 0.9.5 to set ReductionEffort to 0 bydefault, would you like to have a test to see if the RAM usage issue mitigated?

[Ingiboy]

@n0vad3v thanks! I'll try to run it on the prod today-tomorrow.

[Ingiboy]

I tested v0.9.4 for 3 days, now run 0.9.7

[Ingiboy]

0.9.8 looks like best version

[n0vad3v]

@Ingiboy That's great to hear.

In the meantime, we've built a benchmark framework(some how), and done some benchmarks on convert performance and RAM usage from 0.8.0 to 0.9.8.

Benchmark result: https://docs.webp.sh/benchmark/ Benchmark framework: https://github.com/webp-sh/webp_bench

[Ingiboy]

I made a test bench too. But tests and production are very different, the test one is stable and consumes a lot of memory, while the production one has incomprehensible spikes in memory consumption under fairly stable small load.

[n0vad3v]

But tests and production are very different

Agree, do you have some idea on how to improve theses tests?(Maybe on https://github.com/webp-sh/webp_bench), as theses tests are quite important in my view, many times when we are adding new features, we need to ensure that there are no bug regressions or performance declines.

[Ingiboy]

@n0vad3v, I apologize for such a late reply. About a month ago I accepted a job offer and for the next few months I will hardly be able to take an active part in the development of your very interesting and important project. The previous stack remains in my support but without active development for some time.

I will continue to follow your project anyway, in my opinion, it implements a very necessary functionality. I really hope that you will be able to commercialize it, it deserves it.