webp-sh / webp_server_go

Go version of WebP Server. A tool that will serve your JPG/PNG/BMP/SVGs as WebP/AVIF format with compression, on-the-fly.
https://docs.webp.sh
GNU General Public License v3.0
1.79k stars 173 forks source link

When the instantaneous request volume is large, a large amount of CPU and memory will be consumed. #306

Closed mikusaa closed 8 months ago

mikusaa commented 8 months ago

Describe the bug When the instantaneous request volume is large, a large amount of CPU and memory will be consumed.

To Reproduce It seems that when using Webpgo on my blog, which includes webp, jpg, and png images, a fresh installation of Webpgo may experience delays in compressing images due to a high request volume. This delay is causing a significant increase in CPU and memory usage.

here is the compose.yml

  webpgo:
    image: webpsh/webp-server-go
    environment:
      - MALLOC_ARENA_MAX=1
      - TZ=Asia/Shanghai
    volumes:
      - ./webpgo/config.json:/etc/config.json
      - ./static:/opt/pics
      - ./webpgo/exhaust:/opt/exhaust
      - ./webpgo/metadata:/opt/metadata
    ports:
      - 3333:3333
    restart: unless-stopped
    container_name: webpgo

and the config.json

{
    "HOST": "0.0.0.0",
    "PORT": "3333",
    "QUALITY": "75",
    "IMG_PATH": "/opt/pics",
    "IMG_MAP": {},
    "EXHAUST_PATH": "./exhaust",
    "ALLOWED_TYPES": ["jpg","png","jpeg","bmp","gif","svg","webp","heic"],
    "ENABLE_AVIF": true,
    "ENABLE_EXTRA_PARAMS": true
  }

Expected behavior A clear and concise description of what you expected to happen.

Screenshots and logs HTOP image VNC image

Then, my server becomes unresponsive.

Environment (please complete the following information):

Additional context None

BennyThink commented 8 months ago

some workarounds:

  1. limit resource usage in docker-compose.yml
  2. prefetch first

We may not have enough bandwidth to add -jobs parameters for normal conversion workflow. However this is a long term solution.