search

plan2net / webp

Create a WebP copy for images (TYPO3 CMS)
GNU General Public License v3.0
60 stars 33 forks source link

Prevent high server load by (i. e.) set a limit of how many files at once in queue? #17

Open g3ib opened 5 years ago

g3ib commented 5 years ago

The server load is very high (6 cores, 32gb) and almost on 100% and gets slow. Maybe it could be added a scheduler (?) to let the extension execute the processing only at night?

wazum commented 5 years ago

Thanks for your feedback, good idea!

g3ib commented 5 years ago

Or that you could say only n images per hour or whatever. The thing is that the server's load ist too high over the time. The client that I talk about has as almost 260.000 images in fileadmin/processed/, so we if it were only 10 images, this problem would never appear. I hope you understand my problem.

I also tried and set MAGICK_THREAD_LIMIT to 1, but it had no effect, see here: https://www.imagemagick.org/script/resources.php#environment

I am not sure what the best way would be...

But to clean the entire processed-directory is not a solution (in this case).

This is a general problem and would prevent the usage of this extension in other projects too, so a solution for that would be great!

the-hotmann commented 5 years ago

The server load is very high (6 cores, 32gb) and almost on 100% and gets slow. Maybe it could be added a scheduler (?) to let the extension execute the processing only at night?

I like that Idea, but I also think limiting sheduling to a time is very very bad. Things never should be timebased..

I think implementing a constat which will set 2 variables to reduce CPU usage would be enough:

MAGICK_THROTTLE=50 convert [...] and MAGICK_THREAD_LIMIT=1 convert [...]

should do the job.

At MAGICK_THREAD_LIMIT=1 convert [...] I would just take 50% of the CPU-Cores. E.g: 16 Cores * 50% = 8 Cores (the "50%" should be the variable we could set, and it rounds it to the next Core)

Example: variable set to 75% and Server has 9 Cores. 9 Cores * 75% = 6,75 Cores ≈ 7 Cores MAGICK_THREAD_LIMIT=7 convert [...] So you can set your CPU Usage (exactly at whole CPU Cores)

OR (if possible) gettingt he info how many cores the system has and letting the user decide how many to use

@sgatonmpunktde : which IM version are you running? v6 btw has a lot of performance issues which are resolved in v7.

g3ib commented 5 years ago

First of all, thanks for your support!

I tried only using MAGICK_THREAD_LIMIT=1, but it had not really an effect. I can try MAGICK_THROTTLE...

@M4rt1n17

Version: ImageMagick 6.8.9-9 Q16 x86_64 2018-09-28 Features: DPC Modules OpenMP Delegates: bzlib cairo djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png rsvg tiff wmf x xml zlib

UPDATE @wazum @M4rt1n17

I tested now with the following env variables:

It works BUT has a big big weak in my opinion: It pushes convert-commands to the queue and I watched this via htop (ssh). The average load is okay at the beginning but I guess when a bigger image comes that needs more time, the queue gets longer and longer and there is no command like "wait until the slow convert process is ready"... The effect is comparable with a "traffic jam without a reason". Just because the converts are not equal fast. I hope you understand my point. So for now my only solution is to disable the extension and wait (watching htop) for the average load to get down again.

This is a screenshot from htop. I had to disable the extension again.

grafik

the-hotmann commented 5 years ago

Version: ImageMagick 6.8.9-9 Q16 x86_64 2018-09-28 Features: DPC Modules OpenMP Delegates: bzlib cairo djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png rsvg tiff wmf x xml zlib

Is there a reason for not having "webp" in your delegates? Maybe without webp your ImageMagick has to use other convertion methods.. In my ImageMagick (version 7 btw!) webp is included .. could you try

  1. update your IM to the latest possible version (at least 6.9.10-44)
  2. try to compile it by yourself (with make etc) and with webp included

BTW this is my IM:

Version: ImageMagick 7.0.8-27 Q16 x86_64 2019-02-10 https://imagemagick.org
Copyright: © 1999-2019 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenMP
Delegates (built-in): bzlib djvu fftw fontconfig freetype gvc jbig jng jp2 jpeg lcms lqr lzma openexr pangocairo png tiff webp wmf x xml zlib

and I'm running Debian 9. (also possible at debian 8)

If you want to have my code to install always the latest version (automated) pls use this: mkdir /imgmg && cd /imgmg && apt-get install webp libwebp-dev build-essential checkinstall -y && apt update && apt build-dep imagemagick -y && wget https://www.imagemagick.org/download/ImageMagick.tar.gz && tar xf ImageMagick.tar.gz && cd ImageMagick-* && ./configure && make && make install && checkinstall -y && ldconfig && rm -r /imgmg && convert -version

Hope this helps.

g3ib commented 5 years ago

@M4rt1n17

My server is an nginx with plesk on it. I have no idea why webp is not listed, but it works ^^

Upgrading IM to v7 is currently not an option because I don't want to "kill" the website while it is running. But I want to put this on my to-do-list for later.

What I tried now I added two parameter to the default convert command, so it looks like this now:

-quality 95 -define webp:lossless=false webp:thread-level=1 webp:low-memory=true

I am watching the load but it seems this solves my problem for now.

Maybe it would be a good idea (for other users) to use this as default in the project. I guess I am not the only one with this issue. Just to prevent too much load.

When I update IM to v7 maybe I can test this again and write something additionally...

UPDATE A few more words/thoughts/ideas from my side:

the-hotmann commented 5 years ago

Maybe it would be a good idea (for other users) to use this as default in the project. I guess I am not the only one with this issue. Just to prevent too much load.

Nice to see a way to prevent high Server load, but I preffer it as for me having the job done faster is more important then Serverload as I'm on a Dedicated really big Server, But a option which can enable this option would be nice, thats true.

akiessling commented 3 years ago

Any news on this feature?

I like that Idea, but I also think limiting sheduling to a time is very very bad. Things never should be timebased..

It's not just about having the conversion at night, you could also setup a scheduler task to convert new images every minute. IMHO it is more about having a process async to the page creation, so when a user edits a page and wants to preview it, he won't have to wait for a minute to get the page rendered due to multiple picture tags wit a lot of image sources and then the webp conversion additionally kicks in. Since the idea of having a rewrite rule to deliver jpg/png or webp fits the async conversion perfectly, i'd welcome a scheduler task very much :)

mathias-heilmann commented 2 years ago

Any update here for a scheduler tasks? This process has still a high server load.

dennismetz commented 1 year ago

I think the idea with the scheduler task is great, is there any news on this yet?

We generate about 25 more images per image (5 ViewPorts, 5 different resolutions). The generation sometimes takes a really long time...

dennismetz commented 1 year ago

Hi @wazum, I hope I am not annoying, but I would like to ask again for an update. The extension is really great and I love using it, so it would be great if the feature would come.