search

spacecatninja / craft-imager-x

Image transforms, optimizations and manipulations for your Craft CMS site.
Other
26 stars 16 forks source link

PHP process on 100% CPU and transformations not generated by cron jobs #267

Open luke-nehemedia opened 5 months ago

luke-nehemedia commented 5 months ago

I'm submitting a...

Steps to reproduce

  1. Create a page
  2. Add > 20 images with pre-defined transforms
  3. Save page
  4. (optional: Open page in browser)

Description

I have the problem that the server of a client becomes unresponsive by generating many transforms at the same time. I am looking into possible solutions. I thought about optimizing queue-handling. However, the generation of the transformations will not be executed by queue jobs. I run craft queue/listen -v but cannot see any queues of generating images. However, while observing the processes on the server, I can clearly see that imager is active generating transformations, for example by the cwebp process.

Displayed errors, stack trace, relevant logs

Additional info

aelvan commented 5 months ago

Hi,

In general, all PHP processes will gobble up as much CPU as is available, there's no way to throttle this. A queue runner wouldn't make much of a difference, the actual transform will happen in PHP anyway.

There're a couple of things in your description that's a bit unclear:

PHP process on 100% CPU...

When you say "PHP process", is it the webserver php process (ie php-fpm for instance), or the PHP CLI version?

...the server of a client becomes unresponsive...

Does it become unresponsive forever? Or just while the transforms are being generated? Does it eventually become responsive again.

And since you mention that this happens on a client's server, does that mean that it works as expected locally/in other envs?

Add > 20 images with pre-defined transforms

You mean that you've set ut auto generation for the fields to which you add the images?

If so, the the generation of the transforms will happen via queue jobs, there's no other way Imager will be triggered at this point. If the queue jobs doesn't appear when running queue/listen, something is.. very wrong.

How many transforms per image are you creating? What file formats?

luke-nehemedia commented 4 months ago

Hi

PHP process on 100% CPU...

When you say "PHP process", is it the webserver php process (ie php-fpm for instance), or the PHP CLI version? I think it was the php process of the web server. Right now I am handling all queue jobs via a cron job that runs every minute. This seems to somewhat resolve the issue. However, I also have very limited access to the server, which makes it difficult to investigate this problem.

...the server of a client becomes unresponsive...

Does it become unresponsive forever? Or just while the transforms are being generated? Does it eventually become responsive again.

It takes a between 30 seconds and a couple minutes.

And since you mention that this happens on a client's server, does that mean that it works as expected locally/in other envs? I don't have this particular site locally right now. But I will try to reproduce it. I use your plugin on other sites, too. On those sites (which run on my own servers) I do not experience problems.

Add > 20 images with pre-defined transforms You mean that you've set ut auto generation for the fields to which you add the images?

Yes

If so, the the generation of the transforms will happen via queue jobs, there's no other way Imager will be triggered at this point. If the queue jobs doesn't appear when running queue/listen, something is.. very wrong.

I thought I was doing so, yes. But I just discovered, that I do not have enabled auto-generation at the moment. When generating transforms via the "tools"-page of the plugin, they appear in the queue.

How many transforms per image are you creating? What file formats? It really depends. For a title image it's about 5-6 sizes in JPEG en WebP as well as a lazy-load image as jpeg.