High CPU usage related to rocket_saas_warmup async process

Tabrisrp commented 3 years ago

Describe the bug We’re getting some reports about high CPU usage with RUCSS enabled, notably on Kinsta/WP Engine, and both pointing to the asynchonous process of the resource fetcher.

As a reminder, this process is triggered asynchronously on rocket_buffer, so running on uncached pages during the caching process. the AJAX action is rocket_saas_warmup

From their logs, some of those calls are running for too long, up to 6 min.

Looking at the code, during this call, we are:

parsing the HTML for CSS/JS files
for each file, we are getting the URL, path, content, type (and media value for CSS)
For the content itself, we are getting it from the local file, or from the remote file if it’s an external path, and then we are Minifying it
We finally send those to the background process which sends to warmup
I believe what would be causing the issue here is the content fetching + minifying step. Depending on the number of files found, it means a loop with a lot of processing in it.

I’m thinking this specific part should be delegated to the background processing task instead, as it can be running for longer in the background, and it has time and memory limit control.

We can investigate how to move get_url_details() from this class to the background processing class.

Expected behavior No high CPU usage during this process

Screenshots

Backlog Grooming (for WP Media dev team use only)

[x] Reproduce the problem
[x] Identify the root cause
[x] Scope a solution
[x] Estimate the effort

DahmaniAdame commented 2 years ago

NataliaDrause commented 2 years ago

DahmaniAdame commented 2 years ago

vmanthos commented 2 years ago

NielsVornholt commented 2 years ago

Any progress with this issue yet or any estimation on when this will be improved/solved? We also suffer big time from this. rocket_rucss_warmup_resource_fetcher constantly destroys our production stacks due to 100% CPU consumption. In combination with rocket_saas_warmup it hits even worse. Very annoying.

Cloud hosting at IONOS, WP in Docker Swarm Services, Mysql on a 3-node galera cluster in full LB mode, varnish proxy cache. Site 1: 10k+ Articles Site 2: 30k+ Articles Site 3: 11k+ Articles Site 4: 12k+ Articles

vmanthos commented 2 years ago

This was on WPX.

NataliaDrause commented 2 years ago