Tabrisrp commented 3 years ago

Describe the bug We’re getting some reports about high CPU usage with RUCSS enabled, notably on Kinsta/WP Engine, and both pointing to the asynchonous process of the resource fetcher.

As a reminder, this process is triggered asynchronously on rocket_buffer, so running on uncached pages during the caching process. the AJAX action is rocket_saas_warmup

From their logs, some of those calls are running for too long, up to 6 min.

Looking at the code, during this call, we are:

parsing the HTML for CSS/JS files
for each file, we are getting the URL, path, content, type (and media value for CSS)
For the content itself, we are getting it from the local file, or from the remote file if it’s an external path, and then we are Minifying it
We finally send those to the background process which sends to warmup
I believe what would be causing the issue here is the content fetching + minifying step. Depending on the number of files found, it means a loop with a lot of processing in it.

I’m thinking this specific part should be delegated to the background processing task instead, as it can be running for longer in the background, and it has time and memory limit control.

We can investigate how to move get_url_details() from this class to the background processing class.

Expected behavior No high CPU usage during this process

Screenshots

Backlog Grooming (for WP Media dev team use only)

[x] Reproduce the problem
[x] Identify the root cause
[x] Scope a solution
[x] Estimate the effort

engahmeds3ed commented 3 years ago

We can investigate how to move get_url_details() from this class to the background processing class.

I think we can replace this get_url_details with get_url_path so we only return path and inside the background process, we can get each resource contents and bailout if we found empty content for the resources.

engahmeds3ed commented 3 years ago

Grooming

Reproduce the problem No need, I can see the issue on the code level and from the description above.

Identify the root cause We do a lot of work inside the ResourceFetcher Async request class.

Scope a solution We need to split the processing time between ResourceFetcher and ResourceFetcherProcess as follows:-

ResourceFetcher Responsibility

Collect CSS/JS resources from the HTML.
Get URL paths for each resource.
Send URL and path to the ResourceFetcherProcess background process.

ResourceFetcherProcess Responsibility

Receive CSS/JS resources (URL, path).
Get the content of the file based on its type (internal or external)
Minify the content.
Check the file on the local database.
Call the SaaS warmup.

So we will need to move the content-related methods to be on ResourceFetcherProcess

Estimate the effort [S]

NataliaDrause commented 3 years ago

Related: https://secure.helpscout.net/conversation/1549648702/273380?folderId=3864740 A few sites affected for this customer. They are on Litespeed servers

SRHDesign commented 3 years ago

The high CPU usage identified in this issue on Kinsta/WP Engine also occurs on SiteGround's cloud hosting.

DahmaniAdame commented 3 years ago

or128 commented 3 years ago

This new process is chocking our server, we're on DreamVPS.

vmanthos commented 3 years ago

DahmaniAdame commented 3 years ago

This one is more related to the bandwidth but is caused by the same root cause. A potential fix for this issue should help with the bandwidth aspect as well.

Slack discussion - https://wp-media.slack.com/archives/C08N8J6VC/p1625656955038000

vmanthos commented 3 years ago

wpsumo commented 3 years ago

The high CPU usage identified in this issue on Kinsta/WP Engine also occurs on SiteGround's cloud hosting.

Same issue on Cloudways with 2 CPU and 2gb RAM in my test.

RUCSS consume 100% CPU and RAM even when preload cache is disabled. It's too intensive, should be better with a 1 page per x seconds. Seem to be the background process for me as well, and it becomes too intensive if preload cache is enabled as well. Without preload cache it's hitting 60% CPU depending on how many pages is hit per second and queued.

Maybe to stress servers less, why not add an option RUCSS per page or per Post type? Generate a used.css per Post type this will require maybe 3-4 files only which will be 80-20 but actually covers most optimization 99% in my case, and we would result in a better cache hit of utilizing browser cache. This than generate unique per page while all pages or post in a post type has the same styling.

Crawl all posts generate 1 used.css, crawl all pages generate 1 used.css crawl example_cpt_1 generate 1 used.css etc. And maybe reduce the intervals between the crawls, helper plugin or a setting in WPR. Haven't tested on more powerful hardware to see what holds and what crash. But what did you find in your investigation above? You run the generation on your OVH servers, but it must be crawling and hitting the pages on customers sites to scan too frequent.

I would vote for a toggle solution to pick page based or post_type based used css.

DahmaniAdame commented 3 years ago

CPU overload is still there after 3.9.1+ update - https://secure.helpscout.net/conversation/1584697558/283470 Hosting service O2switch - shared plan.

NataliaDrause commented 3 years ago

Related ticket: https://secure.helpscout.net/conversation/1578693639/281920/ Issue is still there with 3.9.1.1

NataliaDrause commented 3 years ago

mifrero commented 3 years ago

Another one with 3.9.1.1 https://secure.helpscout.net/conversation/1587944301/284238/

vmanthos commented 3 years ago

https://secure.helpscout.net/conversation/1578363776/281843

3.9.1.1 is used.

jorgemartine00 commented 3 years ago

https://secure.helpscout.net/conversation/1600518533/287351/

3.9.1.1 is used.

viobru commented 3 years ago

https://secure.helpscout.net/conversation/1604657091/288212

3.9.1.1 is used.

DahmaniAdame commented 3 years ago

Related - https://secure.helpscout.net/conversation/1615837516/290982/ Hosting service: SiteGround WP Rocket version: 3.9.2 Other notes: Large numbers of posts (10k+) + large number of CSS and JS files (40+ for each).

juricazuanovic commented 3 years ago

Related: https://secure.helpscout.net/conversation/1629068596/294200/ Hosting: Kinsta WP Rocket version: 3.9.3

vmanthos commented 3 years ago

Potentially related: https://secure.helpscout.net/conversation/1626572700/293590/ Host: WP Engine WP Rocket version: 3.9.3

vmanthos commented 3 years ago

NataliaDrause commented 2 years ago

Related: https://secure.helpscout.net/conversation/1650139378/299044?folderId=3864740 Host SiteGround WP Rocket version: 3.10

vmanthos commented 2 years ago

In this case, the customer bulk removed trackbacks from 220 posts, and that caused error 503 for admin-ajax.php?action=rocket_rucss_warmup_resource_fetcher requests.

This is probably because their cache was cleared at once, and preload was triggered. This started the tree-shaking process for all the posts and led to high resource consumption and server errors.

vmanthos commented 2 years ago

This is a site with 70k articles.

Whatever clears/preloads the cache of a large number of posts could potentially increase the CPU usage with the same mechanism I described in the previous comment.👆

DahmaniAdame commented 2 years ago

Related - https://secure.helpscout.net/conversation/1659560913/300915?folderId=3864735 Hosting service: Kinsta WP Rocket version: 3.9.4.1 Other notes: Items to preload (4k+) + low count of assets.

lukecav commented 2 years ago

Recent customer example site the wp_wpr_rucss_resources database table was 1.1GB in size. Hosting service: Nexcess WP Rocket version: 3.10.1

NataliaDrause commented 2 years ago

vmanthos commented 2 years ago

NataliaDrause commented 2 years ago

vmanthos commented 2 years ago

DahmaniAdame commented 2 years ago

NataliaDrause commented 2 years ago

vmanthos commented 2 years ago

DahmaniAdame commented 2 years ago

Related - https://secure.helpscout.net/conversation/1729528645/314677/ Hosting - Kinsta

Hbitvof commented 2 years ago

Same problem here. Hosting - Savvii

vmanthos commented 2 years ago

Hbitvof commented 2 years ago

Likely related: https://secure.helpscout.net/conversation/1739451488/316045/

I can't see these secure links.

vmanthos commented 2 years ago

@Hbitvof Those are internal links that are accessible only to our team.

All info important to you will be publicly available here.

DahmaniAdame commented 2 years ago

vmanthos commented 2 years ago

wp-media / wp-rocket

High CPU usage related to rocket_saas_warmup async process #4072

Grooming

ResourceFetcher Responsibility

ResourceFetcherProcess Responsibility