Saving a big post with Yoast enabled is terribly slow.

maciejmackowiak commented 3 years ago

[x] I've read and understood the contribution guidelines.
[x] I've searched for any related issues and avoided creating a duplicate issue.

Please give us a description of what happened.

Saving posts with a lot of links and images takes ages to save.
The other issue is that it also results in timing out the wp autosave ajax request and triggering "connection lost" notice, luckly it doesn't prevent post from being save because.

Looks like the culprit is Indexable_Link_Builder->build() added to wp_insert_post hook in Indexable_Post_Watcher

I've tested it on blank wp install with only Yoast and classic editor installed: First request is with Yoast enabled and the second one is with Yoast disabled, both requests are wp_autosave: obraz

Here is sample post: https://gist.github.com/maciejmackowiak/277427859825d232fe2a37a21d15d7cf

Please describe what you expected to happen and why.

Optimize the Indexable_Link_Builder or if it is not possible to optimize run it in cron job.
Do not run Indexable_Link_Builder on autosaving posts.

How can we reproduce this behavior?

Create a post with a lot of links and images - https://gist.github.com/maciejmackowiak/277427859825d232fe2a37a21d15d7cf
Try to save/autosave the post.

Technical info

If relevant, which editor is affected (or editors):
[ ] Classic Editor
[ ] Gutenberg
[ ] Classic Editor plugin

Which browser is affected (or browsers):
[ ] Chrome
[ ] Firefox
[ ] Safari
[ ] Other

Used versions

WordPress version:
Yoast SEO version:
Gutenberg plugin version:
Classic Editor plugin version:
Relevant plugins in case of a bug:
Tested with theme:

archon810 commented 3 years ago

In real life it's even slower in production, with a larger db and more plugins installed. 40s vs 1s to save.

The severity of this ticket should be marked critical.

Djennez commented 3 years ago

@maciejmackowiak do you have any practical examples for this amount of links / images? Also, have you checked that this issue isn't a duplicate of something that is already present? There might already exist an issue about this.

Using your sample data I went from 0.500 to 5.300 seconds on my test environment.

archon810 commented 3 years ago

@Djennez Even with the shown example, going from 0.5s to 5.3s is a 10x increase in processing, which is already indication that this issue should be fixed. If it can't be sped up with code or SQL changes, then whatever is happening here should go to wp cron so it's executed asynchronously and not slowing down every post update by many seconds.

archon810 commented 3 years ago

Yoast 15.9.1 made a huge difference for saving drafts thanks to this change:

Performance enhancements in the post indexable builder by skipping link creation on drafts.

During my test, saving a large post that used to take 1 minute is now down to 8 seconds. Still not ideal, but so much better and I'd have to look at what causes it to be that slow, could be other plugins. The point is, 8s is usable. 1 minute is not.

However, this only applies to saving drafts and not publishing or updating published posts. So it's still a problem, and the simplest solution I see is going to WP cron jobs for any long work that may need to be done. Either setting a post meta to mark a post as pending and using a single job, or using one job per post.

There could be problems using multiple jobs per post in case someone bulk updates thousands of posts at the same time - unless they're using Cavalcade to handle cron, they may have a bad time with a huge increase in cron jobs (plus some may be overwritten due to race conditions - WP cron's default serialized implementation is really really terrible).

Djennez commented 3 years ago

Internal issue at https://yoast.atlassian.net/browse/AR-121

michaelbriantina commented 1 year ago

Please inform the customer of conversation # 1010744 when this conversation has been closed.

monbauza commented 1 year ago

The above customer was getting a 524 timeout error when they tried to update a post and Yoast SEO was enabled. The timeout error didn't occur with Yoast SEO disabled. The post where the timeout occurs has 311 images (the site has 113,572 media items). Removing all the images from the post fixes the issue. Slack trace below.

Relevant Yoast SEO files and functions:

in attachment_url_to_postid called at /var/www/vhosts/thetrendspotter.net/www/thetrendspotter.net/wp-content/plugins/wordpress-seo/inc/class-wpseo-image-utils.php (70)
in WPSEO_Image_Utils::attachment_url_to_postid called at /var/www/vhosts/thetrendspotter.net/www/thetrendspotter.net/wp-content/plugins/wordpress-seo/inc/class-wpseo-image-utils.php (44)
in WPSEO_Image_Utils::get_attachment_by_url called at /var/www/vhosts/thetrendspotter.net/www/thetrendspotter.net/wp-content/plugins/wordpress-seo/src/builders/indexable-link-builder.php (333)

maybellyne commented 1 year ago

+1 https://wordpress.org/support/topic/slow-post-update/

suascat commented 1 year ago

Please inform the customer of conversation # 1017543 when this conversation has been closed.

monbauza commented 1 year ago

Please inform the customer of conversation # 1014859 when this conversation has been closed.

monbauza commented 1 year ago

The above customer experiences slow response times, and sometimes 502 errors, when Yoast SEO is enabled. This happens on posts with images. The website has 651,534 media items.

On the customer's site (with only Yoast SEO enabled and the Twenty Twenty-Three theme active), a post with a couple of images and no text or links takes around 1 minute to respond after clicking the Update button in the editor (vs just a few milliseconds when Yoast SEO is disabled).

For this customer, enabling media pages under Yoast SEO > Settings > Advanced > Media Pages seems to significantly improve the response times, however, this isn't a realistic workaround. If media pages are enabled, the attachment pages won't be redirected so they could be crawled by search engines which may cause SEO issues (especially on large websites with thousands of attachment pages).

monbauza commented 1 year ago

bilal-ahmed-dev commented 1 year ago

Hi There, Can someone tell me when this issue will be resolved, unnamed unnamed (1) After deactivating Yoast. unnamed (2) unnamed (3)

Can you prioritize this as this issue is affecting our work?

maybellyne commented 1 year ago

+1 https://wordpress.org/support/topic/added-posts-takes-a-lot-of-time/

Yura-87 commented 1 year ago

Hope developers of Yoast plugin will solve this issue soon. I am an owner of large WP website and Yoast SEO Plugin causing this issue to us too.

maybellyne commented 1 year ago

+1 https://wordpress.org/support/topic/saving-post-results-in-404/

sekra24 commented 1 year ago

Dear Yoast team,

is there anything a siteowner can do, to fix this issue? We are not able to save large articles, without getting a 524 timeout. It slows down work significantly.

Thank you, Sebastian

detolah commented 1 year ago

Please inform the customer of conversation # 1035316 when this conversation has been closed.

JiveDig commented 1 year ago

We're seeing the same issue on a few sites with a lot of posts/media, when publishing new posts that have more than a couple images in the content.

ckomarov commented 1 year ago

Hello, I've got same issue when updating some of the posts. Slow logs shows attachment_url_to_postid as the reason of this.

Yura-87 commented 1 year ago

It is really shame for Yoast team than they still didn't fixed it. Our editors are causing with this issue for at least 3 months (we have editorial team about 12 people and they are really angry, because they have to wait a loooong time with edits of every single published article).

@ckomarov: do you have any idea how to handle/temporary disable (until Yoast fix it) the attachment_url_to_postid?

bilal-ahmed-dev commented 1 year ago

@Yura-87 Simply deactivate their plugins, there are good alternatives out there, Just give them a try. I posted this issue two months ago and they did not prioritize this ticket so far, I do not know might be they have some number limit when the ticket is prioritized. Anyhow we have paid membership of Yoast Premium and they are just the worst literally there too.

sekra24 commented 1 year ago

I reported this issue weeks ago but there is no response from the devs how or when this issue will be fixed. Only first level support told me I should try to fix it myself 🙈 Great support 👏 Why I'm paying for this?

Yura-87 commented 1 year ago

@Yura-87 Simply deactivate their plugins, there are good alternatives out there, Just give them a try. I posted this issue two months ago and they did not prioritize this ticket so far, I do not know might be they have some number limit when the ticket is prioritized. Anyhow we have paid membership of Yoast Premium and they are just the worst literally there too.

I can't simply deactivate their plugin. I have website with more thank 32k articles (with 20 years of history), so for not it is not that simple.

But I am very angry with the Yoast SEO Premium support. This is a huge bug of their plugin (which affects only the large websites) and they can't fix it. I will prepare article in English on one large WordPress website (about plugins). I've talked with their editors and they want to publish it, so this will really not be a good mark for Yoast SEO creators.

maybellyne commented 1 year ago

We're sorry a fix for this issue has taken so long, and we have not communicated our progress so far. We understand the huge impact of the bug on our users' sites. We have tested a fix that has shown great improvements and tried as much as possible to avoid regressions. Our devs are actively working on the first MVP of the fix. We appreciate your patience so far.

Yura-87 commented 1 year ago

Thanks for reply, happy than you are working on it. Do you have any ETA, when we can expect the release of fix? Or if you want, I will be happy to test it on our site and help you with debugging.

maybellyne commented 1 year ago

Thanks for your patience while we are working on the bug. The earliest our devs can release a fix is September 5th. We might also share a zipped file to test the fix on your site.

sekra24 commented 1 year ago

@maybellyne we need a fix for this also ASAP, so maybe you could share this zip file publicly with a disclaimer that it is beta.

Best, Sebastian

bilal-ahmed-dev commented 1 year ago

@maybellyne Kindly provide the zip of beta plugin, so i can test this on my staging site first. Thanks

Yura-87 commented 1 year ago

Any news? Today was released a new update, but nothing with solving this problematic bug :(

sebbs121 commented 1 year ago

Any news on that?

sekra24 commented 1 year ago

Same here, we are urgently waiting for the update. Our writers can not publish any posts without waiting 5 minutes for every post to be saved. This is a nightmare!!

JiveDig commented 1 year ago

Our bigger clients are stuck here too. They are currently disabling Yoast temporarily every time the publish. We're now forced to discuss switching to RankMath or SEO Framework.

I'd be happy with a filter or setting to disable this feature.

enricobattocchi commented 1 year ago

Hello everyone, I can assure you that we are actively working on a solution aiming to improve the performance in several of the affected cases. As you can understand, it's not so straightforward to test, and we wouldn't want to release a fix that doesn't work or even makes the problem worse. I hope that we can soon provide news about it, and I thank you for your comprehension and patience!

Yura-87 commented 1 year ago

Hello Enrico, and do you have a more specific date for this fix? We are waiting to solve this for more than 4 months, which is crazy. Someone from your team mentioned as earliest date the 5th of September, so I really need to know how long do we need to wait. Because it will be too long, we will switch on all our sites (12 large websites) to RankMath.

enricobattocchi commented 1 year ago

Hello people, thanks for your patience. We have a preliminary zip containing the fix, we'd really appreciate if you could test this and let us know whether the problem is solved, reduced or still stands.

Please note that this has not passed yet all our quality control checks so we don't recommend installing it in a production environment

We advise you to test it in a development or staging copy of your site.

wordpress-seo-21.1-big-post-tentative-fix.zip

Yura-87 commented 1 year ago

@enricobattocchi Thanks for the temp-fix, but in my case doesn't help. Tested on our production site. After the update to this build we cleared all the caches (WP Rocket, Cloudflare, server cache...), but the problem still persists. Scenario: I opened published article with large number of photos/galleries, click to Update button, wait about +- 60 secs and got the timeout error (same as with the standard plugin version). So nothing changed for us :-(

bilal-ahmed-dev commented 1 year ago

@enricobattocchi Thanks for the fix, In my case this issue is not fully fixed now. I have tested this zip on the staging site. And on staging it is taking approx 7 to 9 seconds to save the post and on staging i have around 650k media items. But when I tested this on production it took approx 10 to 12 seconds and on production i have only 54k media items, yet it is slower than staging.

But there is an issue in console which is my.yoast.com/api/downloads/file/morphology-en-v4?plugin_version=20.6&site=https%3A%2F%2Ffabulesslyfrugal.com:1 Failed to load resource: the server responded with a status of 403 ()

thijsoo commented 1 year ago

Hi @Yura-87 ,

Thank you for your response. We have a couple of follow up questions to better determine a next step for you. Could you tell us the following:

Which WP version are you running currently.
Which editor are you using? (classic, gutenberg, Elementor) or maybe something else.
Are you using a custom plugin for the galleries or are you using the block editor.
Do you have media pages enabled in your YoastSEO setting.

Yura-87 commented 1 year ago

Hello @thijsoo, thank you for reply. I am running on latest WP version 6.3.1, using Classic editor. We are using default WP galleries. Media indexing is disabled. If you want, I can give you access to our staging site to check the issue :)

thijsoo commented 1 year ago

Hi @Yura-87 thanks for your quick reply. I this case we would like to suggest the following:

Can you add the following set of filters whereever you add your filters.

\add_filter( 'wpseo_indexable_forced_included_post_types', function( $post_types ) {
$post_types[] = 'attachment';
return $post_types;
} );
\add_action( 'init', function( $post_types ) {
\add_filter( 'wpseo_indexable_excluded_post_types', function( $post_types ) {
   $filtered_post_types = [];
   foreach ( $post_types as $post_type ) {
      if ( $post_type !== 'attachment' ) {
         $filtered_post_types[] = $post_type;
      }
   }
   return $filtered_post_types;
} );
} );
\add_filter('wpseo_force_creating_and_using_attachment_indexables', '__return_true'  );

After adding these filters you need to run the SEO data optimization again. from the tools menu of the YoastSEO plugin (/wp-admin/admin.php?page=wpseo_tools) Since you have a lot of images this can take quite a while. If you have access to a WP CLI I would sugguest using the wp yoast index command to speed up this process. Read more about the options of this command here: https://developer.yoast.com/features/wp-cli/reindex-indexables/

This solution will create database records for your media pages so we won't have to rely on WordPress to find everything we need.

Let me know if this speeds things up for you :)

Yura-87 commented 1 year ago

Hello @thijsoo. Thanks for detailed guide, but the main problem on our website is, than we can't finish the SEO data optimization process. We are using WP CLI on our server, which looks like finished in terminal correctly (screen is with the debug mode), as you can see on this screenshot.

Even if I run the optimization process from the dashboard (after WP CLI finish his work), I always after few minutes of running get the error message Error parsing the response to JSON.

I was trying to figure this out with the support team few months ago, but without success. But they wrote me, than if the SEO data optimization isn't finish, it won't affect the global functionality of Yoast SEO plugin.

My question is, is there any solution which will bypass the data optimization, because maybe this could be a reason, why your fix won't help on our site.

Or can I try to add filters above in your post into functions.php file even though the data hasn't been optimized?

amboutwe commented 1 year ago

Please inform the customer of conversation # 1056454 when this conversation has been closed.

sekra24 commented 1 year ago

grafik This is from the SEO index optimization.

Main concern: Despite installing the provided fix on our development environment, we've observed no improvement. We're encountering a timeout after 60 seconds when attempting to save a post containing 200 images.

Can you shed light on this issue? Many of us are developers and might be able to assist. I'm curious as to why the Yoast plugin performs indexing during the post save process instead of utilizing a cron job. Isn't it a best practice to use cron jobs for indexing large datasets?

This particular issue impacts numerous Yoast installations, with many users unaware that it's related to the Yoast plugin. This has led to countless wasted debugging hours. It's quite frustrating.

Greater communication around this issue would be beneficial.

Best, Sebastian

thijsoo commented 1 year ago

We're sorry that the improvements didn't improve things for you @Yura-87 @sekra24. Let us provide some more context, along with a couple of potential next steps for you. The improvements are supposed to work for cases where the HTML output of images include the image ID, much like the default output of the block editor, for image, gallery, media and cover blocks. For cases where that canonicalized protocol isn't followed, (like, eg. the classic editor's galleries), the improvements aren't improving times. With these improvements, we also included a way to bypass all that and rely on the SEO optimizations results, so as to have image IDs readily available when saving a post. But, due to it not being able to finish on your sites, for reasons unrelated to the task at hand, you can't use those filters either. For that reason, I would suggest opening separate support tickets in order to get to the bottom of why the SEO optimization process fails and retry with those filters enabled, when that's fixed. A different course of action would be to use images that are covered by our improvements, like the block editor ones, shared above, or Classic Editor's images.

I'm curious as to why the Yoast plugin performs indexing during the post save process instead of utilizing a cron job. Isn't it a best practice to use cron jobs for indexing large datasets?

We actually do have cron jobs performing indexing in the background. What we do during post save is matching image URLs to IDs that will later be used in the post's frontend for SEO purposes. That part can't be done without bloating the database in the background, hence our approach.

bilal-ahmed-dev commented 1 year ago

Hi @thijsoo @enricobattocchi When will you release the temporary fix for public use? Is it safe to use your temporary fix plugin on production? Without a temporary fix and with media reduction with a total no of 54k media items it takes around 8 to 9 seconds for 302 redirects and then it takes 4 seconds for an actual post-saving request. With a temporary fix plugin and with media reduction to 54k takes around 4 to 5 seconds for 302 to redirect the first call and then 3 to 4 seconds for actual post saving. So for me, it's a huge difference when speed matters for the client. Let me know Thanks

Yura-87 commented 1 year ago

We're sorry that the improvements didn't improve things for you @Yura-87 @sekra24. Let us provide some more context, along with a couple of potential next steps for you. The improvements are supposed to work for cases where the HTML output of images include the image ID, much like the default output of the block editor, for image, gallery, media and cover blocks. For cases where that canonicalized protocol isn't followed, (like, eg. the classic editor's galleries), the improvements aren't improving times. With these improvements, we also included a way to bypass all that and rely on the SEO optimizations results, so as to have image IDs readily available when saving a post. But, due to it not being able to finish on your sites, for reasons unrelated to the task at hand, you can't use those filters either. For that reason, I would suggest opening separate support tickets in order to get to the bottom of why the SEO optimization process fails and retry with those filters enabled, when that's fixed. A different course of action would be to use images that are covered by our improvements, like the block editor ones, shared above, or Classic Editor's images.

I'm curious as to why the Yoast plugin performs indexing during the post save process instead of utilizing a cron job. Isn't it a best practice to use cron jobs for indexing large datasets?

We actually do have cron jobs performing indexing in the background. What we do during post save is matching image URLs to IDs that will later be used in the post's frontend for SEO purposes. That part can't be done without bloating the database in the background, hence our approach.

ehh, I am really confused. We are using default WP images/gallery & classic editor and I created a support ticket for this about 4-5 months ago and they told me "It is a bug and our developers will work on this". And now, after 5 months of waiting you told me "Open support ticket"?

To put it simply, Yoast SEO has worked great for us for years. Then you made some change that turned into a problem that we've been waiting months to resolve. I have spent hours troubleshooting this issue and all I get from you after months of waiting is "Open another support ticket"?

amboutwe commented 1 year ago

Hi @Yura-87. We understand how our request may sound confusing. Let me help clear up the confusion but first, I'd like to introduce myself. My name is Angelia. I am a senior technical support agent and have been part of the Yoast support team since 2014.

The potential fix relies on a feature that isn't working for you (the SEO data optimization). Therefore, we'd like to have our support team investigate why the optimization doesn't complete. I searched in our support platform and, sadly, couldn't locate your support request. Could you reply to the most recent support contact you've received from us? Or you can start a new one here. Just mention this issue and that I asked you to contact us directly. I'll let the team know to look out for your reply so I can dig into the issue further.

Yura-87 commented 1 year ago

Hello @amboutwe, thank you for you reaction. Now I am trying to resolve the issue with SEO data optimization process with Suwash (he is really helpful). In my last e-mail (sent few hours ago) at his request, I provided the temporary login into our admin, so I think they are now inspecting the issue.

bilal-ahmed-dev commented 1 year ago

@amboutwe Can you also reply to my previous response here? Can you also tell me why there is a 302 request before the saving request? The 302 requests take 4 to 5 seconds with the temporary fix plugin and after that, it takes again 4 to 5 seconds for the post to save. Without a temporary fix plugin, it takes around 7 to 8 seconds for 302 and 5 to 6 seconds for post-save calls. I have reduced my media items from 600k to 54k but still a total of 13 to 15 seconds is not a good number to save a post. Kindly let me know at your convenience. Thanks

Yoast / wordpress-seo