search

nystudio107 / craft-seomatic

SEOmatic facilitates modern SEO best practices & implementation for Craft CMS 3. It is a turnkey SEO system that is comprehensive, powerful, and flexible.
https://nystudio107.com/plugins/seomatic
Other
162 stars 68 forks source link

429 Too many Request to google.com/ping endpoint / End of life of ping endpoint #1392

Closed d--j closed 7 months ago

d--j commented 7 months ago

Describe the bug

When batch-updating entries via CLI (thousands of them) we sometimes get 429 responses from the Google Sitemap submitting endpoint. They get logged as error and pollute our logging and alerting:

2023-11-30 18:52:32
[-][-][-][error][nystudio107\seomatic\services\Sitemaps::submitSitemapForElement]
Error submitting sitemap index to:
https://www.google.com/ping?sitemap=https%3A%2F%2Fwww.example.com%2Fsitemaps-2-section-sectionAuthors-2-sitemap.xml
- Client error: `POST
https://www.google.com/ping?sitemap=https%3A%2F%2Fwww.example.com%2Fsitemaps-2-section-sectionAuthors-2-sitemap.xml`
resulted in a `429 Too Many Requests` response:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head><meta http-equiv="content-type" content="te (truncated...)

To reproduce

N/A

Expected behaviour

Submitting sitemap changes should only happen once in a while – but at least only once for a single request/script run. Also this error might better be logged as warning – since nothing is really broken when this request failed.

Recommodation

Just remove the ping functionality. According to https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping it will only work until the end of the year anyway. Then every single request will produce a 404 and will be logged as an error (and will destroy our error alerting in the process).

[...] we're deprecating our support for sitemaps ping and the endpoint will stop functioning in 6 months. You can still submit your sitemaps through robots.txt and Search Console, but the HTTP requests ("pings") to the deprecated REST endpoint will result in a 404 error. Any existing code or plugins which use this endpoint will not cause problems for Google Search; you don't need to make any changes (but using the endpoint will also not do anything useful).

Versions

khalwat commented 7 months ago

It's been on my list to remove this anyway, since Google has deprecated the endpoint, and will be EOL'ing it shortly:

https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping

Addressed in: https://github.com/nystudio107/craft-seomatic/commit/23ff111115839b8f452aed6f2ff8c1cc1584d3a7 & https://github.com/nystudio107/craft-seomatic/commit/2159f8268543c09b952493af9423e9a1cc5607f5

Craft CMS 3:

You can try it now by setting your semver in your composer.json to look like this:

    "nystudio107/craft-seomatic": "dev-develop as 3.4.68”,

Then do a composer clear-cache && composer update

…..

Craft CMS 4:

You can try it now by setting your semver in your composer.json to look like this:

    "nystudio107/craft-seomatic": "dev-develop-v4 as 4.0.37”,

Then do a composer clear-cache && composer update