search

nystudio107 / craft-seomatic

SEOmatic facilitates modern SEO best practices & implementation for Craft CMS 3. It is a turnkey SEO system that is comprehensive, powerful, and flexible.
https://nystudio107.com/plugins/seomatic
Other
165 stars 70 forks source link

Sitemap intermittently not generating #1097

Closed rsleath closed 2 years ago

rsleath commented 2 years ago

I am working on four separate Craft CMS websites. Three use SEOmatic for SEO and generating the sitemap and the fourth uses a different SEO plugin. The three SEOmatic plugin sites are having intermittent issues with generating the sitemap. Sometimes when you open the sitemap it gives an error message saying "this sitemap has not been generated yet" and other times you open the sitemap and it works. The issue is filtering through to negatively affect SEO as when it happens when a Googlebot or SEMrushbot scans the site error messages are sent to Google Search Console or the health audit result drops. The sites are maintained and software is updated monthly. Also, we have been using SEOmatic for two to three years and have only experienced this problem in the last month.

khalwat commented 2 years ago

Sitemaps are generated via Queue jobs. Are you using a queue runner ala: https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms

?

khalwat commented 2 years ago

Addressed in: https://github.com/nystudio107/craft-seomatic/commit/71c8eac092fdf590e10c570f2230a51156790e32

You can try it now by setting your semver in your composer.json to look like this:

    "nystudio107/craft-seomatic": "dev-develop as 3.4.29”,

Then do a composer clear-cache && composer update

rsleath commented 2 years ago

Hello Andrew,

I had the change you suggested to composer.json done by one of our techs and we also updated all plugins and software on our Craft sites. It is the beginning of the month and so I am now running monthly health audit reports on these same websites. We are still having the same issue with SEOmatic. It is intermittently failing to generate sitemaps and this is harming the site's health and will therefore impact on Google ranking. This is happening on multiple Craft sites running on different servers, which makes me feel like it is a plugin issue.

Here is what SEMrush is reporting:

[image: Screen Shot 2022-05-03 at 10.38.11 am.png]

This site's health used to be 95% and is now at 85%.

When I check the sitemap in the browser this is what I see. The sitemap lists properly:

[image: Screen Shot 2022-05-03 at 11.09.43 am.png]

Then the first section https://innovateenergy.com.au/sitemaps-1-section-about-1-sitemap.xml is fine:

[image: Screen Shot 2022-05-03 at 11.06.25 am.png]

But then every other section sitemap in the list fails to display:

[image: Screen Shot 2022-05-03 at 11.05.12 am.png] This is a serious problem for my SEO work.

Regards Rebecca

On Tue, Apr 5, 2022 at 4:21 AM Andrew Welch @.***> wrote:

Addressed in: 71c8eac https://github.com/nystudio107/craft-seomatic/commit/71c8eac092fdf590e10c570f2230a51156790e32

You can try it now by setting your semver in your composer.json to look like this:

"nystudio107/craft-seomatic": "dev-develop as 3.4.29”,

Then do a composer clear-cache && composer update

— Reply to this email directly, view it on GitHub https://github.com/nystudio107/craft-seomatic/issues/1097#issuecomment-1087871434, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJABY3ECCC32PE5XYU7NBDVDMXMDANCNFSM5SOTGTPQ . You are receiving this because you authored the thread.Message ID: @.***>

--

Regards, Rebecca Sleath Web Team I work part-time on Monday, Tuesday and Thursdays

E-mail: Phone: Website: Address: @.** +61244418555 opt 4 <//+61244418555> https://www.itbasecamp.com.au 1/50 Owen Street Huskisson NSW 2540 https://goo.gl/maps/YwtMGUej1Qo https://www.facebook.com/itbasecampptyltd http://www.linkedin.com/company/i-t-basecamp-pty-ltd https://goo.gl/maps/dQRU1jR2nLM2 Signup* http://eepurl.com/dao-gb [image: latest news and offers] https://itbasecamp.com.au/free-digital-audit

QLD Govt. Accredited Supplier: GITC Q-5933 NT Govt. Accredited Supplier: CAL I-1-16563-10-16 TAS Govt. Accredited Supplier: GITC T764 NSW Govt. Registered Supplier: 224891

The content of this email and any file attachments or information contained therein are confidential and may contain privileged or copyright information subject to established terms of the agreement between parties. If you have received this email in error we apologise and ask that you notify the sender via return email and then delete the original from your inbox and destroy any printed copies. This note also confirms that this email message has been scanned for viruses and although none were detected, IT Basecamp Pty Ltd accepts no liability for any consequential damage resulting from the email containing any computer viruses. Thank you.

khalwat commented 2 years ago

It is intermittently failing to generate sitemaps and this is harming the site's health and will therefore impact on Google ranking. This is happening on multiple Craft sites running on different servers, which makes me feel like it is a plugin issue.

To be clear, sitemaps do not impact Google ranking directly. Sitemaps are pointers for GoogleBot to tell it what content to prioritize indexing.

If a page is already in Google's index, it does not need to be in the sitemap. Google also discovers URLs to crawl in ways other than sitemaps, such as crawling links from your site and other sites that link to your pages.

Now as for the intermittent problems you've having, I checked every sitemap index in your sitemap.xml: https://innovateenergy.com.au/sitemap.xml

They all responded with 200 OK:

https://innovateenergy.com.au/sitemaps-1-section-about-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-blog-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-contact-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-faq-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-homepage-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-installations-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-services-1-sitemap.xml

As mentioned in the other ticket, sitemaps are generated via queue job. You might want to ensure that you don't have any failed queue jobs that could be holding up the sitemap generation.

Additionally, you might consider setting up a queue runner as per:

https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms

rsleath commented 2 years ago

Hi Andrew,

It doesn't surprise me that you tried all the sitemaps on that site and had them work - as I mentioned it is an intermittent problem. Sometimes fine, other times doesn't work. I escalated the issue to senior tech at our end yesterday and they did a full investigation of setup and server logs and code. They agree the problem is happening and experienced times when the sitemaps worked and also when they didn't. We turned off the auto-regenerate setting in SEOmatic settings and added a sitemap generation command to happen once a day in the cron settings to check if that makes a difference.

This morning I have checked all three problem sites. The site where we have turned off the auto-regenerate settings is working and I am seeing 'failure to generate' problems on the other two sites running SEOmatic. I will continue to check over the course of the next week.

Regards, Rebecca

On Wed, May 11, 2022 at 3:31 AM Andrew Welch @.***> wrote:

It is intermittently failing to generate sitemaps and this is harming the site's health and will therefore impact on Google ranking. This is happening on multiple Craft sites running on different servers, which makes me feel like it is a plugin issue.

To be clear, sitemaps do not impact Google ranking directly. Sitemaps are pointers for GoogleBot to tell it what content to prioritize indexing.

If a page is already in Google's index, it does not need to be in the sitemap. Google also discovers URLs to crawl in ways other than sitemaps, such as crawling links from your site and other sites that link to your pages.

Now as for the intermittent problems you've having, I checked every sitemap index in your sitemap.xml: https://innovateenergy.com.au/sitemap.xml

They all responded with 200 OK:

https://innovateenergy.com.au/sitemaps-1-section-about-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-blog-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-contact-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-faq-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-homepage-1-sitemap.xml

https://innovateenergy.com.au/sitemaps-1-section-installations-1-sitemap.xml https://innovateenergy.com.au/sitemaps-1-section-services-1-sitemap.xml

As mentioned in the other ticket, sitemaps are generated via queue job. You might want to ensure that you don't have any failed queue jobs that could be holding up the sitemap generation.

Additionally, you might consider setting up a queue runner as per:

https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms

— Reply to this email directly, view it on GitHub https://github.com/nystudio107/craft-seomatic/issues/1097#issuecomment-1122678589, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJABY4NAFAJ3OVEC5WGZHDVJKMNJANCNFSM5SOTGTPQ . You are receiving this because you authored the thread.Message ID: @.***>

--

Regards, Rebecca Sleath Web Team I work part-time on Monday, Tuesday and Thursdays

E-mail: Phone: Website: Address: @.** +61244418555 opt 4 <//+61244418555> https://www.itbasecamp.com.au 1/50 Owen Street Huskisson NSW 2540 https://goo.gl/maps/YwtMGUej1Qo https://www.facebook.com/itbasecampptyltd http://www.linkedin.com/company/i-t-basecamp-pty-ltd https://goo.gl/maps/dQRU1jR2nLM2 Signup* http://eepurl.com/dao-gb [image: latest news and offers] https://itbasecamp.com.au/free-digital-audit

QLD Govt. Accredited Supplier: GITC Q-5933 NT Govt. Accredited Supplier: CAL I-1-16563-10-16 TAS Govt. Accredited Supplier: GITC T764 NSW Govt. Registered Supplier: 224891

The content of this email and any file attachments or information contained therein are confidential and may contain privileged or copyright information subject to established terms of the agreement between parties. If you have received this email in error we apologise and ask that you notify the sender via return email and then delete the original from your inbox and destroy any printed copies. This note also confirms that this email message has been scanned for viruses and although none were detected, IT Basecamp Pty Ltd accepts no liability for any consequential damage resulting from the email containing any computer viruses. Thank you.

khalwat commented 2 years ago

We turned off the auto-regenerate setting in SEOmatic settings and added a sitemap generation command to happen once a day in the cron settings to check if that makes a difference.

I think there is still likely some kind of underlying devops issue at play here. Perhaps have your senior tech give this a read: https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms

rsleath commented 2 years ago

PS: we are not running any queue runner.

On Tue, Apr 5, 2022 at 4:21 AM Andrew Welch @.***> wrote:

Addressed in: 71c8eac https://github.com/nystudio107/craft-seomatic/commit/71c8eac092fdf590e10c570f2230a51156790e32

You can try it now by setting your semver in your composer.json to look like this:

"nystudio107/craft-seomatic": "dev-develop as 3.4.29”,

Then do a composer clear-cache && composer update

— Reply to this email directly, view it on GitHub https://github.com/nystudio107/craft-seomatic/issues/1097#issuecomment-1087871434, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJABY3ECCC32PE5XYU7NBDVDMXMDANCNFSM5SOTGTPQ . You are receiving this because you authored the thread.Message ID: @.***>

--

Regards, Rebecca Sleath Web Team I work part-time on Monday, Tuesday and Thursdays

E-mail: Phone: Website: Address: @.** +61244418555 opt 4 <//+61244418555> https://www.itbasecamp.com.au 1/50 Owen Street Huskisson NSW 2540 https://goo.gl/maps/YwtMGUej1Qo https://www.facebook.com/itbasecampptyltd http://www.linkedin.com/company/i-t-basecamp-pty-ltd https://goo.gl/maps/dQRU1jR2nLM2 Signup* http://eepurl.com/dao-gb [image: latest news and offers] https://itbasecamp.com.au/free-digital-audit

QLD Govt. Accredited Supplier: GITC Q-5933 NT Govt. Accredited Supplier: CAL I-1-16563-10-16 TAS Govt. Accredited Supplier: GITC T764 NSW Govt. Registered Supplier: 224891

The content of this email and any file attachments or information contained therein are confidential and may contain privileged or copyright information subject to established terms of the agreement between parties. If you have received this email in error we apologise and ask that you notify the sender via return email and then delete the original from your inbox and destroy any printed copies. This note also confirms that this email message has been scanned for viruses and although none were detected, IT Basecamp Pty Ltd accepts no liability for any consequential damage resulting from the email containing any computer viruses. Thank you.