[seo-magic] Not finding sitemap file.

[danjames68]

SEO Magic has been working perfectly for years, it has started throwing an error: "Sitemap installed but "Sitemap URL" not reachable. SEO Magic requires sitemap to be functional and accurate."

Sitemap is being generated, it works if I check the URL and google search console is finding it and happy with syntax and URL.

Sitemap Plugin is v5.2.1 SEO Plugin is v5.2.1 GRAV is v1.7.45

[rhukster]

@danjames68 that error occurs when the plugin itself is unable to retrieve the plugin. Do you have password protection enabled on the site? Are you using a specific URL? Can you provide me with the URL of the sitemap? Is the URL of the Grav admin on the same domain?

[danjames68]

Hi Andy, no password protection, no specific URL - it's not working site-wide.

http://invitationtotuscany.com/sitemap.xml

Everything on the same domain.

[danjames68]

@rhukster I've updated GRAV and all plugins - and PHP is now 8.2. Still no joy.

[rhukster]

Perhaps its related to the fact it's not SSL. It shouldn't be, but that's all I can think of right now. Been a while since i've seen an http:// address... even my localhost is fully under SSL.

[danjames68]

My mistake, it's all under SSL. Also, it has worked seamlessly for at least two years, or whenever I first installed SEOMagic. I haven't changed anything - I updated everything AFTER I discovered the plugin wasn't working.

My localhost version is still working so perhaps I'll compare the files and the environments and see if anything jumps out at me.

[danjames68]

@rhukster I'm having no joy.

My local dev version works, so I looked for environment differences but couldn't see anything. To double-check I cloned the production site onto my dev server and the plugin on the cloned site works fine locally.

The production server is LiteSpeed V8.1 Cloudlinux 1.3, my dev is Apache 2.0 but I can't see that being a problem.

Production PHP is 8.1.28, current dev PHP is 8.0.30 but I've tried 7.4, 8.2 and 8.3 with no change - ie the plugin always works locally.

PHP modules all look good in the report. I also tried deleting all SEOMagic files in the data folder in case one of them was corrupted.

When I tried to run the /tools/report - which I've done successfully in the past - I got this error:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 118784 bytes)

`

 /home/invitati/public_html/vendor/filp/whoops/src/Whoops/Run.php(514): Whoops\Run->handleError() 
 /home/invitati/public_html/system/src/Grav/Common/Errors/SystemFacade.php(43): Whoops\Run->handleShutdown() 
 [internal function]: Grav\Common\Errors\SystemFacade->handleShutdown() 
 {main}

`

I've uninstalled SEOMagic and Sitemap, and deleted the config files, purged the cache and reinstalled - no change.

What else can I check?

[danjames68]

@rhukster I was getting a 403 from sitemap checkers like this one: https://www.seoptimer.com/sitemap-checker so I went through my .htaccess settings, with no joy. I paused Cloudflare and the 403 went away. I turned Cloudflare back on and it's blocked again. So the error is in Cloudflare - I'll check my settings.

[danjames68]

Cloudflare is challenging Grav SEOMagic Plugin as a bot.

[danjames68]

I've set up a rule so Cloudflare no longer challenges Grav SEOMagic Plugin and all seems to be working. Simple to solve, tricky to find.

Under Security > WAF > Rules.

[rhukster]

ah! Good find.. now you mention this, I remember seeing it before. You can get around this by adding an "ALLOW" rule for the SEO Magic crawler's user agent:

This is probably what you have done already. This is the reason the agent string is configurable.. you can set it to something you want to allow.