search

elementor / wp2static

WordPress static site generator for security, performance and cost benefits
https://wp2static.com
The Unlicense
1.42k stars 270 forks source link

500 error when certain sitemaps are non-existant #751

Closed leonstafford closed 3 years ago

leonstafford commented 3 years ago

Reported here:

https://staticword.press/t/the-plugin-returns-500/353/2

User:

i did some research and i found out what was the problem? /var/www/admin/wp-content/plugins/wp2static/src/DetectSitemapsURLs.php In this file, this line: $parser->parse( $sitemap ); should always throw an error since sitemap is never a full URL.

leonstafford commented 3 years ago

Possibly fixed in this build. Please report if remains:

wp2staticfixmultisitemap.zip

petewilcock commented 3 years ago

Not sure if related but was try to export on a new site and it throws the following:

PHP Fatal error: Uncaught WP2StaticGuzzleHttp\\Exception\\ClientException: Client error:GET http://wordpress.example.co.uk/http://wordpress.example.co.uk/wp-sitemap.xml``

This is an existing site I've imported where I suspect the sitemap didn't exist... not sure why the site_url is being duplicated here but it Error 500s at the start of the crawl.

Strangely, the fix was to install Yoast to ensure a sitemap.xml was generated

leonstafford commented 3 years ago

@petewilcock did that build attached above solve that or are you running from latest source already?

petewilcock commented 3 years ago

@leonstafford I did try your downloaded build but still hit the same issue - my other version is a dev build I pulled a week or so ago

loganmarchione commented 3 years ago

@petewilcock - I had a similar issue and installing Yoast to generate a sitemap.xml fixed it. Thanks for finding this!

Feb  6 22:45:06 web05 php: PHP Fatal error:  Uncaught WP2StaticGuzzleHttp\Exception\ClientException: Client error: `GET https://mydomain.com/https://mydomain.com/sitemap.xml` resulted in a `404 Not Found` response:
Feb  6 22:45:06 web05 php: <!DOCTYPE html>
Feb  6 22:45:06 web05 php: <html lang="en-US">
Feb  6 22:45:06 web05 php: <head>
Feb  6 22:45:06 web05 php: <meta charset="UTF-8">
Feb  6 22:45:06 web05 php: <meta name="viewport" content="width=device-width, ini (truncated...)
Feb  6 22:45:06 web05 php:  in /var/www/wordpress/mydomain/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticguzzle/src/Exception/RequestException.php:113
Feb  6 22:45:06 web05 php: Stack trace:
Feb  6 22:45:06 web05 php: #0 /var/www/wordpress/mydomain/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticguzzle/src/Middleware.php(69): WP2StaticGuzzleHttp\Exception\RequestException::create()
Feb  6 22:45:06 web05 php: #1 /var/www/wordpress/mydomain/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticpromises/src/Promise.php(204): WP2StaticGuzzleHttp\Middleware::WP2StaticGuzzleHttp\{closure}()
Feb  6 22:45:06 web05 php: #2 /var/www/wordpress/mydomain/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticpromises/src/Promise.php(153): WP2StaticGuzzleHttp\Promise\Promise::callHandler()
Feb  6 22:45:06 web05 php: #3 /var in /var/www/wordpress/mydomain/wp-content/plugins/wp2static/src/DetectSitemapsURLs.php on line 125
vyskoczilova commented 3 years ago

@petewilcock Same here, but I've got the sitemap.xml created by WP itself, got the very same error as happened to you (with duplicated URLs) installing Yoast helped.

aprilhalliwell commented 3 years ago

I encountered this issue today, for me It was caused by my robots.txt file including the sitemap URL in it. The URL in the robots.txt file was the full URL (base site + sitemap.xml). Since the Client request includes the base URL this caused the double URL issue.

I worked around this by removing the sitemap from the robots.txt file.

leonstafford commented 3 years ago

@dardalios many thanks for reporting this, it should help get closer to solving it. I'm working on moving out all of the URL detection into its own library, for WP2Static and others to use. That may replace current sitemap/robots parsing code before I fix it in place.

belidzs commented 3 years ago

I had the same issue due to the absolute URLs in robots.txt generated by XML Sitemaps

leonstafford commented 3 years ago

To update on this, I've gotten some time again to work on https://github.com/leonstafford/WordPressURLDetector/, which will will become a drop-in replacement for all URL detection within WP2Static. Can't give an ETA of when it will be ready yet, just doing some tidy up/organising the code as a library. Sitemap bit that came from WP2Static is my least liked part of it, seems unecessarily complex and as we're seeing brittle. I've got a few ideas for much easier solutions.

In the meantime, it seems there's some workarounds people in this thread have suggested.

palmiak commented 3 years ago

Just got the same problem and for a workaround I just this bit to the code:

foreach ( array_keys( $sitemaps ) as $sitemap ) {
    if ( ! is_string( $sitemap ) ) {
        continue;
    }

    $sitemap = '/' . str_replace(
        $wp_site_url,
        '',
        $sitemap
    );

What do you think about adding as a working fix?

leonstafford commented 3 years ago

Thanks @palmiak for the PR!

Here's the build of that zip if anyone who is still able to reproduce the issue can help test:

wp2static-sitemap-500-fix.zip

palmiak commented 3 years ago

Just tested the new release on my server that was causing problems - it's working now with new version 🎉

adrianratajczak commented 3 years ago

I tried fixed version from @leonstafford and still getting 500 error. That's my error log file `#1 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(633): WP2Static\URLDetector::detectURLs()

2 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(745): WP2Static\Controller::wp2staticHeadless()

3 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(301): WP2Static\Controller::wp2staticRun()

4 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(327): WP_Hook->apply_filters()

5 /home/new.skiddou.com/public_html/wp-includes/plugin.php(470): WP_Hook->do_action()

6 /home/new.skiddou.com/public_html/wp-admin/admin-ajax.php(187): do_action()

7 {ma

2021-09-17 13:58:30.450832 [NOTICE] [3370032] [77.65.110.183:37637-Q:E9FCD96ED0517F03-4#new.skiddou.com] [STDERR] in} thrown in /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/DetectSitemapsURLs.php on line 127`

That's my logs from Run section 2021-09-17 14:09:18: Unable to fetch URL contents 2021-09-17 14:09:13: Starting to detect WordPress site URLs. 2021-09-17 14:09:13: Starting URL detection 2021-09-17 14:09:13: Running WP2Static in Headless mode 2021-09-17 14:09:13: Running full workflow from UI 2021-09-17 14:08:38: Unable to fetch URL contents 2021-09-17 14:08:33: Starting to detect WordPress site URLs. 2021-09-17 14:08:33: Starting URL detection 2021-09-17 14:08:33: Running WP2Static in Headless mode 2021-09-17 14:08:33: Running full workflow from UI 2021-09-17 14:05:53: Unable to fetch URL contents 2021-09-17 14:05:46: Starting to detect WordPress site URLs. 2021-09-17 14:05:46: Starting URL detection 2021-09-17 14:05:46: Running WP2Static in Headless mode 2021-09-17 14:05:46: Running full workflow from UI 2021-09-17 14:00:34: Starting to detect WordPress site URLs. 2021-09-17 14:00:34: Starting URL detection 2021-09-17 14:00:34: Running WP2Static in Headless mode 2021-09-17 14:00:34: Running full workflow from UI 2021-09-17 13:58:30: Unable to fetch URL contents 2021-09-17 13:58:25: Starting to detect WordPress site URLs. 2021-09-17 13:58:25: Starting URL detection 2021-09-17 13:58:25: Running WP2Static in Headless mode 2021-09-17 13:58:25: Running full workflow from UI 2021-09-17 13:54:03: Unable to fetch URL contents 2021-09-17 13:53:58: Starting to detect WordPress site URLs. 2021-09-17 13:53:58: Starting URL detection 2021-09-17 13:53:58: Running WP2Static in Headless mode 2021-09-17 13:53:58: Running full workflow from UI 2021-09-17 13:45:04: Unable to fetch URL contents 2021-09-17 13:45:00: Starting to detect WordPress site URLs. 2021-09-17 13:45:00: Starting URL detection 2021-09-17 13:45:00: Running WP2Static in Headless mode 2021-09-17 13:45:00: Running full workflow from UI 2021-09-17 13:44:32: Unable to fetch URL contents 2021-09-17 13:44:27: Starting to detect WordPress site URLs. 2021-09-17 13:44:27: Starting URL detection 2021-09-17 13:44:27: Running WP2Static in Headless mode 2021-09-17 13:44:27: Running full workflow from UI

and error log with latest version of plugin `#8 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticguzzle/sr 2021-09-17 14:10:29.665439 [NOTICE] [3370032] [77.65.110.183:37731-Q:5371547D4AC5C5DF-48#new.skiddou.com] [STDERR] c/Middleware.php(63): WP2StaticGuzzleHttp\RedirectMiddleware->__invoke()

9 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticguzzle/src/HandlerStack.php(75): WP2StaticGuzzleHttp\Middleware::WP2StaticGuzzleHttp{closure}()

10 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticguzzle/src/Client.php(331): WP2StaticGuzzleHttp\HandlerStack->__invoke()

11 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticguzzle/src/Client.php(168): WP2StaticGuzzleHttp\Client->transfer()

12 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/vendor/leonstafford/wp2staticguzzle/src/Client.php(187): WP2StaticGuzzleHttp\Client->requestAsync()

13 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/SitemapParser.php(236): WP2StaticGuzzleHttp\Client->request()

14 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/SitemapParser.php(182): WP2Static\SitemapParser->getContent()

15 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/DetectSitemapsURLs.php(112): WP2Static\SitemapParser->parse()

16 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/URLDetector.php(71): WP2Static\DetectSitemapsURLs::detect()

17 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(633): WP2Static\URLDetector::detectURLs()

18 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(745): WP2Static\Controller::wp2staticHeadless()

19 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(301): WP2Static\Controller::wp2staticRun()

20 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(327): WP_Hook->apply_filters()

21 /home/new.skiddou.com/public_html/wp-includes/plugin.php(470): WP_Hook->do_action()

22 /home/new.skiddou.com/public_html/wp-admin/admin-ajax.php(187): do_action()

23 {main}

Next WP2Static\WP2StaticException: Unable to fetch URL contents in /home/new.skiddou.com/public_html/wp 2021-09-17 14:10:29.665453 [NOTICE] [3370032] [77.65.110.183:37731-Q:5371547D4AC5C5DF-48#new.skiddou.com] [STDERR] -content/plugins/wp2static/src/SitemapParser.php:239 Stack trace:

0 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/SitemapParser.php(182): WP2Static\SitemapParser->getContent()

1 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/DetectSitemapsURLs.php(112): WP2Static\SitemapParser->parse()

2 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/URLDetector.php(71): WP2Static\DetectSitemapsURLs::detect()

3 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(633): WP2Static\URLDetector::detectURLs()

4 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(745): WP2Static\Controller::wp2staticHeadless()

5 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(301): WP2Static\Controller::wp2staticRun()

6 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(327): WP_Hook->apply_filters()

7 /home/new.skiddou.com/public_html/wp-includes/plugin.php(470): WP_Hook->do_action()

8 /home/new.skiddou.com/public_html/wp-admin/admin-ajax.php(187): do_action()

9 {main}

Next WP2Static\WP2StaticException: Unable to fetch URL contents in /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/DetectSitemapsURLs.php:127 Stack trace:

0 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/URLDetector.php(71): WP2Static\DetectSitemapsURLs::detect()

1 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(633): WP2Static\URLDetector::detectURLs()

2 /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/Controller.php(745): WP2Static\Controller::wp2staticHeadless()

3 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(301): WP2Static\Controller::wp2staticRun()

4 /home/new.skiddou.com/public_html/wp-includes/class-wp-hook.php(327): WP_Hook->apply_filters()

5 /home/new.skiddou.com/public_html/wp-includes/plugin.php(470): WP_Hook->do_action()

6 /home/new.skiddou.com/public_html/wp-admin/admin-ajax.php(187): do_action()

7 {ma

2021-09-17 14:10:29.665458 [NOTICE] [3370032] [77.65.110.183:37731-Q:5371547D4AC5C5DF-48#new.skiddou.com] [STDERR] in} thrown in /home/new.skiddou.com/public_html/wp-content/plugins/wp2static/src/DetectSitemapsURLs.php on line 127 2021-09-17 14:11:00.795530 [INFO] [3370032] [skidd88363892.655340]: locked pid file [/tmp/lshttpd/skidd88363892.sock.pid]. 2021-09-17 14:11:00.795549 [INFO] [3370032] [skidd88363892.655340] remove unix socket for detached process: /tmp/lshttpd/skidd88363892.sock 2021-09-17 14:11:00.795617 [NOTICE] [3370032] [LocalWorker::workerExec] VHost:dev.skiddou.com suExec check uid 65534 gid 65534 setuidmode 0. 2021-09-17 14:11:00.795626 [NOTICE] [3370032] [LocalWorker::workerExec] Config[skidd88363892.655340]: suExec uid 1004 gid 1004 cmd /usr/local/lsws/lsphp74/bin/lsphp, final uid 1004 gid 1004, flags: 0. 2021-09-17 14:11:00.795909 [NOTICE] [3370032] [skidd88363892.655340] add child process pid: 3370860 2021-09-17 14:11:00.795969 [INFO] [3370032] [skidd88363892.655340]: unlocked pid file [/tmp/lshttpd/skidd88363892.sock.pid]. `

leonstafford commented 3 years ago

Hi @adrianratajczak, looks like we need to improve our error messaging when that particular exception is thrown, as it doesn't tell us the exact URL it's failing on, only that there was a TransferException.

Could you please list all the sitemaps your site has? And if it references the sitemaps in your robots.txt, if it exists.

adrianratajczak commented 3 years ago

That's my sitemap file https://new.skiddou.com/sitemap.xml

In my robots.txt sitemap is added

leonstafford commented 3 years ago

Thanks @adrianratajczak. I'm not yet sure of the issue, but in the meantime, you may try to remove the sitemaps reference from the robots.txt to see if that's the cause we're seeing here...

gsxryan commented 2 years ago

For those that land here and were a bit clueless like myself, these troubleshooting settings also help.

1) Check Wp2static > Diagnostics, resolve any issues (especially making sure /uploads is writable) 2) verify wp-content has full ownership inside the container. Do NOTE: php had an error i ignored. Even though i was within an acceptable version.

chown -R nginx:www-data wp-content/

I had copied files from the host and broke the file ownership. Symptoms presented as Themes and Addons being unable to be installed. Prompted for FTP login.
Once i fixed those two issues I was able to run OK. I had also went YOLO since it's a temp instance and created 777 permissions on those folder (probably not needed) and installed the and ran a quick SEO with Yoast.

ampliads commented 2 years ago

I am getting this error as well. My Environment : Wordpress 5.9 with Wp2Static 7.1.7 (No Addons) on Laragon Local dev. I tried with both RankMath & Yoast with no success. My error: 2022-02-21_10-12-17

My Wp2Static Log: 2022-02-21 10:12:13: Unable to fetch URL contents 2022-02-21 10:12:10: Starting to detect WordPress site URLs. 2022-02-21 10:12:10: Starting URL detection 2022-02-21 10:12:10: Running WP2Static in Headless mode 2022-02-21 10:12:10: Running full workflow from UI 2022-02-21 10:04:46: Unable to fetch URL contents 2022-02-21 10:04:44: Starting to detect WordPress site URLs. 2022-02-21 10:04:44: Starting URL detection 2022-02-21 10:04:44: Running WP2Static in Headless mode 2022-02-21 10:04:44: Running full workflow from UI 2022-02-21 10:03:51: Unable to fetch URL contents 2022-02-21 10:03:49: Starting to detect WordPress site URLs. 2022-02-21 10:03:49: Starting URL detection 2022-02-21 10:03:49: Running WP2Static in Headless mode 2022-02-21 10:03:49: Running full workflow from UI 2022-02-21 10:03:21: Unable to fetch URL contents 2022-02-21 10:03:19: Starting to detect WordPress site URLs. 2022-02-21 10:03:19: Starting URL detection 2022-02-21 10:03:19: Running WP2Static in Headless mode 2022-02-21 10:03:19: Running full workflow from UI 2022-02-21 10:02:54: Unable to fetch URL contents 2022-02-21 10:02:52: Starting to detect WordPress site URLs. 2022-02-21 10:02:52: Starting URL detection 2022-02-21 10:02:52: Running WP2Static in Headless mode 2022-02-21 10:02:52: Running full workflow from UI 2022-02-21 10:02:18: Unable to fetch URL contents 2022-02-21 10:02:16: Starting to detect WordPress site URLs. 2022-02-21 10:02:16: Starting URL detection 2022-02-21 10:02:16: Running WP2Static in Headless mode 2022-02-21 10:02:16: Running full workflow from UI 2022-02-21 10:01:42: Unable to fetch URL contents 2022-02-21 10:01:39: Starting to detect WordPress site URLs. 2022-02-21 10:01:39: Starting URL detection 2022-02-21 10:01:39: Running WP2Static in Headless mode 2022-02-21 10:01:39: Running full workflow from UI 2022-02-21 09:53:47: Unable to fetch URL contents 2022-02-21 09:53:45: Starting to detect WordPress site URLs. 2022-02-21 09:53:45: Starting URL detection 2022-02-21 09:53:45: Running WP2Static in Headless mode 2022-02-21 09:53:45: Running full workflow from UI 2022-02-21 09:50:35: Unable to fetch URL contents 2022-02-21 09:50:32: Starting to detect WordPress site URLs. 2022-02-21 09:50:32: Starting URL detection 2022-02-21 09:50:32: Running WP2Static in Headless mode 2022-02-21 09:50:32: Running full workflow from UI 2022-02-21 09:47:42: Unable to fetch URL contents 2022-02-21 09:47:40: Starting to detect WordPress site URLs. 2022-02-21 09:47:40: Starting URL detection 2022-02-21 09:47:40: Running WP2Static in Headless mode 2022-02-21 09:47:40: Running full workflow from UI 2022-02-21 09:47:01: Unable to fetch URL contents 2022-02-21 09:46:59: Starting to detect WordPress site URLs. 2022-02-21 09:46:59: Starting URL detection 2022-02-21 09:46:59: Running WP2Static in Headless mode 2022-02-21 09:46:59: Running full workflow from UI 2022-02-21 09:46:45: Unable to fetch URL contents 2022-02-21 09:46:43: Starting to detect WordPress site URLs. 2022-02-21 09:46:43: Starting URL detection 2022-02-21 09:46:43: Running WP2Static in Headless mode 2022-02-21 09:46:43: Running full workflow from UI

And the browser console view: 2022-02-21_10-08-26

Not sure what causing the error.

ampliads commented 2 years ago

SOLVED!

Dear @leonstafford Let me start with greetings! Disclaimer : we are more a digital & SEO agency and not the "hard coders". So, Accept our apologies in case we hurted the "coder's heart" (😄)

After some research and workaround, we have somehow managed to solve the issue. We consistently tried to install wp2static from the downloaded package from the official site. It Didn't work. Finally, we went on black white and green, the console way, cloned the GIT repo, ran composer install, and it worked like a charm!

Moreover, in your Cloudflare Workers Plugin we were not able to install through composer build, but it worked well with composer install Hope this will help other noobs like us!

AmpliAds

gabrfernandez commented 2 years ago

@ampliads Did you solve the 500 error code? I'm having the same issue. Unable to fetch URL contents.

ampliads commented 2 years ago

Hi @gabrfernandez , Yes. We are able to solve it. May you give your working environment details with the log. In our case, the error was due to plugin files. I suggest you to use git clone and then run composer install In case you are not familiar with GIT then simply download the git repo as zip > unzip it into wp-content/plugins (it should create wp2static folder) > in terminal : get into directory and run composer install (provided you have composer installed and set) You may put the log output with browser console error Screenshot if possible. Happy Wordpressing! 😊

john-shaffer commented 2 years ago

@ampliads Glad you got it working! There hasn't been a release in some time, and there are a lot of fixes in the current git repo. In case it helps you, @gabrfernandez, I've attached the most recent build. wp2static-00582cf74c26a486538355e9f78338fd1bb86623.zip

ampliads commented 2 years ago

@john-shaffer You are welcome! Personally, I feel, many wp users are marketing agencies / smo or tech noobs (just like us), may not be comfortable with git and the blacks, whites & greens (Collectively known as "the console" 😊) How about setting up instructions or creating a public blog with steps to get Git repo as zip!

leonstafford commented 2 years ago

Thanks for the feedback, @ampliads! Sorry, I'm playing catchup with questions and releases at the moment.

As of few days ago, you can grab latest release from wp2static.com/download, by filling out a form, it will get emailed to you. I'll get the latest changes bundled into a release soon for easier downloading. Thanks for your patience with all this and good job getting things working!

ampliads commented 2 years ago

Dear @leonstafford ! I must say, the wp2static is doing a marvellous job! Wish long live to you and Wp2Static Team! Yes, Initially, I tried the download link and downloaded from the mail ID link only. But It didn't work. It is quite understandable. How about making an explainer video or a blog with a proper How-to (that will be "nearly once for all" ) Also, May you suggest us where we can have deeper dive and hands on experience for GithubPages Deployments? We have keen interest to move our sites on Github / GitLab Repo + Cloudflare Pages (We wisht o exclude the cloudflare workers) Aplologies for asking such qery on wrong thread.