Failures at various steps without explanation

[abakonski]

Hi. Having lots of issues with generating a static site. I've experienced failures at crawling inital file list, analyzing site, preparing files to send to S3, etc.

I've been able to get past the failure on crawling initial file list by refreshing the page and retrying the export multiple times. By changing deployment to local folder, after about 30 or more tries and refreshes and clearing deployment caches etc, I eventually got the plugin to build the static site one time.

Since then, I've tried and retried tens of times and I'm still getting all sorts of failures at the above points and others. All of these failures occur without any clues left behind in the plugin logs (with and without full debug mode enabled) and there's also nothing in PHP logs.

Is there a way to identify the source of these issues? Flying blind right now.

[abakonski]

@leonstafford do you have any tips/advice on how to debug these failures? Right now they're silent errors with a generic failure message. Any pointers here would be very helpful. Thanks

[leonstafford]

Hi @abakonski - there's a Debug Mode, enabled within the Advanced Tab, that may help.

When there are no clear error messages in your webserver or PHP error logs, this can often be a sign of a resource limit being reached. If not the PHP execution limit (set low on some hosts), then a webserver or load balancer can have various timeout limits.

Increasing those limits or adjusting WP2Static's "Crawl Increment" setting to crawl faster may overcome those.

Development is happening right now on version 7 of WP2Static, which aims to solve a lot of these common problems, but please try the aforementioned and let me know if any results.

[abakonski]

@leonstafford thanks for the hints. Unfortunately Debug Mode doesn't help. Logs just stop suddenly. PHP and NginX logs show nothing. I have execution and timeout times set to over an hour in PHP, NginX and load balancer. Failures tend to happen after 40-60 mins.

Looking at the ajax calls in the web console, just before the failure, I tend to get "net::ERR_NETWORK_IO_SUSPENDED" or "net::ERR_NETWORK_CHANGED" errors from calls to admin-ajax.php and also WP-STATIC-PROGRESS.txt simultaneously.

There are a LOT of images in the media gallery of this site (a little over 12,000 if counting all the various sizes). Not sure if that causes the long processing time, but I have a filter configured to exclude the entire /wp-content/uploads/ path since it's all hosted on AWS (EFS but cached in CloudFront) and I don't need these copied/exported. There are no more than 100 posts/pages in the blog. What's a typical export duration for something of this size, exporting to a local folder?

Any other tips?

Is there an ETA on version 7?

[leonstafford]

Hi @abakonski - that does sound very slow for not such a big site. Did you try increasing that Crawl increment setting to get through things quicker?

You may also try the WP-CLI commands, which may bypass whatever resource limit we're seeing here: https://wp2static.com/developers/wp-cli/

ETA on V7 is Q1 2020

[abakonski]

@leonstafford - Yes I've been increasing it incrementally. I finally got it to complete successfully with the Crawl Increment set to 1000. It completed in 62 mins. It hit 200+ 502 errors on WP_STATIC_PROGRESS.txt along the way. I still need to validate the results, but an hour each time I make changes to the site isn't very appealing.

Any thoughts on how I could speed things up?

[leonstafford]

Hi @abakonski,

Version 7 is getting closer to public testing and will really speed things up via:

• auto-deploy just changes on post updating
• using DB vs flat-files for queues, logging, etc
• crawling/deploying only changes since last attempt
• much simpler logic throughout plugin core (doing less each cycle)

So, those are going to be great, but it's still hard for me to ascertain how much of your export speed delays are the plugin or the site related.

I started some work last year on benchmarking various hosting providers by setting up an identical WP site on each and timing WP2Static exports. For the same $5/month of common VPS's/hosting options, performance varies wildly. I didn't pursue this benchmarking, may get back to it once the dust settles after V7 release, but I'd suggest cloning the site to something like DigitalOcean or Vultr.com's $5/month instances and compare the export speed. If it's still slow, try to isolate your site as the cause by using a default WP site and the FakerPress plugin to generate random content and increase the weight.

There are config tweaks to PHP-FPM, Nginx, etc that may help, but it's a rabbit hole to go down, ie adjusting PHP-FPM pool to use static workers can help max out the server's resources, but takes trial and error.

I'm also working on the side on a free dev site hosting solution I can offer people, so there's a consistent environment that for WP2Static users without public traffic to their dev sites, I should be able to provide

[abakonski]

Thanks @leonstafford Really looking forward to testing out v7. The WP I'm testing the plugin on is on a dedicated EC2 box. Nothing too crazy, but it should definitely be up to the task.

Will v7's logging be more verbose in debug log? i.e. timestamped + what step is taking place? That should help with hinting at bottlenecks.

[leonstafford]

Hi @abakonski - yes, we've got an easily accessible screen now, as below. This one is just showing some stuff I was debugging and will get tidied up further & allow for downloading as file/zip.

[leonstafford]

@abakonski ready to try here: https://github.com/WP2Static/wp2static/releases/tag/7.0-alpha-003

[abakonski]

Thanks @leonstafford I should have some time to test this build out next week. Appreciate the update

[leonstafford]

@abakonski closing this one for now, as I've moved v6.x.x issues into this repo, where that project will continue, then we can create new specific issues as you encounter them either here for V6 or with V7 (https://github.com/wp2static/wp2static)