Slow page load times (TTFB ~20-30 seconds)

[cagross]

Expected Behavior

Current Behavior

I have built a new, one-page site in VVV. When I load that page, I am experiencing longer-than-usual load times: ~20-30 seconds. The Network tab indicates that most of that is due to TTFB (screenshots). I'm not sure if this is a VVV issue, or some other issue. Perhaps VVV has methods to troubleshoot the issue?

I have pushed the same site to my remote web server. When I load the page there, the TTFB is much lower, in the 3-5 second range.

Possible Solution

Steps to Reproduce (for bugs)

1. Currently I can reproduce the issue every time I refresh my home page.

Context

Your Environment

• VVV version: 2.4.0-4-g84e9f4b
• VVV Git Branch: master
• Vagrant version: 2.2.2
• VM Provider name: VirtualBox
• VM Provider version: Version 5.2.22 r126460 (Qt5.6.2)
• Operating System and version: Win 10 x64

[tomjn]

The Network tab indicates that most of that is due to TTFB (screenshots). I'm not sure if this is a VVV issue, or some other issue

So TTFB could be caused by anything here, it could be that you're using a spinning disk drive and it's busy, it could be that your VM has less memory than the server at the other end, or that you're making lots of remote requests. It's not possible to say

So more information is needed, poor performance is a very generic term, we need to isolate it further, e.g. poor filesystem performance, or poor network, poor database performance, etc

So:

• What are the specs of your machine, vs the machine in the data centre? RAM/CPU/disk type in particular
• What does the page do?
• We need more information about what's happening during the request, install Query Monitor, possibly even LAPs, and if you can acquire XDebug information that'd be good too

[cagross]

or that you're making lots of remote requests.

OK interesting that you say that--I did not know that. My page is making a relatively large amount of remote requests--all to Instagram to fetch images. That is the result of one particular plugin. If I deactivate that plugin, the TTFB seems to drop significantly--down to 4-5 seconds, which is more expected. I've done a handful of tests now and it looks like I can consistently reproduce that. So let's assume this is the cause of my TTFB issue.

As a follow-up though, the same number of remote requests are being made on my remote site. Why might the TTFB on that site be relatively unaffected, while my VVV site incurs a relatively big TTFB hit? Are remote requests from a local VVV site amplified for some reason compared to the same remote requests from a site on a web server? Or can my local computer specs affect remote requests that much (including internet connection)? Or neither? Or something else?

[tomjn]

Remote requests and poor performance usually go hand in hand, you'd want to cache the results or push them client side regardless of where it ends up. Which plugin did you deactivate that reduced TTFB so dramatically?

Are remote requests from a local VVV site amplified for some reason compared to the same remote requests from a site on a web server?

I would expect some overhead from the VM and VirtualBoxes network adapter, but nowhere near that level of throttling. I was thinking more like a 5MBps falling to 4.9 or 4.5Mbps at most, or maybe an extra few ms of latency.

Or can my local computer specs affect remote requests that much (including internet connection)?

Oh definitely, run VVV on an SSD and it'll be significantly faster to provision and serve pages than a HD or a USB drive, but that's true of any software. Similarly I would expect that page to take a very long time to generate if you're doing local dev on an airplane with wifi turned off.

Additionally, a remote server will be in a data centre with a fat pipe, and fewer hops to instagrams servers. I doubt VVV will ever compete directly with a decent fresh VPS at a major host purely from a physics and $$$ point of view. Those VMs might be containers, or they'll be running on a tier 1 Hypervisor such as Xen or ESX rather than a tier 2 HyperVisor such as VirtualBox or VMWare Workstation, I'd also expect they'd be running much more expensive CPUs such as Xeons than most consumer hardware.

Having said that, you can't make everybody buy a 20 core CPU on contributor day, or haul a 1kw power supply on a train :) But I'm sure there are things that can be done to improve what we have

Or neither? Or something else?

Something else that struck me is that TTFB being high suggests that there are output buffers preventing things from being sent until the page has finished generating so that post processing can occur, e.g. the Autoptimise plugin.

Undoing that won't make the page any faster to generate, but it will reduce TTFB. Any benefits those plugins give you are usually not worth it when you take into account that the browser can load things in parallel before the page has even finished sending ( great for increasing page speed scores quickly though )

Since you're doing things with Instagram, it might be possible to push that entirely client side, else I would suggest these other options:

• making the API requests in a cron job so that the frontend only renders what's already been fetched
• caching the response for 10 minutes, or even an hour. Having something like IFTTT fire a webhook that flushes that cache could let you have it cached for days yet instantly update when a new post is made
• make the requests in JS and render them client side
• Make the instagram queries in AJAX requests, use a REST API endpoint to do that part. This way even if Instagram takes 5 minutes to respond, the TTFB is nearly instant, and similar responses can be cached

All 3 options are faster, and will scale better regardless of where it's hosted.

As for the plugin you identified, what's it's job? I'm curious if there's anything that can be done on that front to make things a bit easier to work with

[cagross]

Hey thanks very much for all that info, and sorry for the late reply.

Which plugin did you deactivate that reduced TTFB so dramatically?

Well the plugin is Photonic Gallery & Lightbox for Flickr, SmugMug, Google Photos, Picasa, Zenfolio and Instagram. But I should first divulge that I am at least partially at fault for all of the remote requests from the plugin. I am in the process of building my site, and trying to get the Instagram section to look the way I want. So while I do that, I've actually been displaying two separate Instagram feeds on this particular page, one with configuration A and the other with configuration B. Once I settle on a configuration, I'll remove the other. Plus, at least one of those Instagram feeds (maybe both) is currently displaying a lot more images than it should. I'm trying to get the 'limit' parameter to work, but it's not--so each feed is fetching and displaying the default number of images, which I believe is 40. I want to eventually display only 20 images.Once I finalize the configuration of the plugin, I'll try to run another test and compare the dev site TTFB to my live site.

edit: Actually, now that I think about it, there's even a third Instagram feed displayed on the page--in the footer--that makes even more remote requests (it displays 8 images). That actually uses a different plugin. The plan is to eventually replace that with the existing Instagram plugin, so hopefully there are no duplicate remote requests from Instagram.

FYI my live site is on a shared server from SiteGround (the lowest-tiered shared hosting plan).

Something else that struck me is that TTFB being high suggests that there are output buffers preventing things from being sent until the page has finished generating so that post processing can occur, e.g. the Autoptimise plugin.

Right. I'm don't have any explicit optimization plugins installed. But who knows, maybe my theme, or this particular Instagram plugin, have such 'features.' I do have a security plugin installed--Wordfence. It is installed on both dev and live site, but I'm not entirely sure that it is configured identically. Maybe I can try disabling on the dev site as a test.

What about adding some code that will defer all JS requests? Would that help alleviate the issue?

making the API requests in a cron job so that the frontend only renders what's already been fetched caching the response for 10 minutes, or even an hour. Having something like IFTTT fire a webhook that flushes that cache could let you have it cached for days yet instantly update when a new post is made make the requests in JS and render them client side Make the instagram queries in AJAX requests, use a REST API endpoint to do that part. This way even if Instagram takes 5 minutes to respond, the TTFB is nearly instant, and similar responses can be cached

OK thanks very much for all those. But in this case, wouldn't most (or all?) require me to modify the core files of this particular plugin? Or would it be possible to implement those four suggestions by way of my my child theme, without altering any plugin files? If anything, the second option (using IFTTT) might be something I can do without altering plugin files. Also, interesting to know about IFTTT--didn't know it could be used in that manner.

As for the plugin you identified, what's it's job?

Well it just serves to fetch some images from my Instagram page and display them nicely on my page, all while being responsive, and giving me a fair amount of customization options (e.g. different layouts, different lightbox galleries, etc). Maybe I can run this TTFB issue by the dev--he seems pretty helpful. But if finds out that the issue is not present on my live site, he may file it in the not-my-problem department. Who knows though.

I guess we can close this issue? Or do you want to do some more troubleshooting/investigation? I'm happy to do that if you want. Or we can call it closed and move on. Either way.

[tomjn]

You could use something like instafeed.js and do it clientside without any plugins. I'm also unsure of how WordFence does some of what it does so it's possible but I couldn't comment.

Eitherway it might be easier to just get an access token, and make the requests to the API yourself

[cagross]

OK understood on both, thanks. I have a small update. I carried out two more tests:

1. I finalized my Instagram feed and deleted the second, extraneous feed. In 12 consecutive page loads after that, the average TTFB was 14 seconds. Individual times were:

17 10 10 13 15 15 15 15 15 15 16 15

2. I disabled the Wordfence plugin, and that lowered the average TTFB down to 9 seconds. Individual scores:

4 7 13 7 7 14 5 12 13 12 12 7

The standard deviation on the second set is a bit high--not sure why they fluctuated like that.

Just giving some more data here. Another test I guess I could do is disable all unnecessary plugins, and switch to the default Twenty Nineteen theme.

[tomjn]

DId you install the query monitor plugin? It or the laps plugin will tell you how much time remote requests took. Knowing the total time is not particularly helpful and doesn't tell us much that we don't already know.

LAPs will also tell you a lot of information about what was happening in a server request, right now all we know is that taken in its entirety something happened, and it took X seconds. More finegrained data is needed to draw any more conclusions, or identify what the performance issue is, else we're just grasping at straws.

Eitherway, where remote requests are involved, you're at the mercy of your internet connection and instagrams servers. It could be that their API is slow, or your network is slow, who knows without more information

[tomjn]

e.g. on my own sites homepage:

Compared to the WP Admin updater showing large chunks of time spent on remote requests

Or with query monitor, we can get a list of queries, and remote requests, with timings:

Without this information, it could be that the culprit is this:

<?php
sleep( 5 );

There's no way of knowing from the frontend

[cagross]

DId you install the query monitor plugin?

No. My mistake, I overlooked that part. I just installed Query Monitor and see some interesting results (screenshot). TTFB was ~9.5 seconds. But only ~2.2 seconds of that was from remote requests ("Photonic" is the name of the Instagram plugin I'm using). Thoughts on that? Is it possible to obtain information on the other ~7 seconds of TTFB? Also, I'm not sure what the "Certificate verification disabled" entry is specifying, or if it's something I should resolve.

Not to steer this ticket in another direction, or add scope creep to it, but I should perhaps say that, now that I'm researching this, I am also experiencing unexpectedly long TTFBs on nearly all of my back-end admin pages. ~4-8 seconds on each page load. Here is an example from loading Dashboard-->Users. Perhaps the two are related, or perhaps they are not. Just throwing this out there for best possible analysis of the issue.

[tomjn]

@cagross I don't think the chrome inspector tools view of TTFB, or any client side metrics are particularly helpful, I'd focus exclusively on server side metrics.

I'd also establish a baseline, right now you're talking about an install that has a tonne of unknowns that only you can see ( some of which you may not even consider relevant, but how would you know ).

So set up a standard WP with the default theme, and measure on that. If TTFB is still an issue there, then we can continue. Else it's quite unlikely to be VVV.

Also, there's no information about queries, time spent in hooks, etc. Query Monitor provides a lot of information, if it's taking a long time to generate results, what exactly is it doing? Poor performance is too generic and vague, I need you to be more specific, and time to first byte isn't a problem, it's a symptom

Additionally, is TTFB a problem for readme.html? Non-WP PHP scripts?

For reference, the custom site template with the default theme gave me a TTFB of 276ms over http and took 3ms to transfer:

readme.html gave 14ms

https://tomjn.com served from a memcached page cache on a Digtal ocean VPS/Nginx had a 253ms TTFB

[tomjn]

I installed and ran WP Performance Tester, the same plugin used on the reviewsignal website to benchmark hosts.

I would suggest using this fork I made that fixes some PHP warnings https://github.com/tomjn/WPPerformanceTester/tree/tomjn_fix_warnings

Here's the results on my VVV for stock WP 5:

The bulk of the time is spent in MySQL queries ( 7s ), but overall my instance outperformed the average. My personal site in comparison took 13s to perform the query part of the test

[cagross]

I don't think the chrome inspector tools view of TTFB, or any client side metrics are particularly helpful, I'd focus exclusively on server side metrics.

Right O. What about the page generation time that Query Monitor displays? Would that be a better metric?

Additionally, is TTFB a problem for readme.html? Non-WP PHP scripts?

Good question. No. I just checked for readme.html and license.txt--no issues at all.

Poor performance is too generic and vague, I need you to be more specific, and time to first byte isn't a problem, it's a symptom

OK I hear you, and understand. I am indeed trying to ascertain more specific information.

So set up a standard WP with the default theme, and measure on that. If TTFB is still an issue there, then we can continue. Else it's quite unlikely to be VVV.

OK. What about using the wordpress-default site that comes with VVV? That is of course devoid of any extra plugins and media, but it also should have a reasonably simple and default database. That site loads blazing fast for me, as expected. Doesn't exhibit any TTFB at all. Conversely, if I switch my site to the default 2019 theme,

I did some more tests on the default VVV WordPress site, and can perhaps better articulate the issue I'm seeing. With the default VVV WordPress site, there are no page load speed issues--the home page has page generation times in the 0.01 s range (or less), as expected. If I then install and activate different plugins, one by one, the page generation time increases noticeably. Some plugins cause it to increase more than others. Activating Wordfence causes a ~2 second increase in page generation time. So once I install and activate all 12 of my plugins, the page generation time jumps to the 8-10 second range. The same is not true on my remote server. On my remote server, if I deactivate all plugins and switch to the default 2019 theme, the page generation time for the home page drops to ~0.1 s, as expected. If I then install Wordfence, the page generation time simply jumps to ~0.2-0.4 seconds, not the 2-3 seconds that occurs on the default VVV site. Can you replicate that on your end?

To be clear, I'm not saying Wordfence itself is the culprit for the long page generation times on the VVV site. I'm saying that installing and activating any plugin causes a noticeable jump in page generation time. Wordfence is just a good test because on my end, it seems to cause a large, and easily identified jump.

FYI I just now am seeing your new reply about the performance tester plugin. I can give that shot on my dev site soon. It may have to wait until tomorrow though.

[tomjn]

Right O. What about the page generation time that Query Monitor displays? Would that be a better metric?

Helpful, but not too much, these are broad strokes, too vague.

For example, lets say you're complaining that a delivery took 2 weeks. Measuring total delivery time, and analysing your own house won't be particularly helpful in diagnosing why it took 2 weeks. All it does is confirm that it took a long time.

You're in the same boat, client side metrics have given you everything you could have gotten already. Now you need to move to diagnosing the cause, be that remote requests, file system manipulation, queries, or something else. Poor performance is too vague, more digging needed

OK. What about using the wordpress-default site that comes with VVV? That is of course devoid of any extra plugins and media, but it also should have a reasonably simple and default database. That site loads blazing fast for me, as expected. Doesn't exhibit any TTFB at all. Conversely, if I switch my site to the default 2019 theme,

I did some more tests on the default VVV WordPress site, and can perhaps better articulate the issue I'm seeing. With the default VVV WordPress site, there are no page load speed issues--the home page has page generation times in the 0.01 s range (or less), as expected.

This suggests this isn't a VVV problem at all, and it's to be expected that adding plugins would increase page load. The extra work plugins do isn't free afterall :)

Activating Wordfence causes a ~2 second increase in page generation time. So once I install and activate all 12 of my plugins, the page generation time jumps to the 8-10 second range. The same is not true on my remote server. On my remote server, if I deactivate all plugins and switch to the default 2019 theme, the page generation time for the home page drops to ~0.1 s, as expected. If I then install Wordfence, the page generation time simply jumps to ~0.2-0.4 seconds, not the 2-3 seconds that occurs on the default VVV site. Can you replicate that on your end?

I have no idea which plugins, what they're doing, how they work, or how they're configured. Personally I would avoid WordFence ( though it doesn't make much sense running it on a local environment ).

Using the analogy from earlier, the delivery supervisor has discovered that the delivery man logged a number of items labelled ??? that increased delivery time. Further investigation needed. E.g. is this time spent doing queries?

Also keep in mind that you don't know what's going on at the remote end, there may be mitigations that speed things up, or there may be one component of the system that's significantly faster.

But until you identify the performance bottleneck, which you still haven't done, this is all speculation. Telling me performance is slow is unhelpful. Telling me database queries are slow on the other hand, or that file system access is slow, etc is super helpful.

I'd also note that you should not expect 2 different systems to perform the same. VVV and SiteGround will never have the same performance in all areas. Even 2 VVV instances will never have the same performance. My SSD and memory may be faster than yours, you might have a newer computer. My internet might be running over fibre and yours over copper.

It could even be as simple as your shared host having a basic object cache installed, or the plugins behaving differently ( or even using PHP 5.6 in VVV, and 7.2 in production )

[tomjn]

Wordfence is just a good test because on my end, it seems to cause a large, and easily identified jump.

I would also note that this is not specific to you, there's a number of people complaining about jumps in TTFB after updating WordFence

[cagross]

This suggests this isn't a VVV problem at all, and it's to be expected that adding plugins would increase page load. The extra work plugins do isn't free afterall :)

I'd also note that you should not expect 2 different systems to perform the same. VVV and SiteGround will never have the same performance in all areas. Even 2 VVV instances will never have the same performance. My SSD and memory may be faster than yours, you might have a newer computer. My internet might be running over fibre and yours over copper.

I'm not expecting the two systems to perform the same. But if the site on the remote server outperforms the VVV site, why would I continue to use VVV? Please don't take this as an attack or harsh criticism. But here is the reality from my point of view. I used to develop new sites on VVV because it was fast--back end and front end pages loaded much faster compared to any remote web server. But in this case, simply after adding a few plugins, the VVV site has become significantly slower than the same site on the remote web server. So in this case, what's the benefit to me of using VVV? There is, in fact, a large penalty for using VVV.

But until you identify the performance bottleneck, which you still haven't done, this is all speculation. Telling me performance is slow is unhelpful. Telling me database queries are slow on the other hand, or that file system access is slow, etc is super helpful.

I would personally love to troubleshoot this with you, if you think this is an issue that can be resolved. But I would admittedly need some guidance on your end. For example, on my VVV site, on this particular page load (screenshot, Query Monitor is reporting a Page Generation Time of 8.7 s. Only 0.01 s of that was from Database Query Time. Another 2.6 s was from HTTP API calls. How can I determine the duration of other tasks? You mentioned time spent in hooks--how can I see that in QM? There is a "Hooks & Actions" section, but that doesn't appear to show the time (screenshot).

If you don't think this is an issue worth pursuing, or if you believe this to be expected behavior--no problem, I understand. I'd just have to cease using VVV in favor of building directly on my web server. That's not a threat, nor is it said out of spite, nor is it trying to force your hand. I think in reality, if others are experiencing similar issues, I'd expect them to make a similar decision. Who knows though, this could just be an issue with my machine only.

I am indeed using an SSD drive on my local computer by the way. I also ran your performance plugin on my site, with these results.

[tomjn]

Did you try the LAPs plugin? That gives a lot more visual information about what is taking up time, including main hooks etc, if it's not remote requests or database writes, I can only think of 2 other things:

• your production site is using an object cache, in which case, dropping the memcached object cache dropin into the site would tell us if that's the case
• filesystem performance

Keep in mind that whatever might be causing issues with VVV could also happen elsewhere under heavy traffic. For all we know, there's a toggle in a plugins options page that's unchecked on one but not the other. It could even be as simple as an overzealous antivirus, those are known to severely impact filesystem performance

[cagross]

Did you try the LAPs plugin? That gives a lot more visual information about what is taking up time, including main hooks etc,

OK I just installed and began testing with it. It does indeed give more info than Query Monitor.

For the time being, I'm testing on my VVV site. I should probably be testing on the default VVV site, but here is what I have for now. If I switch to the default 2019 theme, and deactivate all plugins, the home page loads with no apparent delay. If I activate all of my 12 plugins except for

• Wordfence
• Photonic (Instagram plugin)

Then home page load speed rises ever so slightly, but still remains in the <1 second range. That is expected and desired.

Wordfence plugin: If I then activate Wordfence, the page load speed rises to ~4 seconds. Laps indicates that the biggest culprit is "Plugins Load," which jumps to >2 seconds (screenshots). I don't understand why that time increases though--doesn't Laps indicate that 2 MB is loaded during this process? What's curious is that Laps indicates '2 MB' for this process, regardless of how many plugins are actually activated. Here is a screenshot with only one single, small plugin activated. I'm not sure why that remains at 2 MB. Separately, after activating Wordfence, the time spent on the init hook also jumps significantly. Furthermore, after activating Wordfence, all back-end admin pages also experience a similar increase in load times. I guess that is probably expected though.

Photonic plugin: If I instead activate the Photonic plugin, the page load speed jumps to ~5-6 seconds. Laps indicates the main culprit in this case is 'Main Loop' (screenshots). As expected, there are no changes to the back-end page load times--they do not increase.

Theme: If instead I de-activate all plugins, and then activate my main theme, the page load speed jumps a little, but remains under 1 second, which is fair and expected. The same occurs if I activate my child theme.

your production site is using an object cache, in which case, dropping the memcached object cache dropin into the site would tell us if that's the case

I could do that as a test. But how would I go about doing that?

filesystem performance

How would I be able to test this?

For all we know, there's a toggle in a plugins options page that's unchecked on one but not the other.

If I'm pushing the local DB to the live DB, don't we have to assume that the settings are identical?

It could even be as simple as an overzealous antivirus, those are known to severely impact filesystem performance

I can try disabling my antivirus (Windows Defender in this case).

[tomjn]

On my machine installing wordfence on an empty site increased the homepage from ~0.7s to ~1.1s, similarly photonic did the same, but since neither plugins functionality was actually in use I don't consider it a meaningful test.

I could do that as a test. But how would I go about doing that?

Same as any other WP install, acquire the Memcached WP object cache dropin and place it in wp-content, just as you would with a custom sunrise.php or any of the other dropins. I believe QueryMonitor said as much in its note on the main page.

I cannot account for the differences in performance at the moment, my suspicion based on what we've seen is that WordFence and Photonic are responsible, and it may be filesystem performance.

But I would advise against WordFence and Photonic, both in VVV, and on your server, or even in general. Security can be gained by following best practices, and plugins like WordFence add a lot of overhead with minimal benefit. WordFence won't protect you from missing escaping in your theme, broken or missing validation on remote requests, misconfigured http security headers, or endpoints that never bother to check if the user has the ability to do something.

I've dealt with a lot of large enterprise grade sites, and none of them run WordFence. In my opinion, it's a false sense of security. Some of the things it protects against are known vulnerabilities, but it's better to remove or fix issues than to cover them up with WordFence. Other features such as brute force protection are provided in smaller faster plugins, or even via Jetpack, 2FA is available as a core feature plugin, and the WordFence scanning is implemented on their own server not yours. None of it makes sense on a local environment.

As for photonic, for what you're doing, it would be much faster to do it client side in JS, or to just make the queries yourself. There are enough examples out there of how to do it. The way photonic is doing it with remote requests means that any performance metrics will be unreliable and at the mercy of network conditions. What works amazing today may not work so great if Instagram goes down or responds slowly, bringing your site down with it

I don't understand why that time increases though--doesn't Laps indicate that 2 MB is loaded during this process? What's curious is that Laps indicates '2 MB' for this process, regardless of how many plugins are actually activated. Here is a screenshot with only one single, small plugin activated. I'm not sure why that remains at 2 MB.

The time increases because it took longer to process. WordFence is doing things, and nothing is free. The 2MB indicates that at the end of that section, 2MB had been used in total to generate the page. This way you can see that if one section says 2MB, and the next finishes with 20MB, then you know which stage of the load process is taking up all the memory and where to look. Keep in mind it's a timeline, not a linear pie chart.

How would I be able to test this?

For filesystem performance, keep in mind that VMs have performance tradeoffs in this particular area, as do Docker containers. A common tactic is microcaching, but if realtime updates are desired, that means trading caching/performance for CPU time, which is what docker does.

With this in mind, you can use a standard linux filesystem benchmarking system once SSH'd into the VM via vagrant ssh, I haven't figured out which one I prefer and is the simplest yet though.

The other part, is that those tradeoffs apply to mounted folders. FS performance of /srv/www and /vagrant inside the VM won't be the same as /tmp or /home. The latter 2 would be faster, but then how would you modify your files from the host OS

[cagross]

Same as any other WP install, acquire the Memcached WP object cache dropin and place it in wp-content, just as you would with a custom sunrise.php or any of the other dropins. I believe QueryMonitor said as much in its note on the main page.

OK. Yes I see that QM notifies me:

! External object cache not in use The Memcache extension for PHP is installed but is not in use by WordPress The Memcached extension for PHP is installed but is not in use by WordPress The Zend OPcache extension for PHP is installed but is not in use by WordPress

I have never carried this out on any site, much less a VVV site, nor have I worked with dropins. Do you have any instructions you can point me to that would help me put the Memcached extension into use on my VVV site? I've seen this page--do I simply need to copy object-cache.php to wp-content?

But I would advise against WordFence and Photonic, both in VVV, and on your server, or even in general.

OK. I hear you on both, and you haven't said anything inaccurate. But the fact is, neither plugin is causing any noticeable issue on my live site. So in this case, it's difficult to justify the extra work required to remove the plugins and implement the alternatives. I'm happy to continue troubleshooting on the VVV site though--thanks for the help so far. I would love to see some closure to this, regardless of what it is.

The time increases because it took longer to process. WordFence is doing things, and nothing is free. The 2MB indicates that at the end of that section, 2MB had been used in total to generate the page. This way you can see that if one section says 2MB, and the next finishes with 20MB, then you know which stage of the load process is taking up all the memory and where to look. Keep in mind it's a timeline, not a linear pie chart.

OK got it--that all makes sense. Thanks for the clarification.

With this in mind, you can use a standard linux filesystem benchmarking system once SSH'd into the VM via vagrant ssh, I haven't figured out which one I prefer and is the simplest yet though.

OK. I've never done this before. Are you able to point me to at least one or two that you have used in the past?

I cannot account for the differences in performance at the moment, my suspicion based on what we've seen is that WordFence and Photonic are responsible, and it may be filesystem performance.

OK. It's clear from my tests that activating (and configuring) Wordfence and Photonic certainly result in a noticeable decrease in performance. But I agree that it would be prudent to fully test Memcached, and my file system.

[tomjn]

hmm still working on filesystem benchmarking, but a note to keep in mind is that tools such as Query Monitor and LAPs rely on the WP HTTP apis being used, e.g. wp_remote_get. A quick check of WordFence suggests a big chunk of curl usage, so WordFence may or may not be making remote requests, and it wouldn't show up in the tooling

[cagross]

OK I'll keep that in-mind about Wordfence, thanks. Keep me posted on the file system benchmarking.

[cagross]

Also, if you can follow-up about my Memcached questions, I could make progress on that testing as well.

[cagross]

Some more testing. With Photonic and Wordfence deactivated, the database push from local to live requires ~1:45. With Photonic and Wordfence activated, the database push requires ~5:00 minutes. I'm using the WP Sync DB plugin to sync the two databases.

[tomjn]

Also, if you can follow-up about my Memcached questions, I could make progress on that testing as well.

There's more than one dropin to choose from that implements memcached object cache, it's a case of personal preference, but the process is the same for VVV as any other WordPress site, or plugin. Pick one then copy and paste the file into wp-content, done. Similar to how WP Supercache puts an advanced-cache.php in wp-content, or how some sites put a custom sunrise.php or pluggable.php goes in wp-content.

See the installation instructions of https://wordpress.org/plugins/memcached/ for example, although competing ones exist elsewhere

VVV's custom site templates don't do this automatically because most hosting environments don't come with Memcached

[cagross]

Pick one then copy and paste the file into wp-content, done.

OK. I'm trying to follow the instructions here. The three steps are below. Do I need to carry out either of the first two steps? Or have they already been done on my VVV site?

_1. Install memcached on at least one server. Note the connection info. The default is 127.0.0.1:11211.

2. Install the PECL memcache extension

3. Copy object-cache.php to wp-content_

[tomjn]

nope they've already been done, it's just the dropin that needs putting in the appropriate place. You'll see in Query Monitor confirmation when it's done, otherwise there'll be no UI changes except that the plugins page will list object-cache.php under dropins

On Fri, Jan 4, 2019 at 3:03 PM cagross notifications@github.com wrote:

Pick one then copy and paste the file into wp-content, done.

OK. I'm trying to follow the instructions here https://github.com/Automattic/wp-memcached. The three steps are below. Do I need to carry out either of the first two steps? Or have they already been done on my VVV site?

_1. Install memcached http://danga.com/memcached on at least one server. Note the connection info. The default is 127.0.0.1:11211.

1.

Install the PECL memcache extension http://pecl.php.net/package/memcache 2.

Copy object-cache.php to wp-content_

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1695#issuecomment-451467966, or mute the thread https://github.com/notifications/unsubscribe-auth/AADl509OKQP-QPmy8iyYprAIuZIH6FrPks5u_20ngaJpZM4ZkGAR .

[cagross]

OK. I did that, and managed to get memcached implemented. Here's a screenshot of QM, and it also appears in Dashboard-->Plugins-->Dropins. Alas, home page load speeds are still in the 8-10 s range. Thoughts on that? Is there any way to confirm the memcached is actually functioning as expected?

Also, now that I've implemented Memcached on my VVV site, is it safe to push the database to my live site? I've already added wp-content/object-cache.php to .gitignore, as well as the new file it creates-db.php. So those files will not get pushed to the live server. But what about the database? Is it safe to replace my live DB with my VVV DB?

[tomjn]

It looks like it's operating fine judging from what QM is saying. There's also http://vvv.test/memcached-admin/

[cagross]

OK. But separately, can you confirm that at this point it's OK for me to push my local VVV database to my live site? I've been doing so very regularly while building this site. But now that I've implemented Memcached on my VVV site, I want to be sure pushing the DB to my live site will not result in any issues.

Does anything get written to the database after Memcached implemenation? If not, then I should be fine.

I can't make any progress on building my site until I can confirm this :-/

[tomjn]

Object caches don't use the database, they're just caches. The same cache is already in use in WordPress using PHP variables on most installs, it's just that the values are all forgotten at the end of the request. The only change you should see is improved performance, except for code that uses wp_cache_get and wp_cache_set incorrectly, which is rare

[tomjn]

As an aside, if anything was written to the database, QueryMonitor would tell you

[cagross]

OK thanks for confirming all that.

There's also http://vvv.test/memcached-admin/

OK good to know.

So we've checked Memcached. I guess the last thing on our list is for me to test my file system--correct? And I was waiting on you to recommend one or two. No rush--just confirming that's the next step.

[tomjn]

That's correct, I'm on the tail end of fighting off a cold. Once I've done some more internal bug fixing I can focus on the digital variety

[cagross]

Sure, take your time.

[cagross]

FYI earlier in this ticket I mentioned that, when I pushed my DB from local to live, it was noticeably slower. in the ~4-5 minute range when it is usually in the 1-2 minute range. I made a change on my end and it seems to have resolved that issue. Specifically, when I setup my theme, it came with a lot of demo content. Much of that content I didn't need, e.g. anything related to WooCommerce. I noticed that many database tables were WooCommerce related. So I dropped those tables. Now the DB migration is ~1:15 seconds. So we can remove that as a possible symptom of this particular issue.

[cagross]

@tomjn any update on the file system check?

[tomjn]

At the moment no, I've been busy

[tomjn]

I've tried a few times to look for an appropriate filesystem benchmark but:

• Nobody seems to have settled on a single tool
• None of the tools are straight forward and come with a bewildering array of options that have major impacts on the results
• Cross platform support is needed else how are we meant to compare results?
• Some of them require reserving file space to prevent caching, but I don't have 150GB of spare disk space to throw at it, and I know that Vagrant based shares rely on micro caching, so such a test would represent a worst case scenario, and that any test would need to be ran multiple times as mounted folders and non-mounted folders have different performance metrics

Put simply the TLDR:

1. You installed WordFence and it made your site slower, but you noticed the effect was greater in VVV than on an unknown remote system in a data center
2. Your database backups run faster when the database is smaller
3. Remote requests are awful for performance and can have wildly different performance characteristics even on the same system based on variables such as time of day, current internet bandwidth usage in your city, etc

I really don't know what I can suggest at this point, other than that we have yet to actually identify anything concrete that's slow, and I really don't see the value in making WordFence run fast or even running it in VVV, and fundamentally disagree with how the Photonic plugin works. There are plenty of major performance improvements that can be made here using pure WP expertise without making the local dev tech faster.

There are many other advantages to local development beyond speed, such as not having to edit files remotely, better debugging, or the whole not developing in production thing, but so far we've yet to identify anything that's responsible, so other than generic platitudes I'm at a loss as to what to suggest

The root cause may just be that you didn't spend as much money on SSD's and CPU's as your hosting provider. Sure you can make it run just as fast for low loads, but a child can pick up a piece of paper just as quickly as My Olympia, it's not the same as a 100kg weight. For all we know it is indeed filesystem performance, but if that's the case, there really isn't much I can suggest either

[cagross]

There are many other advantages to local development beyond speed, such as not having to edit files remotely, better debugging, or the whole not developing in production thing, but so far we've yet to identify anything that's responsible, so other than generic platitudes I'm at a loss as to what to suggest

In my case, there is only one advantage to developing locally--it is typically much faster to do so than develop remotely. Installing either of two fairly widely used plugins has completely eliminated that advantage. In fact, it makes developing locally much slower than developing remotely.

This isn't a complaint, or a gripe, or shot at you. I'm hopefully offering some constructive feedback. Until now, developing with VVV has been a godsend for me--thank you. It significantly increased the speed in which I can build a WordPress site. But now, my reality is:

I can use VVV to very quickly build a site locally. Unless I want to install the Wordfence plugin, or the Photonic plugin. If I want to install either of those, I have to completely forego VVV and develop remotely. The causes of the performance issues are irrelevant--my reality is that I can't use VVV if I use either of these plugins. Period.

If this issue is isolated to me and me only, then that bodes well for the future of VVV. But on the chance this affects many other users, and additionally, if these symptoms occur with many other widely used WordPress plugins, then it seems to me that VVV usage could drop considerably. One way to determine that is to of course monitor your GitHub issues for similar issues. I gather you haven't had too many, which is a good sign.

As a result, one thing added to my to-do list is to test out VVV alternatives (Docker, WAMP). I think earlier in this ticket you mentioned something about a Docker setup perhaps suffering from the same issues. I will hopefully have a chance to test that on my machine. I can report back here if you'd like, in a completely objective manner.

[tomjn]

I'm closing this out, it's been 6 months, and there really isn't a concrete ask or answer here

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.