Slow unit testing when preformed from inside VVV?

[dingo-d]

I was doing some unit/integration testing on a plugin, and since I didn't install Redis locally I ssh'd to vagrant and ran them there. The test took 43.61 seconds. I had Xdebug on and created code coverage as well.

That was for 5 relatively easy tests.

Then I thought to myself: can't I install Redis on my mac? Sure I can! And Xdebug as well, and I can generate code coverage there. So I installed it on my mac and ran them and it took 5.98 seconds for the same tests.

What gives? How come it takes so much to run unit tests inside vagrant than outside of it?

I know that Xdebug makes unit testing slow, but I had it enabled in both cases.

Your Environment

• VVV version: 2.1.0
• VVV Git Branch: develop
• Vagrant version: 2.0.2
• VM Provider name: VirtualBox
• VM Provider version: 5.2.6
• Operating System and version: macOS High Sierra 10.13.4

[tomjn]

Can you test this on 2.2-develop and vbox 5.2.8? VVV doesn't come with a Redis install, and I'm not familiar with Redis performance characteristics, perhaps you can be more specific as to the kind of performance issue you're facing? E.g. Networking vs disk vs cpu etc. It's also not clear what your PHPUnit tests are doing either

[dingo-d]

I updated my virtualBox, and VVV to 2.2.0-beta and I added one test, and now inside vagrant I get 1 minute testing time, and on my mac I get 7.1 seconds.

But I tried to disable Xdebug and the tests took 2.84 seconds inside vagrant O.o

So I guess Xdebug is the cause.

I knew it slows down unit tests, but didn't know that the effect was this big. Granted, the code coverage won't be generated, but I think I can live with that.

[Mte90]

I confirm this issue that are very slow and after a lot of debug seems something that regard the database, especially on installing wordpress.

[Mte90]

A simple test is execute phpunit of the wordpress-develop site and see that stuck on install wordpress and when is done the rest is very very fast.

[Mte90]

The problem also happen inside the machine itself. I switched to a local db in my host machine that use the webserver (codeception) on VVV and now the tests are using less of 1 seconds. I am curiosing to test that with the wordpress-dev tests.

[Mte90]

Infact also the wordpress-develop unit tests now are very fast.

[tomjn]

Note that wordpress-develop is being deprecated in favour of https://github.com/Varying-Vagrant-Vagrants/custom-site-template-develop

[tomjn]

Otherwise yes, with XDebug enabled, PHPUnit tests take significantly longer, which is to be expected regardless of the environment

[Mte90]

Yes I know but I am not using xDebug :-)

[tomjn]

I'm not entirely sure about things now, can we get a clearer view of things with timings? A nice table showing timings or some bullet points would be great, having it spread across multiple messages with relative units from multiple people with incomplete coverage is confusing

In general though, it will always be slower inside the VM than natively, and it will be considerably slower with XDebug regardless of the environment. I have no metrics for this though, so being told it's slower without any context isn't particularly helpful

[Mte90]

Let me know what I can do for that. Maybe execute the same unit test inside and outside of wordpress that are standard?

[tomjn]

Basically fill in the gaps here:

Test Performance	With XDebug	Without XDebug
VVV VBox	?s	?s
MacOS native	?s	?s
Windows native	?s	?s

Assuming PHP 7.2, and the same PHPUnit version. Bonus points for running multiple times and averaging. Additional points for adding results for the same tests on a different vagrant environment

[Mte90]

With test performance we are talking about to run the test inside the machine? I am actually on linux and I was running unit test using the db of VVV but phpunit was executed on my machine without xdebug. I saw that the majority of problems are based on mysql in any case I can run the test inside VVV with both the cases just as reference :-)

[tomjn]

With test performance we are talking about to run the test inside the machine?

Unknown, I'm still trying to figure out exactly which issue is being talked about, I suspect there are multiple being conflated with eachother.

I saw that the majority of problems are based on mysql

hmmm can you isolate this further to general MySQL performance? It's difficult to tell as everything said outside of the initial screenshot @dingo-d posted is all relative terms and non-quantifiable. As a result this issue is a bit muddled and confused. Or are there actual problems, bugs that are cropping up that prevent tests from running?

[dingo-d]

If you have a test example case I can test on macOS (in and out of VVV), and I could try to set it up on my Windows machine (which might take a while since I haven't used that in a while...)

[tomjn]

If you have a test example case I can test on macOS

I've been assuming this issue was about the WP unit tests, right now I'm looking to you to identify such a test case. I do not have a firm grasp on what it is this issue is about, and I'm mostly trying to get hard quantifiable information that I can do something with, and sweep away relative terms.

As I understand it, an issue has been identified, but has only been communicated across using subjective terms, so as long as you pick a test and state clearly what it is before running it across multiple platforms to demonstrate the issue, then all is good. I want to be able to see the impact xdebug vs no xdebug has, as well as VM vs no VM. This way I can ascertain how big a performance difference each contributes.

As for MySQL, I'm unsure how to factor that into results, but since the only unit tests VVV comes with are those used by WP core, lets stick with those. If you'd prefer to create small readable tests to compare with, that works too

[Mte90]

mte90:/var/www/VVV/www/wordpress-develop/public_html  ⑂ master +  $  phpunit tests/phpunit/tests/widgets.php 
Installing...
Running as single site... To run multisite, use -c tests/phpunit/multisite.xml
Not running ajax tests. To execute these, use --group ajax.
Not running ms-files tests. To execute these, use --group ms-files.
Not running external-http tests. To execute these, use --group external-http.
PHPUnit 6.5.8 by Sebastian Bergmann and contributors.

...............................................                   47 / 47 (100%)

You should really fix these slow tests (>150ms)...
 1. 619ms to run Tests_Widgets:test_wp_widget_is_preview
 2. 375ms to run Tests_Widgets:test_retrieve_widgets_move_orphaned_widgets_to_inactive
 3. 284ms to run Tests_Widgets:test_wp_widget_save_settings
 4. 209ms to run Tests_Widgets:test_register_and_unregister_widget_instance

Time: 43.1 seconds, Memory: 24.00MB

OK (47 tests, 214 assertions)
mte90:/var/www/VVV/www/wordpress-develop/public_html  ⑂ master +  $  phpunit tests/phpunit/tests/widgets.php 
Installing...
Running as single site... To run multisite, use -c tests/phpunit/multisite.xml
Not running ajax tests. To execute these, use --group ajax.
Not running ms-files tests. To execute these, use --group ms-files.
Not running external-http tests. To execute these, use --group external-http.
PHPUnit 6.5.8 by Sebastian Bergmann and contributors.

...............................................                   47 / 47 (100%)

Time: 1.49 seconds, Memory: 24.00MB

OK (47 tests, 214 assertions)

The first test use the db of VVV, the other use a db locally on my host machine. This is on linux, as I can see the long time is required on installing wordpress. I executed phpunit on my host machine and there isn't xDebug, so I don't think that the problem is xDebug atleast on my case (also because was turned off).

[tomjn]

hmm, assuming both are with xdebug turned off and the same PHP version, it looks like we need to do a basic timing test of a mysql connection, lets say SHOW TABLES or equivalent, for comparison.

That way we can eliminate PHPUnit and open a new issue on MySQL performance. What happens in you use the VVV DB from inside VVV? Does it give similar results or better?

[Mte90]

This is the output from inside the machine for the same tests with xdebug turned off:

vagrant@vvv:~$ cd /srv/www/wordpress-develop/
vagrant@vvv:/srv/www/wordpress-develop$ cd public_html/
vagrant@vvv:/srv/www/wordpress-develop/public_html$ phpunit tests/phpunit/tests/widgets.php 
Installing...
Running as single site... To run multisite, use -c tests/phpunit/multisite.xml
Not running ajax tests. To execute these, use --group ajax.
Not running ms-files tests. To execute these, use --group ms-files.
Not running external-http tests. To execute these, use --group external-http.
PHPUnit 6.5.8 by Sebastian Bergmann and contributors.

...............................................                   47 / 47 (100%)

You should really fix these slow tests (>150ms)...
 1. 773ms to run Tests_Widgets:test_wp_widget_is_preview
 2. 533ms to run Tests_Widgets:test_retreive_widgets_with_single_widget

Time: 38.32 seconds, Memory: 26.00MB

OK (47 tests, 214 assertions)

[Mte90]

There are like 5 seconds of difference but is can be acceptable because they are working on a network, the problem that is also inside there is a problem. If you let me know what kind of test to do inside the machine I will do ASAP.

[tomjn]

I wonder if it's a small difference that gets compounded? I'm away from my.comouter now but I think the approach to take here is general SQL performance tests.

As an aside, VVV switches from.a shared folder for DB storage to storing it inside the VM, but kept the old behaviour for existing users. Can you check if this is the case? If so the performance hit is likely due to traversing shared folders and network overhead On Sat, 21 Apr 2018 at 18:14, Daniele Scasciafratte < notifications@github.com> wrote:

There are like 5 seconds of difference but is can be acceptable because they are working on a network, the problem that is also inside there is a problem. IF you let me know what kind of test to do inside the machine I will do ASAP.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1464#issuecomment-383313627, or mute the thread https://github.com/notifications/unsubscribe-auth/AADl50IFcSbbefIMuFQLOW2UR2D86o29ks5tq2jQgaJpZM4TSUr5 .

[Mte90]

I executed a benchmark inside VVV:

vagrant@vvv:/srv/www/wordpress-develop/public_html$ sysbench prepare --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=wordpress_unit_tests --mysql-user=root --mysql-password=root --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...
vagrant@vvv:/srv/www/wordpress-develop/public_html$ sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=wordpress_unit_tests --mysql-user=root --mysql-password=root --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run        
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            489958
        write:                           0
        other:                           69994
        total:                           559952
    transactions:                        34997  (583.19 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 489958 (8164.67 per sec.)
    other operations:                    69994  (1166.38 per sec.)

Test execution summary:
    total time:                          60.0095s
    total number of events:              34997
    total time taken by event execution: 479.9454
    per-request statistics:
         min:                                  0.44ms
         avg:                                 13.71ms
         max:                                571.19ms
         approx.  95 percentile:              23.85ms

Threads fairness:
    events (avg/stddev):           4374.6250/66.59
    execution time (avg/stddev):   59.9932/0.00

I don't know how to read this benchmark but seems that there are problems with timeout and for something can be very slow.

[Mte90]

As an aside, VVV switches from.a shared folder for DB storage to storing it inside the VM, but kept the old behaviour for existing users. Can you check if this is the case? If so the performance hit is likely due to traversing shared folders and network overhead

Do you have suggestions for this point? I am not sure about what to do.

[tomjn]

hmm I don't see any timeouts in that output, it doesn't indicate anything terrible to me.

Here is the relevant database mapping code I mentioned:

https://github.com/Varying-Vagrant-Vagrants/VVV/blob/develop/Vagrantfile#L306-L325

If you installed VVV 2 fresh, then this won't be active.

Eitherway it looks like there is still some digging to do to determine where the slowdown is. Right now all we know is that when connecting from the host to the MariaDB instance inside the VM, it's slower than connecting to a native MySQL instance by a sizeable margin, but we don't know where the slowdown is, if it's the network connection, if it's latency, if it's the speed of the instance, or if it's dependent on what the instance is doing, or what the tests are doing.

[tomjn]

Have you tried running your test suite against the VMs DB from inside the VM? Rather than on the host? What kind of results does that give?

[Mte90]

I executed the test from inside the VM with the db of the VM https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1464#issuecomment-383313566

I can try to execute VVV without that code of vagrantfile and see what happen

[tomjn]

So this is our current results:

Test Performance	With XDebug	Without XDebug
Inside VVV MariaDB	?s	38.32s
Linux Host to VVV MariaDB	?s	43.1s
Linux Host Native MySQL	?s	1.49s

[tomjn]

I can try to execute VVV without that code of vagrantfile and see what happen

Unless the condition is true, and you do indeed have the mysql_upgrade_info present, removing those lines will have no effect.

I'm unsure what we can do from here, without doing a deep dive into MariaDB performance, which I haven't the time to

[Mte90]

Yes the results are right and for MariaDB maybe we can ask to the community if someone has more experience.

[tomjn]

I had a thought, by default only a single cpu core is given to the VM, that could be part of the problem, changing this in vvv-custom.yml may help:

vm_config:
  cores: 4
  memory: 2048

[dingo-d]

Nope, tests are still slow for me

[Mte90]

I use 2 cores since years....

[tomjn]

Can you check what the CPU availability or usage max is via virtualbox? I couldn't find what the defaults are googling On Tue, 15 May 2018 at 10:41, Daniele Scasciafratte < notifications@github.com> wrote:

I use 2 cores since years....

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1464#issuecomment-389107405, or mute the thread https://github.com/notifications/unsubscribe-auth/AADl5y4_Iq99bUkoxPIiK22w5gZDolPxks5tyqLIgaJpZM4TSUr5 .

[Mte90]

usually is based on the system and I think that the default to avoid problem is 1

[tomjn]

what's yours set to though? If it's already at 1 then it's not a useful avenue to pursue, else it's worth looking into as setting via modifyvm

[Mte90]

vm_config:
    memory: 4096
    cores: 2

[tomjn]

Closing this out as it's old, and a lot has changed. Ubuntu 18 has better virtualization support than 14, vagrant has gone through some big changes, and VirtualBox 6 was released. We also made changes to how mapped folders worked to avoid nesting

[dingo-d]

Just to update a bit, since I've discovered pcov extension and pcov-clobber for PHPUnit7, the test speed increase was drastic.

Avoid using xdebug for code coverage generation and use it for debugging and it should be good 👍

[tomjn]

@dingo-d is there any danger in having those just installed and enabled all the time? Or would pcov_on pcov_off helpers be needed?

[dingo-d]

I've been running it like this for some time now and I don't see any drawbacks or slowdowns. If I'm not mistaken, PHPUnit8 has pcov set as the default code coverage driver, so it should be ok to use.

I've created a utility for it https://github.com/dingo-d/pcov-vvv-utility, but I need to test it (I've set it up without a utility because I needed it on the replacement laptop but it should be ok)

GitHub

dingo-d/pcov-vvv-utility

pcov utility for the VVV. Contribute to dingo-d/pcov-vvv-utility development by creating an account on GitHub.

[Mte90]

I think that we can open a new ticket to pcov integration in VVV, the code is quite simple so we can add as utility and if works we can merge it https://github.com/dingo-d/pcov-vvv-utility/blob/develop/pcov/provision.sh

GitHub

dingo-d/pcov-vvv-utility

pcov utility for the VVV. Contribute to dingo-d/pcov-vvv-utility development by creating an account on GitHub.

[tomjn]

the main issue with it at the moment is that it'll only install to the current PHP version, this is the kind of extension I'd be curious about having in VVV core

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.