Slower to run than native PHPUnit command

[titolendable]

I came across fastest today and sounded promising, exactly what our team were looking for. I gave it a go but for my surprise, it takes twice the time to run the unit tests. The project is powered by Symfony2 and using LiipFunctionalTestBundle. I executed the following command to run the tests:

find src/Alpha/ -name "*Test.php" | php bin/fastest -o "php bin/phpunit -c app {};"

And takes over 4 minutes to run all the 235 tests. The phpunit command takes almost 2 minutes to run the same tests. Fastest is using two processes and I get two sqlite files for the database, so it seems to be set up properly. How can I debug why it is so slow? What might causing it? Both tests run under on the same machine with the same environment, with xdebug on.

[liuggio]

@titolendable thanks a lot for reporting this, all the feedbacks we had was really good... When/How the fixtures are loaded? How do you use the liipFunctionalBundle?

[titolendable]

Thanks @liuggio for getting back to me so quickly. Running a simple example on a folder with basic tests that extend the PHPUnit_Framework_TestCase (no usage of the LiipFunctionalTestBundle or access to the database), fastests takes way longer to run. I am scratching my head how can it be possible, eventually something on my environment, not sure, hopefully you will be able to figure out.

PHPUnit:

$php bin/phpunit -c app/ src/Alpha/SiteBundle/Tests/FraudCheck/

PHPUnit 4.8.16 by Sebastian Bergmann and contributors.

...............................................................  63 / 284 ( 22%)
............................................................... 126 / 284 ( 44%)
............................................................... 189 / 284 ( 66%)
............................................................... 252 / 284 ( 88%)
................................

Time: 1.04 seconds, Memory: 22.00Mb

OK (284 tests, 685 assertions)

Fastest:

$ls src/Alpha/SiteBundle/Tests/FraudCheck/*Test.php | php bin/fastest -o "php 

bin/phpunit -c app {};"
2
- 26 tests into the queue.
- Will be consumed by 2 parallel Processes.

26/26 [============================] 100% 14 secs 4.5 MiB 

     0 failures                              

    ✔ You are great!
    Time: 13873 ms, Memory: 4718592 b

Details about my PHP version

$php -v
PHP 5.6.14 (cli) (built: Oct  3 2015 14:46:59) 
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies
    with Xdebug v2.3.3, Copyright (c) 2002-2015, by Derick Rethans

And info about fastest:

$composer show -i | grep fastest

liuggio/fastest                                 1.1                   Simple parallel testing execution... with some goodies for functional tests.

[liuggio]

@titolendable
Your suite tests is already small and fast (1 second), I think is difficult divide by 1 seconds :) fastest creates parallel processes so there's overhead, how many cores do you have on your machine why do you limit by 2? How is the result with no argument 2?

[titolendable]

@liuggio That was an example, I just picked one folder, the entire suite of tests takes almost 2 minutes. I have only two cores on my computer, the script got it right.

[liuggio]

@titolendable if you increase the verbosity, can you see what's the slow thing?

[titolendable]

@liuggio this is the output I get when adding the -vvv option:

find src/Alpha/SiteBundle/Tests/FraudCheck/ -name "*Test.php" | php bin/fastest -vvv  -o "php bin/phpunit -c app --stop-on-failure {};"
- 26 tests into the queue.
- Will be consumed by 2 parallel Processes.
1   1/26    ✔ 1229 ms     src/Alpha/SiteBundle/Tests/FraudCheck/ATest.php
2   2/26    ✔ 1460 ms     src/Alpha/SiteBundle/Tests/FraudCheck/BTest.php
2   3/26    ✔ 1543 ms     src/Alpha/SiteBundle/Tests/FraudCheck/CTest.php
1   4/26    ✔ 2101 ms     src/Alpha/SiteBundle/Tests/FraudCheck/DTest.php
1   5/26    ✔ 1277 ms     src/Alpha/SiteBundle/Tests/FraudCheck/ETest.php
2   6/26    ✔ 1689 ms     src/Alpha/SiteBundle/Tests/FraudCheck/FTest.php
2   7/26    ✔ 1357 ms     src/Alpha/SiteBundle/Tests/FraudCheck/GTest.php
1   8/26    ✔ 1703 ms     src/Alpha/SiteBundle/Tests/FraudCheck/HTest.php
2   9/26    ✔ 2111 ms     src/Alpha/SiteBundle/Tests/FraudCheck/ITest.php
1   10/26   ✔ 2365 ms     src/Alpha/SiteBundle/Tests/FraudCheck/LTest.php
2   11/26   ✔ 2416 ms     src/Alpha/SiteBundle/Tests/FraudCheck/MTest.php
1   12/26   ✔ 1970 ms     src/Alpha/SiteBundle/Tests/FraudCheck/NTest.php
2   13/26   ✔ 1872 ms     src/Alpha/SiteBundle/Tests/FraudCheck/OTest.php
1   14/26   ✔ 1933 ms     src/Alpha/SiteBundle/Tests/FraudCheck/PTest.php
1   15/26   ✔ 2053 ms     src/Alpha/SiteBundle/Tests/FraudCheck/QTest.php
2   16/26   ✔ 2707 ms     src/Alpha/SiteBundle/Tests/FraudCheck/RTest.php
1   17/26   ✔ 1804 ms     src/Alpha/SiteBundle/Tests/FraudCheck/STest.php
2   18/26   ✔ 1849 ms     src/Alpha/SiteBundle/Tests/FraudCheck/TTest.php
1   19/26   ✔ 1841 ms     src/Alpha/SiteBundle/Tests/FraudCheck/UTest.php
2   20/26   ✔ 1758 ms     src/Alpha/SiteBundle/Tests/FraudCheck/VTest.php
1   21/26   ✔ 2067 ms     src/Alpha/SiteBundle/Tests/FraudCheck/WTest.php
2   22/26   ✔ 1965 ms     src/Alpha/SiteBundle/Tests/FraudCheck/ZTest.php
1   23/26   ✔ 2207 ms     src/Alpha/SiteBundle/Tests/FraudCheck/XTest.php
2   24/26   ✔ 1998 ms     src/Alpha/SiteBundle/Tests/FraudCheck/JTest.php
1   25/26   ✔ 1959 ms     src/Alpha/SiteBundle/Tests/FraudCheck/BaseCheckTest.php
2   26/26   ✔ 1902 ms     src/Alpha/SiteBundle/Tests/FraudCheck/KTest.php

    ✔ You are great!
    Time: 24666 ms, Memory: 4456448 b

Each file takes almost 2 seconds to run, when with phpunit the overall time spend is around than 1s.

[perk11]

I think likely one of the two things happening:

• Your system takes a lot of time to start phpunit process for some reason.

Or

• You have some init code that takes ~1.5-2 seconds that runs each time when starting a test as a single file, but only once when you run a whole test suite.

[titolendable]

Sounds reasonable @perk11 but I can not figure out what can be. We include the default Symfony2 configuration for phpunit, nothing specific to our environment. And strange enough, the whole tests in the folder run around 1 second with native phpunit, from your suggestion, the minimum time would be ~1.5-2.

[perk11]

@titolendable try timing it using time, phpunit may report time wrong. Just add time before the command you use to run the suite.

[liuggio]

I agree with @perk11 maybe is the phpunit bootstrapping :) or the fastest overhead