run test by file, thread limit, thread number

[razielsd]

I have small changes :) 1 .Run test by file (now by dir)

2. Thread limit
3. Thread number for easy use selenium servers pool
4. Can use custom phpunit runner
5. Test file mask

When you wanna merge, i'm send pool request.

https://github.com/razielsd/parallel-phpunit-run-by-file

[siivonen]

Looking good! Go ahead and do the pull request. This is a good improvement and not taking any of the old advantages away.

Few comments:

• Could you split the one big commit into several smaller ones? (easier to review and comment, leaves more informative history, ...)
• Could you also update the README page since after these changes it will be a bit out of date?

[razielsd]

1. split commit - can't split, all code refactored
2. update readme - at next week

[razielsd]

Changed readme, but i bad speak english, translate is so bad :( https://github.com/razielsd/parallel-phpunit-run-by-file

[siivonen]

Something seems to be broken:

$ parallel-phpunit example Running parallel wrapper for phpunit Starting command: phpunit example/1/ATest.php thread_id 0 Starting command: phpunit example/2/3/CTest.php thread_id 1 Starting command: phpunit example/2/4/DTest.php thread_id 2 Starting command: phpunit example/2/BTest.php thread_id -1 /Users/samulisiivonen/parallel-phpunit/bin/parallel-phpunit: line 190: threads[$thread_id]: bad array subscript Command ended: phpunit example/2/3/CTest.php PHPUnit 3.7.10 by Sebastian Bergmann.

.

Time: 1 second, Memory: 5.75Mb

OK (1 test, 1 assertion) Command ended: phpunit example/2/4/DTest.php PHPUnit 3.7.10 by Sebastian Bergmann.

.

Time: 1 second, Memory: 5.75Mb

OK (1 test, 1 assertion) Command ended: phpunit example/2/BTest.php PHPUnit 3.7.10 by Sebastian Bergmann.

.

Time: 1 second, Memory: 5.75Mb

OK (1 test, 1 assertion) Command ended: phpunit example/1/ATest.php PHPUnit 3.7.10 by Sebastian Bergmann.

.

Time: 2 seconds, Memory: 5.75Mb

OK (1 test, 1 assertion) Success: 4 Fail: 0 Error: 0 Skip: 0 Incomplete: 0

[siivonen]

Also --pu-thread has no effect.

L316:parallel-phpunit samulisiivonen$ time parallel-phpunit --pu-thread 1 example > /dev/null expr: syntax error /Users/samulisiivonen/parallel-phpunit/bin/parallel-phpunit: line 25: $pos-1: substring expression < 0 /Users/samulisiivonen/parallel-phpunit/bin/parallel-phpunit: line 186: threads[$thread_id]: bad array subscript

real 0m2.427s user 0m0.506s sys 0m0.297s L316:parallel-phpunit samulisiivonen$ time parallel-phpunit --pu-thread 4 example > /dev/null expr: syntax error /Users/samulisiivonen/parallel-phpunit/bin/parallel-phpunit: line 25: $pos-1: substring expression < 0 /Users/samulisiivonen/parallel-phpunit/bin/parallel-phpunit: line 186: threads[$thread_id]: bad array subscript

real 0m2.420s user 0m0.510s sys 0m0.292s

[razielsd]

/Users/samulisiivonen/parallel-phpunit/bin/parallel-phpunit: line 190: threads[$thread_id]: bad array subscript Command ended: phpunit example/2/3/CTest.php

Can't reproduce this bug :(

Also --pu-thread has no effect.

use --pu-thread=1

https://github.com/razielsd/parallel-phpunit-run-by-file NOTE: parallel-phpunit using unlike phpunit parameter format !!!

may be in future write like phpunit parameters, now no time.

[razielsd]

not installed pgrep ?

[razielsd]

try

bash -x bin/parallel-phpunit example

and paste output

[siivonen]

I fixed all the problems that I had and now everything seems to be working. I created a release 1.1 that contains all your changes and my fixes/improvements on top of that. I have done following modifications:

• Rename --pu-thread to --pu-threads
• Use --test-suffix instead of --pu-test-mask (dublicates)
• Remove --pu-verbose switch (not adding any value)
• Remove config file restriction
• Allow --pu- switches with = or space(s)
• Delete the example/Web since it's duplicate to example

I didn't understand why XML config wouldn't work. Can you explain?

And could you test that everything with release 1.1 works for you also?

[razielsd]

I didn't understand why XML config wouldn't work. Can you explain?

When use file filtering in xml you can't paralleling, need parse xml config, example: http://www.phpunit.de/manual/3.2/en/appendixes.configuration.html#appendixes.configuration.blacklist-whitelist

P.S. I'm not remove phpunit.xml from production testing server and have trouble surprise :))

At next week run tests.

[siivonen]

These file excludes and includes are used in code coverage only i think. Code coverage and paralleling are anyway not a good match since code coverage can only produce a report from one php process. If you want code coverage you should produce it with one (big and slow) phpunit command.

[razielsd]

May be parse phpunit.xml for parameters directory/file and show warning ?

[siivonen]

phpunit.xml is already currently parsed. And there is this section in README:

Known limitations

• The reporting switches (like --coverage-* or --log-*) are not guaranteed to work since all parallel executions are writing to the same directories or files. Only --log-junit is ensured to work since it is handled as special case by parallel-phphunit.

I don't know if there is a need for warning in command line tool. I think the problem is not phpunit.xml but the reporting switches. In parallel case they create multible reports in same directory. For example 'parallel-phpunit --coverage-html /tmp/coverage .' will create coverage reports for all parallel phpunit executions to the same directory /tmp/coverage. You can do this and all the exclude and include roles in phpunit.xml will be used but the end result in /tmp/coverage are is not very useful.

[razielsd]

I don't know if there is a need for warning in command line tool. I think the problem is not phpunit.xml but the reporting switches

It's make be easy migration for using parallel-phpunit. I faced this issue personally and therefore added an error message if phpunit.xml configuration file is found.

[razielsd]

After update for tests need more time, debugging now.

Time log: //After update:

3249 10 minutes

3248 11 minutes

3247 11 minutes

3246 10 minutes

//Before update

3245 7 minutes 59 seconds

3244 8 minutes 9 seconds

3243 8 minutes 0 seconds

3242 7 minutes 54 seconds

3241 8 minutes 21 seconds

3240 7 minutes 51 seconds

3239 7 minutes 52 seconds

3238 7 minutes 41 seconds

3237 7 minutes 55 seconds

[razielsd]

Lost changes from function test_files:

function test_files {
    if [[ $ARGS == *--group* ]];then
param=$ARGS
        param=${param#*--group}
        param=${param%--*}
        param=${param//,/\\|}
        grep -F -R -s --include=$CFG_TEST_MASK '@group' $ROOT | awk '{ print $1 " " $4}' | grep "$param" | awk '{print $1}' | sed 's/://' | sort | uniq
    else
find $ROOT -name \*$CFG_TEST_MASK
    fi
}

[razielsd]

Need for select only reaquired files when using parameter --group.

[razielsd]

• When using --group - run only files with selected groups
• print_summary show only at end.

https://github.com/razielsd/parallel-phpunit-run-by-file/commit/8dab8b396c9db30e1c510869ef883ec9bdd04106

[siivonen]

Groups can be set in XML file also and this "if --group then something" -solution will only work with command line arguments. I think it is OK to just pass the --group* arguments to the parallel phpunit command lines. The end result will be correct but there can be these 'No tests executed!' executions.

I just pushed few changes to master:

1. Warnings about --coverage-, --log- and --testdox-* switches
2. Lowered the latency (now every 'No tests executed!' only adds 0.1 seconds to the total test execution)
3. Added some groups to example tests

I think now the --group thing shouldn't slow you down:

l316:parallel-phpunit samulisiivonen$ time phpunit --group b example/ > /dev/null

real 0m1.112s user 0m0.089s sys 0m0.019s l316:parallel-phpunit samulisiivonen$ time parallel-phpunit --group b example/ > /dev/null

real 0m1.695s user 0m0.694s sys 0m0.471s

These changes are in master so could you check that they work for you? If they work I can merge them to release 1.1.

[razielsd]

These changes are in master so could you check that they work for you? If they work I can merge them to release 1.1.

All my commits tested and checked. (my changes from internal git branch)

[siivonen]

I think you misunderstood me. I meant to say that you should test the master branch of original parallel-phpunit. Test if that works for you now. I don't see the reason for handling --group switches as a separate case so I wouldn't want to merge your latest changes. I think group parameters should be passed directly to phpunit commands. They can produce some extra phpunit executions that say 'No tests executed!', but these shouldn't slow things down.

[razielsd]

They can produce some extra phpunit executions that say 'No tests executed!', but these shouldn't slow things down.

This true, but PHPUnit have some bullshit :( provider always run, when you have many providers run all providers.

P.S. no time, when you need example - write tomorrow

[razielsd]

NOT STABLE https://github.com/razielsd/parallel-phpunit-run-by-file/commit/e6a4719ad7604de953813ad6067baf759f733eb2

• Add thread number for test parameter, can be use in tests, example: choose free selenium host
• Valid count active tests, when use shell script for run phpunit, you cannot use count child process

[razielsd]

Changes tested, bugs not found, you can merge changes.

[siivonen]

Few things before the merge:

1. What server are you using to run your tests? I can't reproduce the "when use shell script for run phpunit, you cannot use count child process" error in Mac, Redhat or Ubuntu.
2. How do you use 'parallel-phpunit-thread' value? I can't use it in PHP code. $ php -d parallel-phpunit-thread=1 -r "var_dump(ini_get('parallel-phpunit-thread'));" bool(false)
3. The 'print_summary' can't be last line of the script since it returns the wrong exit value.

[razielsd]

1. Need to more shell scripts, it's to hard for reproduce, i'm lost many time for fix it :( Simple scheme: .sh = shell script .sh -> .sh -> parallel_phpunit -> .sh -> phpunit

razielsd@razielsd-vm:~/projects/avito/avito$ phpunit --help  | grep php.ini
  -d key[=value]            Sets a php.ini value.

var_dump($_SERVER['argv']);

1. i'm not use print_summary, may be with bug :( Move back after wait last thread ?

[siivonen]

OK, I merged your commit to master branch of https://github.com/siivonen/parallel-phpunit. Could you now test if that works in your environment?

[razielsd]

It's work bad, you remove --group filter.

1. When not found test for run in file, PHPUnit make failure exit code.
2. PHPUnit always run providers. When you have providers, lost time per running. Example:

<?php
class DataTest extends PHPUnit_Framework_TestCase
{
    /**
     * @dataProvider provider
     * @group somegroup
     */
    public function testAdd($a, $b, $c)
    {
        $this->assertEquals($c, $a + $b);
    }

    public function provider()
    {
        echo "Surpise:\nPeace of shit here\n";
        return array(
          array(0, 0, 0),
          array(0, 1, 1),
          array(1, 0, 1),
          array(1, 1, 3)
        );
    }
}

run:

razielsd@razielsd-vm:~/test$ phpunit --group othergroup DataTest.php
Surpise:
Peace of shit here
PHPUnit 3.7.7 by Sebastian Bergmann.

Time: 0 seconds, Memory: 4.00Mb

No tests executed!

[siivonen]

This is good to know and very stupid feature in phpunit!

How can running data provider function of another test break some other test the tests are running in totally separate PHP-processes. Your data providers are somehow modifying some shared resource (like database, filesystem, server configuration, ...) that other tests are relying on?

This problem seems to be more in your tests than in missing parallel-phpunit feature.

I never added the --group feature since I thought it wouldn't be needed. I though that it is only needed in your test setup but not in general. Is there a way to fix this problem by modifying your data providers?

[razielsd]

Really problem:

1. When not found test for run in file, PHPUnit make failure exit code.

[siivonen]

Again I can't reproduce this error. Can you reproduce this with the example tests? Maybe some of your test executions resulted in an fatal erro? Non-zero exit status means that this command returns something:

find $ROOT -name parallel-phpunit.${PID}.*.fail

What does it return?

[razielsd]

Example of exit code bug:

razielsd@razielsd-vm:~/projects/fork/test1/pu/bin$ cat test.sh
#!/bin/sh
exit 1

Run test:

razielsd@razielsd-vm:~/projects/fork/test1/pu/bin$ ./parallel-phpunit --pu-cmd ./test.sh ../example/1/
Running parallel-phpunit 1.1
Starting command with thread 0: ./test.sh -d parallel-phpunit-thread=0  ../example/1/ATest.php
Command ended: ./test.sh -d parallel-phpunit-thread=0  ../example/1/ATest.php
Success: 0 Fail: 0 Error: 0 Skip: 0 Incomplete: 0

http://en.wikipedia.org/wiki/Exit_status

[siivonen]

In your example test.sh returns non-zero exit status so the whole parallel-phpunit execution should fail. I don't see any problem in this behavior. Phpunit returns 0 in 'No tests executed!' so there shouldn't be any problem. Maybe your phpunit script returns non-zero exit status when it shouldn't?

Works:

$ parallel-phpunit --filter foo example/1/ ; echo "Exit status: $?"
Running parallel-phpunit 1.1
Starting command with thread 0: phpunit -d parallel-phpunit-thread=0  --filter foo example/1/ATest.php
Command ended: phpunit -d parallel-phpunit-thread=0  --filter foo example/1/ATest.php
PHPUnit 3.7.10 by Sebastian Bergmann.

Time: 0 seconds, Memory: 5.50Mb

No tests executed!
Success: 0 Fail: 0 Error: 0 Skip: 0 Incomplete: 0
Exit status: 0

Fails:

$ parallel-phpunit --filter foo --pu-cmd ./test.sh example/1/ ;echo "Exit status: $?"
Running parallel-phpunit 1.1
Starting command with thread 0: ./test.sh -d parallel-phpunit-thread=0  --filter foo example/1/ATest.php
Command ended: ./test.sh -d parallel-phpunit-thread=0  --filter foo example/1/ATest.php
Success: 0 Fail: 0 Error: 0 Skip: 0 Incomplete: 0
Exit status: 1

But when I change 'exit 1' to 'phpunit $@' in test.sh:

$ parallel-phpunit --filter foo --pu-cmd ./test.sh example/1/ ;echo "Exit status: $?"
Running parallel-phpunit 1.1
Starting command with thread 0: ./test.sh -d parallel-phpunit-thread=0  --filter foo example/1/ATest.php
Command ended: ./test.sh -d parallel-phpunit-thread=0  --filter foo example/1/ATest.php
PHPUnit 3.7.10 by Sebastian Bergmann.

Time: 0 seconds, Memory: 5.50Mb

No tests executed!
Success: 0 Fail: 0 Error: 0 Skip: 0 Incomplete: 0
Exit status: 0

[razielsd]

You can doing as you wish.