No info on failing / risky tests

[jeanCarloMachado]

PHPunit now distincts between errors and failures. When we have failures the suite exits with status != 0 as expected but we dont get any clue on the problem.

 ./vendor/bin/paratest                              

Running phpunit in 5 processes with /home/jean/projects/compufacil/Backend/vendor/bin/phpunit

Configuration read from /home/jean/projects/compufacil/Backend/phpunit.xml.dist

.........................................S.............SS....   61 / 1534 (  3%)
.....S............SSS.........................S..............  122 / 1534 (  7%)
........................................SSSSSSSSS............  183 / 1534 ( 11%)
..........S..................................................  244 / 1534 ( 15%)
.............................................SSSSSS..........  305 / 1544 ( 19%)
.............................................................  366 / 1545 ( 23%)
.............................................................  427 / 1545 ( 27%)
...............SS............................................  488 / 1545 ( 31%)
.............................................SSS.........S...  549 / 1545 ( 35%)
.............................................................  610 / 1545 ( 39%)
.............................................................  671 / 1547 ( 43%)
.............................................................  732 / 1547 ( 47%)
..........................S......S...........................  793 / 1554 ( 51%)
.........S..............=.....................................  854 / 1556 ( 54%)
.............................................................  915 / 1556 ( 58%)
......SSS.........S..........SSSSSSSSSS......................  976 / 1556 ( 62%)
............................................................. 1037 / 1557 ( 66%)
......S...................................................... 1098 / 1562 ( 70%)
............................................................. 1159 / 1576 ( 73%)
............................................................. 1220 / 1613 ( 75%)
............................................................. 1281 / 1616 ( 79%)
............................................................. 1342 / 1621 ( 82%)
............................................................. 1403 / 1621 ( 86%)
.....S..............................SSSSS.................... 1464 / 1621 ( 90%)
...........................S.........SSSSSSSSS............... 1525 / 1621 ( 94%)
..............................................S.............. 1586 / 1623 ( 97%)
.................S...................

Time: 1.08 minutes, Memory: 48.00MB

FAILURES!
Tests: 1623, Assertions: 3900, Failures: 1, Errors: 0.

[michaelbutler]

What version of Paratest and PHPUnit are you running? I couldn't reproduce this one on the latest version of both. do you know what the specific failure was caused by? Was it simply a failed assertion or was it something else, like an E_ERROR raised.

[marden]

I've faced same issue. Paratest doesn't show warnings triggered by phpunit. How to reproduce:

1. Create a test and mock some class with non-existent method.

2. Run phpunit with default settings. It will show warning: Trying to configure method "XXX" which cannot be configured because it does not exist, has not been specified, is final, or is static

3. Run paratest and you will not see a warning.

[ghost]

Do anyone know some workaround around this? Its really hard to chase the warning down if you have really pesky test-set that you are not able to run in PhpUnit natively due to hug ememory requirements.

[michaelbutler]

I'm pretty sure this was fixed in later versions of paratest and phpunit. But if you're not able to upgrade (I highly recommend you do if you can), you can solve this by adding a simple test listener class and add it in phpunit.xml. on the test listener you can throw an exception on test warning, this way you will get a stack trace.

Test listeners can also be used to debug unknown failure issues, by putting a log statement for test start and test end... If there is any start without an end, you know that test crashed.

[ghost]

I am on version 1.1.0 of paratest and 6.5.7 phpunit. Do you suggest to move on 7?

[ghost]

Thanks a lot for that listener hint, trying it now :)

[michaelbutler]

1.1.0 and 6.5.7 should be fine

[ghost]

Here is a listener I used if anyone else got into same trouble as I did.

<?php

use PHPUnit\Framework\AssertionFailedError;
use PHPUnit\Framework\Test;
use PHPUnit\Framework\TestListener;
use PHPUnit\Framework\TestSuite;
use PHPUnit\Framework\Warning;

/**
 * @see https://github.com/paratestphp/paratest/issues/282
 */
class MakeWarningErrorAgainTestListener implements TestListener
{

    /**
     * @inheritdoc
     */
    public function addWarning(Test $test, Warning $e, $time)
    {
        throw new LogicException($e->getMessage(), $e->getCode(), $e);
    }

    // all other methods are just empty
}

and phpunit.xml just add

    <listeners>
        <listener class="MakeWarningErrorAgainTestListener"></listener>
    </listeners>

[willemstuursma]

I am still having this issue with the latest paratest and PHPUnit 7.3, but for warnings.

Time: 26.25 seconds, Memory: 90.00MB
FAILURES!
Tests: 14851, Assertions: 44497, Failures: 2, Errors: 0.```

PHPUnit gives the warnings:

Time: 2.81 minutes, Memory: 728.00MB

There were 2 warnings:

1) Warning
The data provider specified for zzzTest:xxx is invalid.
...
2) Warning
The data provider specified for zzzTest::yyy is invalid.
...

[filkaris]

^ Same issue

[chq81]

^ Same issue

[chq81]

^ Same issue here

[donotnoot]

I'm experiencing this but only when I specify a runner, also the number of tests change? Weird.

$ vendor/bin/phpunit
PHPUnit 7.5.1 by Sebastian Bergmann and contributors.

Error:         No code coverage driver is available

...............................................................  63 / 117 ( 53%)
.........................W............................          117 / 117 (100%)

Time: 15.42 seconds, Memory: 42.00MB

There was 1 warning:

1) Warning
No tests found in class "Tests\Feature\CRUD\SomeControllerTest".

WARNINGS!
Tests: 117, Assertions: 1076, Warnings: 1.

$ vendor/bin/paratest -p 8           

Running phpunit in 8 processes with /Users/user/www/project/vendor/phpunit/phpunit/phpunit

Configuration read from /Users/user/www/project/phpunit.xml

...........................................W...................  63 / 115 ( 54%)
.......................................................

Time: 10.76 seconds, Memory: 8.00MB

There was 1 warning:

1) No tests found in class "Tests\Feature\CRUD\SomeControllerTest".

FAILURES!
Tests: 117, Assertions: 1076, Failures: 1, Errors: 0.

$ vendor/bin/paratest -p 8 --runner SqliteRunner

Running phpunit in 8 processes with /Users/user/www/project/vendor/phpunit/phpunit/phpunit

Configuration read from /Users/user/www/project/phpunit.xml

...............................................................  63 / 114 ( 55%)
......................................................

Time: 7.92 seconds, Memory: 8.00MB

FAILURES!
Tests: 117, Assertions: 1076, Failures: 1, Errors: 0.

$ vendor/bin/paratest -p 8 --runner WrapperRunner

Running phpunit in 8 processes with /Users/user/www/project/vendor/phpunit/phpunit/phpunit

Configuration read from /Users/user/www/project/phpunit.xml

...............................................................  63 / 115 ( 54%)
......................................................

Time: 8.29 seconds, Memory: 8.00MB

FAILURES!
Tests: 117, Assertions: 1460, Failures: 1, Errors: 0.

PHPUnit 7.5.1, Paratest 2.1, Laravel 5.7 if that matters, OS X Mojave

[crzdeveloper]

As a temporary solution. you can try to use the Paratest Logger https://github.com/paslandau/paratest/blob/wrapper-runner-log/docs/logging.md

[willemstuursma]

We are willing to sponsor support for Warnings and Risky tests.

[andreasschroth]

@willemstuursma Thank you for your generous offer. I gave this issue the label "most-upvotes" now (which basically means "often requested for"). I'll take a look at this once I have some more spare time (but I don't accept any money :-) ).

[willemstuursma]

@andreasschroth we can send you some Mollie swag or stuff from your Amazon wishlist instead, please contact me at willem@mollie.com.

[willemstuursma]

We are also having problems with errors in the setUp() method. They are ignored by paratest and work with PHPUnit.

We are also willing to sponsor fixing this.

[rask]

Same issue here, not seeing what failed and why is a hindrance. Using WrapperRunner here, and I'm calling paratest on a Docker container application via SSH.

Example: made empty test method that does not run assertions. PHPUnit complains about a risky test, but Paratest says there was an error but does not tell me it was a risky test.

Will check if those listeners and loggers are an OK solution in the meantime.

Let me know if you need any help with this, I think I can get a bit of company time to assist here and there.

[crzdeveloper]

The PR #402 might fixed the problem with warnings. I'm not sure about the risky tests, IMHO, they must be implemented in the same way as I did with warnings

[oprypkhantc]

The workaround from above did not properly work for risky tests. We were getting weird framework-related object instantiation errors without any valid stacktrace.

I researched it a little bit and the problem seems to be in phpunit itself (partly), as it's JUnit logs are different for risky tests.

There's a way to rewrite it - basically by providing own test names in the output instead of relying on strings from JUnit, but both require a modification to PHPUnit.

https://github.com/sebastianbergmann/phpunit/issues/3937

[Slamdunk]

Fixed in master:

#~/repos/paratest (master %=)$ vendor/bin/phpunit test/fixtures/warning_tests/
PHPUnit 9.3.7 by Sebastian Bergmann and contributors.

W.WW                                                                4 / 4 (100%)

Time: 00:00.008, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

WARNINGS!
Tests: 4, Assertions: 1, Warnings: 3.

#~/repos/paratest (master %=)$ bin/paratest test/fixtures/warning_tests/

Running phpunit in 4 processes with /paratest/vendor/phpunit/phpunit/phpunit

Configuration read from /paratest/phpunit.xml.dist

W.WW

Time: 00:00.196, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.
OK (4 tests, 1 assertion)

#~/repos/paratest (master %=)$ bin/paratest --runner WrapperRunner test/fixtures/warning_tests/

Running phpunit in 4 processes with /paratest/vendor/phpunit/phpunit/phpunit

Configuration read from /paratest/phpunit.xml.dist

W.WW

Time: 00:00.216, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.
OK (4 tests, 1 assertion)

#~/repos/paratest (master %=)$ bin/paratest --runner SqliteRunner test/fixtures/warning_tests/

Running phpunit in 4 processes with /paratest/vendor/phpunit/phpunit/phpunit

Configuration read from /paratest/phpunit.xml.dist

W.WW

Time: 00:01.011, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.
OK (4 tests, 1 assertion)

[neclimdul]

Awesome! Its been a while since I've looked but I remember the logic was basically the same for warnings and risky so does that mean this works for risky tests too?

[Slamdunk]

No because of https://github.com/sebastianbergmann/phpunit/issues/3937

[Slamdunk]

Good news: https://github.com/sebastianbergmann/phpunit/pull/4438 has been merged, and together with https://github.com/paratestphp/paratest/pull/523 now risky tests are reported correctly in ParaTest too:

There were 2 risky tests:

1) ParaTest\Tests\fixtures\failing_tests\UnitTestWithMethodAnnotationsTest::testRisky
This test did not perform any assertions

./paratest/test/fixtures/failing_tests/UnitTestWithMethodAnnotationsTest.php:68

2) ParaTest\Tests\fixtures\failing_tests\UnitTestWithErrorTest::testRisky
This test did not perform any assertions

./paratest/test/fixtures/failing_tests/UnitTestWithMethodAnnotationsTest.php:68

[cyberhicham]

I still have risky tests (mostly no assertion tests) reported as errors PHPUnit 9.4.2 && ParaTest 5.0.4

[Slamdunk]

That's correct, because until the end they cannot be distinguished by errors due to a JUnit XML schema.

By now risky tests are reported correctly in ParaTest too I meant that now they are verbose and you can debug them.

[cyberhicham]

so no way to ignore them in paratest or report them as warnings instead ?

[Slamdunk]

No, because this is what ParaTest gets from PHPUnit: https://github.com/sebastianbergmann/phpunit/blob/3866b2eeeed21b1b099c4bc0b7a1690ac6fd5baa/tests/end-to-end/loggers/log-junit.phpt#L40-L45

[cyberhicham]

No, because this is what ParaTest gets from PHPUnit: https://github.com/sebastianbergmann/phpunit/blob/3866b2eeeed21b1b099c4bc0b7a1690ac6fd5baa/tests/end-to-end/loggers/log-junit.phpt#L40-L45

I see, well the quick solution for me is to address those risky tests I guess, thanks :+1: