Symfony 5.4 phpunit 9.5 with forceCoversAnnotation and beStrictAboutCoversAnnotation giving random failures

[rbaarsma]

We just started a new project with Symfony 5.4 and phpunit 9.5. We've only implemented standard authetnication and some basic entities and cruds, but already see strange random issues in the tests.

We are executing:

find tests/ -name \"*Test.php\" | vendor/liuggio/fastest/fastest \"XDEBUG_MODE=coverage bin/phpunit {} --coverage-php cov/fastest/{n}.cov;\"
XDEBUG_MODE=coverage tools/phpcov merge cov/fastest/ --text cov/text.txt --html public/test.html

The errors we are getting are all warnings about Risky tests, ex:

There was 1 risky test:

1) Tests\Repository\OrganizationRepositoryTest::test
This test executed code that is not listed as code to be covered or used:
- /var/www/app/src/Entity/User.php:17

/var/www/app/vendor/phpunit/phpunit/src/Framework/TestResult.php:859
/var/www/app/vendor/phpunit/phpunit/src/Framework/TestSuite.php:678
/var/www/app/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:670

In every case the risky message is simply not true. If we execute the test solo (with coverage) or execute the suite without fastest, it works without any risky tests.

For now I've disabled beStrictAboutCoversAnnotation and it works fine.

[DonCallisto]

Hi, could you spin up a project in order to reproduce this behaviour? After that I'll look for the issue. The best would be for you to produce a PR as I'm the only active developer working on fastest at the moment, but if you don't know where to start, just spin up a simple POC for me or other willing to work on this. Thank you.

[DonCallisto]

Sorry, I've closed this by mistake.