[Bug]: `--log-junit` and `--coverage-xml` conflicts

[ndench]

What Happened

I use the junit and xml coverage output from Pest for Infection. This has been working until Pest 2.32.0 where junit support was added to 2.x. Before this, I believe that PHPUnit was outputting it's own Junit output which somehow was working fine.

The previous junit.xml format

```xml ```

The current junit.xml format

```xml ```

The new format is much better, however I believe the xml coverage report also needs to be updated, as it now references tests that do not exist <covered by="P\Tests\CalculatorTest::__pest_evaluable_it_adds"/>

The coverage-xml report

```xml ```

This causes infection to fail because it cannot find the necessary tests:

In TestFileNameNotFoundException.php line 48:

  [Infection\TestFramework\Coverage\JUnit\TestFileNameNotFoundException]
  For FQCN: P\Tests\CalculatorTest. Junit report: /var/www/html/build/junit.xml

How to Reproduce

Using reproducer

I created a small reproducer: https://github.com/ndench/pest-junit-xml-conflicts To use this, you'll need:

• git
• docker compose
• make

1. git clone git@github.com:ndench/pest-junit-xml-conflicts.git
2. make vendor
3. make test

Now you can see the junit output at build/junit.xml and the coverage XML output at build/coverage-xml. You can also run make infect to see the error from infection.

Switch to Pest 2.31

1. make pest-2.31
2. make test

Now you can see the previous build/junit.xml output, and also run make infect to see infection pass.

You can also use make pest-2.32 which will install 2.32.0 to see the first version which introduced the change. And make pest-latest to get back to the latest version.

From scratch

It's also very easy to do this yourself from a blank project.

1. Install a fresh php app (framework doesn't matter, I used Symfony in my sample).
2. Create a PHP file that does something
3. Create a Pest test for the PHP file
4. Run pest with the --log-junit and --coverage-xml flags
5. Note conflicts detailed above in the junit.xml and xml coverage

Sample Repository

https://github.com/ndench/pest-junit-xml-conflicts

Pest Version

2.32.0-2.33.6

PHP Version

8.2.13

Operation System

Linux

Notes

By manually editing the vendor/pestphp/pest/src/Bootstrappers/BootOverrides.php, I can make Pest use the old PHPUnit format to workaround this issue temporarily. But I don't believe this is a solution to the problem on it's own.

[michael-rubel]

I can confirm that Pest 2.x is unusable with Infection. Before merging a proper JUnit logging it was not failing on CI, but also was not reporting any mutants, you were always getting the 100% MSI.

[nuernbergerA]

Hey @ndench

Can you confirm that it works if you manipulate the coverage XML to match the junit output

<covered by="Tests\CalculatorTest::it adds"/>

[ndench]

Hey @nuernbergerA, yes I can confirm that changing the coverage XML manually makes infection work.

ie. I made the following change to coverage-xml/index.xml

    <tests>
-      <test name="P\Tests\CalculatorTest::__pest_evaluable_it_adds" size="unknown" status="success"/>
+      <test name="Tests\CalculatorTest::it adds" size="unknown" status="success"/>
-      <test name="P\Tests\CalculatorTest::__pest_evaluable_it_subtracts" size="unknown" status="success"/>
+      <test name="Tests\CalculatorTest::it subtracts" size="unknown" status="success"/>
    </tests>

And the following change to coverage-xml/Calculator.php.xml

    <coverage>
      <line nr="9">
-        <covered by="P\Tests\CalculatorTest::__pest_evaluable_it_adds"/>
+        <covered by="Tests\CalculatorTest::it adds"/>
      </line>
      <line nr="14">
-        <covered by="P\Tests\CalculatorTest::__pest_evaluable_it_subtracts"/>
+        <covered by="Tests\CalculatorTest::it subtracts"/>
      </line>
    </coverage>

[Sjustein]

Is anything blocking the junit output from changing back? If not, it should be relatively simple to 'rollback' the changes to the junit output right?

[nuernbergerA]

Just rolling back isn't the right thing because then the location is messed up again.

I will open a PR that will output a working XML for both scenarios.

[nuernbergerA]

@ndench @michael-rubel @Sjustein

please check the #1145 if anything works now

[Sjustein]

Infection does indeed understand this format again, but it always reports a 100% mutation score. I'm not sure if this is an issue specific to your implementation though, it might just be something with infection. Thanks for the great speed!