PHPUnit's --log-json is getting deprecated

[Jean85]

This is a HUGE issue:

https://github.com/sebastianbergmann/phpunit/blob/5.7/ChangeLog-5.7.md#changed PHPUnit will not support the --log-json option in future releases (6.0?) and it's getting deprecated. This is the actual fundation of how Paraunit is interpreting test results. I can't find the reasons behind this deprecation, so I've asked Sebastian directly.

Luckily our composer.json reports "phpunit/phpunit": ">=4.6,<6", so we don't have an open constraint that could cause an issue, for now. I'm going to investigate alternative ways to approach the issue, like switching to a different log format, but the JSON one was the most lightweight.

PS: TAP format is also being deprecated.

[Jean85]

I've looked into the two remaining log formats:

• --log-junit is a XML format; it's less lightweight, and has the downside of not being incremental: if a fatal happens, no log is produced, so we would loose the ability to identify possible culprit method behind fatals
• --log-teamcity seems more viable: it is still incremental, since it's used to show results in real time during execution, and it's a simple text stream. I hope it's here to stay, and I've asked that to Sebastian too

[Jean85]

I started working on the TeamCity format, but this seems really limited: there isn't any differentiation between skipped and incomplete tests, and also there isn't no explicit difference between failures, errors and warning.

I fear that I will have to fall back to the JUnit XML format, thus losing the possibility of outputting the results for fatal errors and pointing the user toward the test method at the origin of the error.

[Jean85]

The JUnit format seems to have his difficulties too. Incomplete and skipped tests doesn't get into the log by default, a piece of XML config is needed:

<logging>
    <log type="junit" target="tests/Stub/JUnitLogOutput/Skipped.xml" logIncompleteSkipped="true"/>
</logging>

I will have to inject that piece into every XML config passed to each test, and that may result into a really big I/O slowdown. Fortunately this seems not needed for PHPUnit 6, where the default is reversed and the <skipped/> tag is implemented, but that's a long way to go...

In either cases, incomplete and skipped messages are lost in the process.

[Jean85]

Ok, I'm re-evaluating the situation. The only way that I see feasible for this is the following:

• leave the JSON log parsing in place for now
• refactor it in a \JSON namespace (done in fd816be)
• move this issue to the 1.0 milestone, since it's PHPUnit 6 related
• when working on PHPUnit 6 support:
  • implement a generic, abstract way to store test outcomes, not tied to log parsing
  • add a new strategy for log parsing, based on JUnit format
  • use the new strategy only if PHPUnit 6 is used, otherwise fall back to JSON

In this way the JSON log format will still be used when possible, and the JUnit one will be used only when necessary.

With the JUnit format we will still lose some functionality (diff between skipped and incomplete, fatal pinpoint) but this should be the safer strategy

[dxops]

Using test listener to create own log is an option here. Instead of trying to adopt existing format you can declare own one. Fixes #65 as well

[Jean85]

That would force me to declare the listener in the XML config file, forcing the user to do that to make Paraunit work; in this way, the listener would be in place anyway, even when launching the tests otherwise, which is a bad thing IMHO.

[dxops]

--printer
Specifies the result printer to use. The printer class must extend PHPUnit_Util_Printer and implement the PHPUnit_Framework_TestListener interface.

https://phpunit.de/manual/current/en/textui.html

A printer will work, just add new parameter during PHPUnit call

[Jean85]

Well, at this point I could use directly a copy-paste of \PHPUnit_Util_Log_JSON, which is the original class that produces the JSON log... Nice suggestion, thanks!

[Jean85]

Ok I have first working implementation. I have to overcome one last issue, passing the log dir to the printer which is in a different process. I think I will use PHPUnit's config to do that, since it has a parameter that is passed to the printer constructor, and I can hijack that.

To do that, I will have to copy the config and edit it on the fly. I'm moving this issue to the 0.8 milestone, since it will make Paraunit cross-compatible between multiple PHPUnit versions, without having to wait for the 6 release.