Closed Jean85 closed 7 years ago
I've looked into the two remaining log formats:
--log-junit
is a XML format; it's less lightweight, and has the downside of not being incremental: if a fatal happens, no log is produced, so we would loose the ability to identify possible culprit method behind fatals--log-teamcity
seems more viable: it is still incremental, since it's used to show results in real time during execution, and it's a simple text stream. I hope it's here to stay, and I've asked that to Sebastian tooI started working on the TeamCity format, but this seems really limited: there isn't any differentiation between skipped and incomplete tests, and also there isn't no explicit difference between failures, errors and warning.
I fear that I will have to fall back to the JUnit XML format, thus losing the possibility of outputting the results for fatal errors and pointing the user toward the test method at the origin of the error.
The JUnit format seems to have his difficulties too. Incomplete and skipped tests doesn't get into the log by default, a piece of XML config is needed:
<logging>
<log type="junit" target="tests/Stub/JUnitLogOutput/Skipped.xml" logIncompleteSkipped="true"/>
</logging>
I will have to inject that piece into every XML config passed to each test, and that may result into a really big I/O slowdown. Fortunately this seems not needed for PHPUnit 6, where the default is reversed and the <skipped/>
tag is implemented, but that's a long way to go...
In either cases, incomplete and skipped messages are lost in the process.
Ok, I'm re-evaluating the situation. The only way that I see feasible for this is the following:
\JSON
namespace (done in fd816be)In this way the JSON log format will still be used when possible, and the JUnit one will be used only when necessary.
With the JUnit format we will still lose some functionality (diff between skipped and incomplete, fatal pinpoint) but this should be the safer strategy
Using test listener to create own log is an option here. Instead of trying to adopt existing format you can declare own one. Fixes #65 as well
That would force me to declare the listener in the XML config file, forcing the user to do that to make Paraunit work; in this way, the listener would be in place anyway, even when launching the tests otherwise, which is a bad thing IMHO.
--printer
Specifies the result printer to use. The printer class must extend PHPUnit_Util_Printer and implement the PHPUnit_Framework_TestListener interface.
https://phpunit.de/manual/current/en/textui.html
A printer will work, just add new parameter during PHPUnit call
Well, at this point I could use directly a copy-paste of \PHPUnit_Util_Log_JSON
, which is the original class that produces the JSON log... Nice suggestion, thanks!
Ok I have first working implementation. I have to overcome one last issue, passing the log dir to the printer which is in a different process. I think I will use PHPUnit's config to do that, since it has a parameter that is passed to the printer constructor, and I can hijack that.
To do that, I will have to copy the config and edit it on the fly. I'm moving this issue to the 0.8 milestone, since it will make Paraunit cross-compatible between multiple PHPUnit versions, without having to wait for the 6 release.
This is a HUGE issue:
https://github.com/sebastianbergmann/phpunit/blob/5.7/ChangeLog-5.7.md#changed PHPUnit will not support the
--log-json
option in future releases (6.0?) and it's getting deprecated. This is the actual fundation of how Paraunit is interpreting test results. I can't find the reasons behind this deprecation, so I've asked Sebastian directly.Luckily our
composer.json
reports"phpunit/phpunit": ">=4.6,<6"
, so we don't have an open constraint that could cause an issue, for now. I'm going to investigate alternative ways to approach the issue, like switching to a different log format, but the JSON one was the most lightweight.PS: TAP format is also being deprecated.