search

EnricoMi / publish-unit-test-result-action

GitHub Action to publish unit test results on GitHub
Apache License 2.0
620 stars 184 forks source link

TAP support? #484

Open BytewaveMLP opened 1 year ago

BytewaveMLP commented 1 year ago

The Test Anything Protocol is yet another way some testing frameworks might output test suite information for reporting purposes. I'm looking to integrate some sort of test reporting to a Ruby project, and existing reporting formats for RSpec leave a lot to be desired. I understand JUnit is an option, but the most popular available formatter that outputs JUnit XML groups all tests into one testsuite tag per runner, rather than grouping actual test suites together, which is far from optimal. The RSpec TAP formatter, on the other hand, outputs group data as expected.

The downside of TAP is that it would require a bespoke parser, rather than coming as a standard format eg JSON, XML. I'm also not sure how trivial it would be to implement in a project like this; it seems there are separate modules for different suite reporting schemes, which would be a start I suppose.

EnricoMi commented 1 year ago

I looked into supporting TAP and wasn't very delighted by the set of information provided on test failure. What this action needs to be useful (except for merely provide success and failure counts) is the test file name, test name, line number, and some message. The only information TAP provides is some test name string and message. Looks like TAP 14 provides a structured YAML block with some of those information.

If you like to provide some example tap (TAP 14 spec) files that include failures and errors, I could take a look again.

BytewaveMLP commented 1 year ago

Hm. So upon closer investigation, the community-made TAP reporter for RSpec that I wanted to use doesn't actually support TAP 14, but rather an extended version of TAP 13. However, I might give a go at extending it to use TAP 14, as it doesn't seem excessively difficult. If I manage that, I'll provide you with a sampling of output.

nicolalunghi-xlnx commented 1 month ago

Hi, having this even as a basic implementation would be useful. My use case is the linux kernel kunit test results, that is reported only in tap format. At the moment I am using the python package tap2junit to convert tap to junit xml but would be nice if this extension would support tap natively