Add Common Test Report Format (CTRF) support.

[steve-bate]

I'm not loving TAP. I feel like the "spec" is messy and not well-supported by the related software libraries. Much of the useful information is in comments that will need to be parsed to do result aggregation.

I ran across a reference to Common Test Report Format (CTRF). I'm still investigating, but this looks like a JSON-based "standard" report format similar to xUnit reports (but not XML). This would allow us to represent all the test results and test metadata in JSON and should interoperate with other tooling (static content generation, in-browser rendering, etc.) more effectively than TAP.

What do you think about me adding CTRF report support to feditest?

[jernst]

I admit tap is a bit of an odd format. We added it based on a customer request :-)

Let's enumerate what we are actually trying to accomplish:

• a tester needs to be able to understand what just happened -- any human-readable format is fine
• we want to publish test results on a website -- tabular report in HTML

We don't really have requirements to automatically open issues or do some other kind of integration? As far as I can think?

What else? It occurs to me it may be easier to emit HTML directly than to emit something else (whether that's tap or CTRF or JSON) and then parse that again and go to HTML.

[steve-bate]

I admit tap is a bit of an odd format. We added it based on a customer request :-)

I understand. If I had known about CTRF previously, I would have pushed back harder on the TAP recommendation. I even preferred using xUnit XML over TAP, but I know that many devs have a viscerally negative reaction to anything XML. ;-) And I understand that reaction, to some extent.

What else? It occurs to me it may be easier to emit HTML directly than to emit something else (whether that's tap or CTRF or JSON) and then parse that again and go to HTML.

I'd prefer to have a JSON test data format. We may want to aggregate test result data from multiple sessions and/or test plans before rendering the HTML. JSON will give us the flexibility to do that relatively easily. This may be an important feature when we have 100s of tests multiplied by many sessions.

(Side note: Since any JSON structure can be represented as YAML, which some people consider more human-readable, that's another option -- dump CTRF as YAML)