Refine Tool Report Standardization

[bbertucc]

If testaro is creating a standard format for automated scan reports, I would suggest a few changes to the formatting, expressed in README.md Standard Format section.

Comments:

• pages: How are pages expressed in results?
• "totals" could be gathered after processing results. to my knowledge scans don't output totals when scanning, and so this also would not need to be a standard output.
• "what" is fairly abstract. Perhaps this could be replaced with something like "description"
• "ordinalSeverity" - I would recommend expressing severity as a tag since, as you mentioned, tools report this factor in different ways. After the fact processing can be used to understand statistics around severity.
• "tagName" - This would be able to be looked up via any HTML associated with the instance, and so I don't see a reason for standardizing the inclusion of it.
• How would relatedElements from axe fit this schema?
• Where would links to additional information go?

Additionally, I don't know if Testaro is the best vehicle for standardizing reports. The value I see is in validating reported issues. Wouldn't it be of more use (and less headache) to start from a single format? I started to lay out that idea here: https://github.com/EqualifyEverything/equalify-formatter/tree/main

Curious to hear your thoughts @jrpool

[jrpool]

These detailed considerations by @bbertucc are much appreciated. Standardized reporting was added to Testaro in May 2023. I avoided doing this on the principle that Testaro should not try to do what can be done on an ordinary server. But not internalizing standardization meant that the bulky origiinal results of all tools would need to be included in every report so that external utilities can standardize them. With reports sometimes reaching ~ 20MB and hundreds of reports in a run, that seemed to interfere with efficient use of the results. So I introduced a lightweight type of standardization, aiming to make it empirically based on the information that most tools expose. For those who find that satisfactory, it is then possible to tell Testaro to omit the original results from reports and include only the standardized results.

As for the specific properties of a the standardResult object, my initial thoughts are:

• Page URLs are represented in reports outside of the standard result.
• Totals aggregate the ordinal severities across instances, except for the testaro tests, which make use of sampling, so for those tests the totals are adjusted to make population estimates from sample results. Yes, they can be computed later from the standard instances. My motivation was to simplify scoring.
• I agree that thewhich and what property names would be more human-friendly if revised, and that is on my agenda. Thanks for the nudge.
• The various schemas for severity and related properties are expressed in the original results, but I believe a standardized result must be tractable, and I am not sure how a tagging approach would be tractable. Perhaps you can give an example.
• The tag names are easy once one knows the element, but choosing the element for any reported instance is complex and subject to interpretation, so the rules for tag-name identification have been incrementally improved with experience. Some instances in testaro are summaries of multiple real instances, and the tag name in those cases is the shared one if it exists, or blank if the underlying instances vary in element types.
• Related elements are not currently reflected in the standard instances. My impression is that they are reported by few tools, and the standard format emphasizes properties that most tools report on.
• The what property could contain links, but if several tools offer diagnosis or remediation advice then perhaps it would be useful and easy to add a property for a link to that advice. That had not occurred to me until you mentioned it.

Enough for now. We can discuss alternative formats in subsequent threads. Thanks again for thinking so hard about all this.

[bbertucc]

Your reasonings make sense, and I'm glad to hear "what" and "which" may become more human-readable.

Wearing the Equalify hat, I would love to see a common standard so that we could integrate Testaro results almost instantly. The biggest holdup to Equalify integrating services beyond WAVE and axe-core is that it takes some time to write a conversion script for results. Axe-core does have an EARL output, as do other scans. That said, I am less convinced that EARL (as it is) is the answer and would hope for another option.

Linking to the Equalify thread where I discuss this more: https://github.com/EqualifyEverything/equalify/issues/257