search

digitalbazaar / mocha-w3c-interop-reporter

Mocha W3C Interoperability Test Suite reporter
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Provide more details about failing tests in the report to help developers #24

Open filip26 opened 1 year ago

filip26 commented 1 year ago

Hi, today's report shows OK/FAIL state only. There is some error message (when hover over an error), but it's the same generic message for all tests.

Including more details, like requests that caused the fail, would be super helpful and could boost the compliance. Now it's very hard to understand what went wrong, and impossible to replicate the test conditions because test requests are generated. (No need to include auth tokens or anything sensitive)

Thank you!

aljones15 commented 1 year ago

So after looking into this, adding more data to the error modal here: https://github.com/digitalbazaar/mocha-w3c-interop-reporter/blob/main/templates/error.hbs

might be a bit difficult. Additionally adding a second modal would also be a bit hard. Some options:

  1. Click on failure to see error and test data in a modal.
  2. Mouse over test title to see test data
  3. Use the existing test data stuff such as that seen here

In both cases how that data is transmitted to the reporter continues to be the issue. Version 2 of this library is supposed to run in parallel, but last time I checked mocha strips meta data when passing test suites and test between processes. So we have existing code that could show a single request body. I'll continue to think about this, but for now the easiest way is to use the existing reportData array in the test suites where we need more information on the request body.