search

NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a developer, I want to report on system tests in a way that best leverages the reports that junit produces #221

Open epag opened 3 weeks ago

epag commented 3 weeks ago

Author Name: James (James) Original Redmine Issue: 73070, https://vlab.noaa.gov/redmine/issues/73070 Original Date: 2020-01-06

Expected behavior:

Given a desire to report on system test results produced by junit in a developer-friendly way When I consider how to report those results Then I want to make best use of the output already produced by junit

( junit produces a bunch of stuff, notably a summary test report in html w/ associated reports for individual tests and a style sheet, all under build/reports, as well as the xml reports for individual tests under build/test_results. Other testing frameworks produce a consolidated test report that is suitable for dispatch via e-mail, such as the @emailable-report.html@ produced by testng. We could build our own style of report on top of the junit stuff using a gradle task. Our chosen delivery mechanism is e-mail, but embedded html may not be the way to go vs. text. At the same time, it's nice to see the report right there, without having to unpack it and open it. Whatever we decide, we should make best use of the stuff that junit already produces )

I lean towards an e-mail with a text/plain summary of what passed/failed and the html report as an attachment (possibly consolidated), allowing the developer to bury down into the individual tests, pass/fail state and even the standard out, which is recorded by junit. The execution log would disappear as an attachment.

In short, opinions welcome.

Actual behavior:

Currently we e-mail a summary of the test results prepared by scanning the log file (I believe), together with the execution log itself as an attachment. In short, we don't make best use of the stuff that junit provides automatically, on completion of all tests.

epag commented 3 weeks ago

Original Redmine Comment Author Name: James (James) Original Date: 2020-01-06T18:58:29Z

Some discussion around #69423-117.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-09-29T14:56:54Z

The weird assertions/messaging/printouts might be part of this but this ticket sounds more like something to do with emailing results, not assertions and printing.

epag commented 3 weeks ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-09-29T15:04:21Z

Yeah, this was more about reporting the stuff that junit produces. Overall, the system test code needs to be broadly refactored, but none of that was this ticket.