search

NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a programmer, I want meaningful test failures in the systests #279

Open epag opened 3 weeks ago

epag commented 3 weeks ago

Author Name: Jesse (Jesse) Original Redmine Issue: 96860, https://vlab.noaa.gov/redmine/issues/96860 Original Date: 2021-09-29

Given an execution of systests When a systest fails Then I want the message to be meaningful So that I can fix the code or fix the test

Redmine related issue(s): 97014, 101825

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-09-29T15:00:39Z

Example to the contrary:

junit.framework.AssertionFailedError: Comparison with benchmarks failed with code 48. expected:<0> but was:<48>
    at junit.framework.Assert.fail(Assert.java:57)
    at junit.framework.Assert.failNotEquals(Assert.java:329)
    at junit.framework.Assert.assertEquals(Assert.java:78)
    at junit.framework.Assert.assertEquals(Assert.java:234)
    at junit.framework.TestCase.assertEquals(TestCase.java:401)
    at wres.systests.ScenarioHelper.assertOutputsMatchBenchmarks(ScenarioHelper.java:188)
    at wres.systests.Scenario009.testScenario(Scenario009.java:82)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

What is good about this is the "Comparison with benchmarks failed."

What is lacking is "with code 48."

The origin of the "code 48" is a transliteration of the bash assertions in the bash systests into JUnit.

Some improvements that could be made: