epag
commented
1 month ago Original Redmine Comment Author Name: James (James) Original Date: 2019-06-27T18:45:37Z
So, yeah, to clarify, I would:
- Disaggregate the current tests, looking for what they are testing, making a top-level distinction between data format, data source, and shape of evaluation
- Nominate any tests that are evaluation-shape-important (i.e., important for the evaluation shape, not the data input format or source). Rationalize them to distinct cases and make them stable. I don't care how they are rendered stable, but NWIS-real is not. Files are probably easiest. These tests are invariant to data format and source.
- Nominate any tests that are data-source-important. Separate these into tests that need a real service call and tests that can be mocked. Rationalize each to distinct cases and make them stable. Making the first set stable means downgrading the assertions of those tests to make them robust to changes in NWIS. The second set should be unit/integration tests w/ mocked calls. These tests are invariant to shape of evaluation which, to me, implies that they are not system tests.
- Nominate any tests that are data-format-important. Rationalize them to distinct cases and make them stable. I don't care how they are rendered stable, but NWIS real is not. Files are probably easiest. These tests are invariant to shape of evaluation which, to me, implies that they are not system tests. They are possibly invariant to format too (although NWIS WaterML may be NWIS-centric).
Mostly, when this is done, I would expect 2-3 7xx system tests remaining of which, perhaps, 1-2 make real service calls with high-level assertions like "this clown car looks good, but the drivers may be suspect and I'm not checking them". Mostly, I would expect more unit/integration tests, some of which may involve mock-NWIS-service calls. This is where we check the clowns and make sure they have proper clown shoes and red noses.
I believe Jesse and I were heading in roughly the same direction.
Author Name: Hank (Hank) Original Redmine Issue: 65465, https://vlab.noaa.gov/redmine/issues/65465 Original Date: 2019-06-27
The current tests are too unstable, being reliant on a USGS service wherein the data can change often even for older data sets.
From the development discussion on 6/27/19, I took these notes which are crude but touch on various points:
I will consider this ticket resolved once we have identified a reasonable approach to testing the ability of the WRES to use the USGS web service and then implement that new capability. Its expected that this will impact the structure/purpose of the current 700 series tests.
Targeting this for Release 1.10 until I hear otherwise. Thanks,
Hank