search

NREL / OpenStudio-measure-tester-gem

Other
3 stars 0 forks source link

Sometimes Dashboard in measures/lib/test_results shows just 1 Failure but under test at top shows many #70

Open DavidGoldwasser opened 9 months ago

DavidGoldwasser commented 9 months ago

I was wondering why Windows has 7 test failures while Mac and CI just had one, but then when I dug in a bit more, mac also had 7 failures. I thought I'd document with screenshots of same Mac run on lib/measures/test_results/dashboard/index.html on the left and test/html_reports/index.html on the right.

Screen Shot 2023-11-17 at 3 09 59 AM Screen Shot 2023-11-17 at 3 10 36 AM

It happens to be the test that shows is the last measure. I thought maybe somehow a single test is overridden in the dashboard, but in the other case with multiple failures RadiantSlabWithDoas in model articulation that has 4 failures. All of them show in the dashboard.

Screen Shot 2023-11-17 at 3 20 17 AM
DavidGoldwasser commented 9 months ago

I also realized same thing is happening with XcelEDAReportingandQAQC on common measures, but in that case it is multiple errors and not multiple failures.

DavidGoldwasser commented 9 months ago

Oddly I noticed on local Mac dashboard it shows tested with 3.6 but the file in test output for measure are 3.7.0 models. the test_results dashboard does show as just edited so not looking at old file. But I'll wipe everything out and re-run maybe some elements of it are older.

DavidGoldwasser commented 9 months ago

When I wiped test results for local common measures away and run then it doesn't populate the dashboard. But on CI the dashboard is generated properly?