NREL / OpenStudio-measure-tester-gem

Other
3 stars 0 forks source link

Sometimes Dashboard appears to have no errors but when click view details Errrors shows up #71

Open DavidGoldwasser opened 9 months ago

DavidGoldwasser commented 9 months ago

Ulike #70 which still stops CI from passing but just under-reports errors/failures this will make CI pass without clear indication of problem.

You can see this happens with GenericQAQC. On common measures

Screen Shot 2023-11-17 at 3 57 49 AM

I think something similar may be happening with OpenStudio results. It is failing tests on Mac and Windows locally but seems to Pass on CI. Maybe it is ok on Linux and not Mac or Windows, or maybe some variation of issue above

Screen Shot 2023-11-17 at 3 58 27 AM

I'm not sure how to drill in and look at the other dashboard on CI

DavidGoldwasser commented 9 months ago

I went back on the CI and confirmed this issue pre-dates rc2 (when we switched to C++ workflow in place of Workflow Gem). The rc1 CI looks just like this. Prior run with 3.6.9 was fine. The actual error is likely E+ change, but the missed logging on Dashboard happens in both 3.7.0 rc1 and rc3 (rc2 test failed to early to test this)

DavidGoldwasser commented 9 months ago

Here is local screenshot of generic qaqc, I think this is failing on QAQC but doesn't show up in CI as failure because the error doesn't make it to the dashboard. Screen Shot 2023-11-28 at 9 56 11 AM