search

byte-physics / igortest

Igor Pro Universal Testing Framework
https://docs.byte-physics.de/igor-unit-testing-framework/
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

TestsCases are counted wrong due to inconsistency in Hooks called #473

Closed MichaelHuth closed 2 months ago

MichaelHuth commented 2 months ago

At the end of a test suite the TestCaseBegin hook is called again that increases the test case count, but it shouldn't.

There seems to be an issue with calling the internal test case hooks.

To reproduce just e.g. run a single TC from Various.ipf and check the tests="X" in JUnit. It will be 2, but should be 1.

Garados007 commented 2 months ago

I think this is intended. The JUnit output is just the value from wvTestSuite[testSuiteIndex][%NUM_TESTS], which is itself incremented for each IUTF_Reporting_Control#TestCaseBegin call. This function is called for each test case and at the start of each hook.

The test case number in wvTestSuite[testSuiteIndex][%NUM_TESTS] is not just the number of real test cases but includes the number of virtual test cases (for each call of a hook) as well. This is required to know how many entries (real test cases and virtual test cases combined) are associated to a single test suite.

Garados007 commented 2 months ago

And I haven't looked it up but it could be, that JUnit expects the number of virtual and real test cases combined as well. Each hook is printed as a test case in JUnit and a mismatch in the number of entries and the test=X attribute can be considered as invalid JUnit.

MichaelHuth commented 2 months ago

Ok, thanks for the explanation.