byte-physics / igortest

Igor Pro Universal Testing Framework
https://docs.byte-physics.de/igor-unit-testing-framework/
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

Introduce data generators as virtual test cases #389

Closed Garados007 closed 1 year ago

Garados007 commented 1 year ago

Each data generator is now wrapped as a virtual test case which is also reported into the test results. To achieve this the setup for data generator has been changed into three phases:

  1. Loading the list of data generate and do some general checks (for example if the data generator can be found or its function signature is correct).
  2. Execute all data generators one by one and wrap each of them in their own test case. Do some basics checks with the generated results (for example if an invalid wave is returned).
  3. Assign these data generators to their dependend test cases. Do some checks (for example if the types of the generated results match the function signature of the test case).

Close #353


This PR does also consists of two refactoring steps. In one refactoring commit are all data generator, multi data and multi-multi data test related functions moved in their own file which reduces the size and complexity of UTF_Basics a lot. The second refactoring commit moves some polyfill methods into their own file.

Garados007 commented 1 year ago

@t-b @MichaelHuth Ready for review.

Garados007 commented 1 year ago

@MichaelHuth I have changed it in that way that it will call ReportErrorAndAbort if I get an invalid wave back from the data generator. This will report the error and stops executing RunTest without any writing JUnit or TAP output. This should be the same behavior as before if I am right.

The other idea with collecting the errors and printing JUnit only for the failed data generator suite is quite interesting but has some problems:

Garados007 commented 1 year ago

@MichaelHuth Please re-review this.

t-b commented 1 year ago

Assigning @MichaelHuth as I think this was the intention.

Garados007 commented 1 year ago

@MichaelHuth Ready for review.

t-b commented 1 year ago

@MichaelHuth Just to be sure that this works now: Did you try to run one MIES hardware test with this PR?

MichaelHuth commented 1 year ago

I did not wrote that it works now. I was about to do some MIES tests as final test.

@Garados007 It turns out that the setup for test cases for the data generators is not complete as I get an IUTF internal RTE when using an assertion in a data generator function from a non-existing igor_debug_assertion in DebugFailedAssertion.

If the global is created the MIES HW tests start running...

t-b commented 1 year ago

I did not wrote that it works now. I was about to do some MIES tests as final test.

Thanks, sorry for being impatient.

Garados007 commented 1 year ago

I have refactored the debug helper methods into their own file. This allowed me to copy the set/reset of the Igor Debugger from regular test cases to the data generators. During this fixed also the bug with the non-existent igor_debug_assertion as it is always created before each data generator execution.

Garados007 commented 1 year ago

@MichaelHuth Ready for review.

t-b commented 1 year ago

Thanks both!