byte-physics / igortest

Igor Pro Universal Testing Framework
https://docs.byte-physics.de/igor-unit-testing-framework/
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

FIX: error counter for test assertions #313

Closed Garados007 closed 2 years ago

Garados007 commented 2 years ago

During some refactoring of the handling of internal error messages the error counter got broken. It will increase the error always by one for each error printing even it was an expected error.

This is now fixed as such as an option was introduced to select if the error counter should be increased (normally used by UTF errors) or not (normally the case for UTF assertions).

Close: #311.

Garados007 commented 2 years ago

Small example using this fix:

/// UTF_EXPECTED_FAILURE
Function Test()

    FAIL()
End

Function Foo()

    FAIL()
End
  Start of test "Unnamed"
  Entering test suite "Proc0"
  Entering test case "Test"
  Expected Failure: 0: is false. Assertion "FAIL()" failed in Test (Proc0, line 11)
  Leaving test case "Test"
  Entering test case "Foo"
  0: is false. Assertion "FAIL()" failed in Foo (Proc0, line 16)
  Leaving test case "Foo"
  Failed with 1 errors
  Leaving test suite "Proc0"
  Test finished with 1 errors
    ▶ Assertion "FAIL()" failed in Foo (Proc0, line 16)
  End of test "Unnamed"
Garados007 commented 2 years ago

Without this fix:

  Start of test "Unnamed"
  Entering test suite "Proc0"
  Entering test case "Test"
  Expected Failure: 0: is false. Assertion "FAIL()" failed in Test (Proc0, line 11)
  Leaving test case "Test"
  Entering test case "Foo"
  0: is false. Assertion "FAIL()" failed in Foo (Proc0, line 16)
  Leaving test case "Foo"
  Failed with 3 errors
  Leaving test suite "Proc0"
  Test finished with 3 errors
    ▶ Assertion "FAIL()" failed in Foo (Proc0, line 16)
  End of test "Unnamed"
t-b commented 2 years ago

Nope, that was too quick.

/// UTF_EXPECTED_FAILURE
Function Test()

    FAIL()
End

Function Blahh()

End

Function Foo()

    FAIL()
End

gives

•runtest("procedure")
  Start of test "Unnamed"
  Entering test suite "Procedure"
  Entering test case "Test"
  Expected Failure: 0: is false. Assertion "FAIL()" failed in Test (Procedure, line 6)
  Leaving test case "Test"
  Entering test case "Blahh"
  Test case "Blahh" doesn't contain at least one assertion
  Leaving test case "Blahh"
  Entering test case "Foo"
  0: is false. Assertion "FAIL()" failed in Foo (Procedure, line 15)
  Leaving test case "Foo"
  Failed with 1 errors
  Leaving test suite "Procedure"
  Test finished with 1 errors
    ▶ Test case "Blahh" doesn't contain at least one assertion
    ▶ Assertion "FAIL()" failed in Foo (Procedure, line 15)
  End of test "Unnamed"

please also cite commit introducing the bug with git log --pretty=reference.

Garados007 commented 2 years ago

That was indeed to quick. I forgot to set the default of the parameter which caused the new issue. 🤦‍♂️