byte-physics / igortest

Igor Pro Universal Testing Framework
https://docs.byte-physics.de/igor-unit-testing-framework/
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

Improve error messages #449

Open Garados007 opened 1 year ago

Garados007 commented 1 year ago

All error messages are right now single-line and difficult to read and understand. The format has grown historically and is difficult to handle.

For example, the following code produces right now:

static Function TestCase()
    DoTest(1)
End
static Function DoTest(variable foo)
    INFO("should be zero")
    CHECK_EQUAL_VAR(0, foo)
    INFO("check waves")
    CHECK_EQUAL_WAVES({ 0 }, { foo })
End
  0 == 1: is false. Assertion "CHECK_EQUAL_VAR(0, foo)" failed in Foo#TestCase➔Foo#DoTest (TestProc, line 8➔12)
    ⓘ should be zero
  Assuming equality using mode WAVE_DATA for waves _free_ (0x8ac395a0) and _free_ (0x8ac39100); detailed: Waves difference:
Wave1: _free_ (0x8ac395a0)
Wave2: _free_ (0x8ac39100)
 Dimensions | Labels | Value |
------------|--------|-------|
 [0]        |        | 0     |
            |        | 1     |
: is false. Assertion "CHECK_EQUAL_WAVES({ 0 }, { foo })" failed in Foo#TestCase➔Foo#DoTest (TestProc, line 8➔14)
    ⓘ check waves

There is a lot to improve:

  1. Generate a multi-line stack trace with each function call on its own line
  2. Use more understandable error messages on the first line right after the error keyword WARN, CHECK or REQUIRE. If the error message has some data (e.g. the diff for *_EQUAL_WAVE) keep them on their own lines with some indicators.
  3. Keep error info on their own lines.

An example could be:

CHECK_EQUAL_VAR: 0 is not equal to 1
    ⓘ should be zero
    -> Foo#DoTest: TestProc:L12
       Foo#TestCase: TestProc:L8
CHECK_EQUAL_WAVES: WAVE_DATA failed
    ■ Wave1: _free_ (0x8ac395a0)
    ■ Wave2: _free_ (0x8ac39100)
    ■  Dimensions | Labels | Value |
    ■ ------------|--------|-------|
    ■  [0]        |        | 0     |
    ■             |        | 1     |
    ⓘ check waves
    -> Foo#DoTest: TestProc:L14
       Foo#TestCase: TestProc:L8

Not only can the log output in Igor or JUnit be improved with this change but the extended error reporting for Github Actions (#448) as well.

Look into MIES ASSERT for standardizing stacktrace output. With this change, all messages for error reporting need to be reconsidered and brought into a uniform format.