Improve assert failure reporting

metarhia / metatests

Extremely simple to use test framework and runner for Metarhia technology stack 🧪

https://metarhia.com

Other

18 stars 4 forks source link

Improve assert failure reporting #77

Closed lundibundi closed 6 years ago

lundibundi commented 6 years ago

Parts of https://github.com/metarhia/metatests/pull/64 I wanted to have even without util.inspect.

This case:

metatests.testSync('', (test) => test.strictSame('1', 1));

Was reported as:

 Test failed:

        Type: strictSame
       Stack: Error
            ImperativeTest.strictSame (/lib/imperative-test.js:194:10)
            ImperativeTest.metatests.testSync [as func] (/test/imperative.js:182:39)
            ImperativeTest.runNow (/lib/imperative-test.js:357:25)
            process.nextTick (/lib/imperative-test.js:346:12)
    Expected: 1
      Actual: 1
     Message: undefined

Now it is:

 Test failed:

        Type: strictSame
     Message: undefined
    Expected: 1 || number
      Actual: "1" || string
       Stack: Error
            ImperativeTest.strictSame (/lib/imperative-test.js:194:10)
            ImperativeTest.metatests.testSync [as func] (/test/imperative.js:183:39)
            ImperativeTest.runNow (/lib/imperative-test.js:357:25)
            process.nextTick (/lib/imperative-test.js:346:12)

belochub commented 6 years ago

Should we also maybe skip message printing if it is not set?

tshemsedinov commented 6 years ago

For test.error(err); we can also skip Expected and Actual and print just an error stack.

lundibundi commented 6 years ago

@tshemsedinov hmm, I have actually been thinking of setting custom values in such cases. I.e 'No error' string as expected in 'error' case etc. As for skipping, I think it would be better to skip stack and only set the 'actual' to make it clear of what the error is.

tshemsedinov commented 6 years ago

@lundibundi If no message provided for test.error(err); we really can print default message, but if we explain that error is not expected here why do we need Expected: null and Actual: Error message...?

lundibundi commented 6 years ago

I didn't suggest to print a message. I wanted to

print Expected: No Error instead of Expected: null
don't print trace because it's much more clear that we have Expected and Actual than just trace as having trace doesn't explain what the error was (you have to deduce that because this test is error it means that it must not have an error and therefore trace is actually the error that has not been expected. Which is in my opinion much less clear than having Actual and Expected for every test)

lundibundi commented 6 years ago

PTAL @belochub @tshemsedinov. I addressed the comments, this is left for the follow-up PRs.

lundibundi commented 6 years ago

Landed in 11b9a228805a11a99955287b1519deb63840a714.