search

lispci / fiveam

Common Lisp regression testing framework
BSD 3-Clause "New" or "Revised" License
186 stars 33 forks source link

Feature request: Expected failures #55

Closed uint closed 4 years ago

uint commented 6 years ago

Would you see handling expected failures as something that should/could be a part of 5am itself, or implemented by 5am users for their projects (via, say, a custom explainer)?

If it's the former, would you accept a PR and do you have a way you'd prefer it to be implemented?

The way I'd see it is that a list of test cases expected to fail (nil by default) can be passed to the #:run! (or #:run, or #:run-all-tests) function and checked against, but I'd need to dig into the code.

foretspaisibles commented 5 years ago

Why wouldn't you SKIP tests you expect to fail? (Or, the other way around, why would one use SKIP if not to tag expected failures?)

uint commented 5 years ago

You're making the assumption that I knew about skipping. It might be wrong.

I honestly don't remember at this point what my reasoning was, but glancing through 5am code I probably thought skipping tests is purely a thing introduced for tests whose dependencies are not satisfied. I see now that there is a macro that more explicitly generates a TEST-SKIPPED result too, though. Thanks for that.

Why wouldn't you SKIP tests you expect to fail?

Probably to separate the list of expected failures from test definitions. I know I'd prefer that.

That said, seeing that there is some functionality that allows skipping expected failures, I'm more than happy to close this issue if no one feels the need for a different mechanism.

sionescu commented 4 years ago

SKIP seems like a good solution. If that's not enough, reopen this bug and we'll see.