EastAgile / robber.py

BDD / TDD assertion library for Python
MIT License
8 stars 1 forks source link

[b] @must_fail does not work properly with multiple expectations #53

Closed catriuspham closed 7 years ago

catriuspham commented 7 years ago

Currently, the must_fail function checks if the decorated test raises BadExpectation. This leads to a case where there are multiple expectations in one single test, only one of them fails leads to the whole decorated test passes. For example, this test passes, although it must fail.

@must_fail
def test_eq_failure(self):
    expect(1).to.eq(2)
    expect(1).not_to.eq(2)
hieueastagile commented 7 years ago

Oh, that's what we intended to do?

catriuspham commented 7 years ago

Well, then I think we have to make some changes here. I know that my logic seems weird to you. But if we keep it this way, we can trick the test runner into thinking that a must_fail test passes just by adding one failed expectation. Take a look at this code which I take from: https://github.com/EastAgile/robber.py/blob/master/tests/integrations/test_equal.py#L15-L18

@must_fail
def test_eq_failure(self):
    expect(1).to.eq(2)
    expect(1).not_to.eq(1)

As I mentioned is the last comment, because the first two expectations are correct, I can add any expectations, fails or passes, and the test still pass. Therefore, I suggest that we should make it like this:

A must_fail test only passes if all of the expectations inside of it pass.