search

sebastianbergmann / phpunit

The PHP Unit Testing framework.
https://phpunit.de/
BSD 3-Clause "New" or "Revised" License
19.7k stars 2.2k forks source link

@xfail annotation #4481

Closed MikkelPaulson closed 2 years ago

MikkelPaulson commented 4 years ago

A useful feature of pytest is its @pytest.mark.xfail annotation, which indicates that a test should run but is expected to fail. If a test so annotated does pass, the test run interprets it as a failure.

Generally this is used in TDD and BDD to indicate that a bug or feature is still a work in progress. It's distinct from skipping in that the failure when the test starts passing should prompt the developer to remove the annotation or update the test accordingly.

I did some looking around and was surprised to see that this seems to be a unique feature of pytest (at least by that name). I'd like to add it to PHPUnit, but wanted to get a 👍 👎 before starting work.

sebastianbergmann commented 4 years ago

I am open to the idea but I do think we should deprecate (and later remove) markTestIncomplete().

MikkelPaulson commented 4 years ago

I'm not sure that they're exactly synonymous. @xfail indicates that the feature is incomplete, and markTestIncomplete() indicates that test coverage is missing/deficient and needs to be updated – at least in theory. They're both ways of saying that the test and the code are out of sync, but they do have different meanings and behaviour.

I feel like in most real-world projects incomplete tests are a sloppy way of saying "I don't feel like testing this" and the annotation will remain until the test is discarded completely, but maybe I should have more faith in people.

I'm not opposed to deprecating it, though. It doesn't do anything you couldn't do by ending your test with $this->skip("incomplete").

morozov commented 4 years ago

FWIW, besides being often abused, markTestIncomplete() is useful in the cases where the same test is run against multiple implementations and some of them are known to fail. This way, an incomplete test documents a known issue.

remorhaz commented 4 years ago

This feature can be useful for testing of different PHPUnit constraints and extensions. It would be very convenient just to annotate that a concrete test "should fail".

sebastianbergmann commented 2 years ago

Won't implement.

weitzman commented 2 years ago

Any chance yu are open to this in some way other than an @xfail annotation? markTestIncomplete() isnt a great substitute IMO.