search

m-burst / flake8-pytest-style

A flake8 plugin checking common style issues or inconsistencies with pytest-based tests.
MIT License
228 stars 15 forks source link

Rule that suggests explicit `strict=` in `@pytest.mark.xfail` #267

Open lengau opened 11 months ago

lengau commented 11 months ago

Rule request

Description

Detect use of the @pytest.mark.xfail decorator and ensure it explicitly sets a strict=True or strict=False rather than allowing silent (and possibly accidental) use of the default False configuration.

Rationale

The pytest.mark.xfail decorator has a strict parameter that defaults to False, making xfail in CI roughly equivalent to a skip rather than failing if the test passes. This would prevent accidentally committing xfail tests that get hidden in CI.

m-burst commented 10 months ago

Hi @lengau,

Thanks for the issue! Unfortunately I'm not sure about it. I agree that strict xfail should always be preferred to non-strict, but it is probably better to set it globally in the config file, instead of bugging developers with lint errors on each individual test.

There already is issue #24 which suggests to verify that the config file follows the best practices (including xfail_strict = True)

RonnyPfannschmidt commented 10 months ago

the plan in pytest is to make xfail strict by default again