[x] Update docker build to install things before source copy.
Fixes #12 (nested dependency cannot build without gcc and musl-dev on alpine image).
Means that sudo make test does not require the docker image to rebuild every time source is updated.
[x] Add setup_raises marker, re-factor logic to be reused.
See below checklist.
[x] Add match=r"expr" argument. Being able to supply regular expressions is incredibly useful.
Very similar to with pytest.raises(...), except explicit failure given if both message and match are provided.
I cannot think of a valid reason why you would ever need both at the same time.
Not allowing both makes logic for message testing easier. If allowing both, which to you test? Both match and message? How do you format the failure_message? Basically, force message or match not both can be removed, but need counsel on what you want when both supplied.
[x] Add some testing coverage :tada:
Strange behavior with import statements, likely because of how pytest actually uses this plugin.
Attempts at mocking cannot rectify, see setup.cfg.
See it in action! It was really useful to have this during this re-structure because there were previously many things I introduced that were never tested and now they are :slightly_smiling_face:
[ ] Would you like to add an integration with something like https://codecov.io/ ? It will basically always be at 98% due to unknown coverage parsing, but hey that's pretty good :wink:
All you (or Authentize) need to do is enable the integration and I can add the badge and update .travis.yml to actually do the reporting. Codecov isn't the only one, coveralls is popular, I just love the codecov interface / it's really easy to upload to.
[ ] Update README to explain new additions (match and setup_raises).
I'm hoping to get some feedback on if what I'm proposing here is actually valid, instruction update pending discussion on pytest/#4175
How it All Works
This is also detailed thoroughly in the docstrings, but here's the overview of how it all works together.
Share as much code as possible between raises and setup_raises markers.
Hook impl executed (either pytest_runtest_setup for setup_raises or pytest_runtest_call for raises like usual).
item, outcome, and string marker_name indicating whether its coming from setup or call forwarded to _pytest_raises_validation.
If marker setup_raises or raises are found, there are now three distinct scenarios tested.
Case 1: test raised exception, and it isinstance of the expected exception class (either via exception=ExceptionClass or the default catch-all Exception if unspecified.
If the message or match are specified, the exception message is then validated.
If the message or match do not check out, _pytest_fail_by_mark_or_set_excinfo is called.
Case 2: test raised an exception, but it was of an unexpected type.
_pytest_fail_by_mark_or_set_excinfo fails the test.
Case 3: the test did not raise, but since it was marked with either setup_raises or raises test is failed using _pytest_fail_by_mark_or_set_excinfo.
Distinctions must be made between setup_raises and raises. I don't understand the details (hence the pytest issue linked above), but basically there are problems with setting outcome.excinfo or calling pytest.fail(...).
The distinctions between the two are made because we do not want tests to ERROR, we want them to FAIL. See pytest/#4175 for details discovered by trial-and-error.
So to deal with this, I create a "handshake" using an intentionally obtusely named marker for the setup_raises so that tests properly FAIL rather than ERROR. The general idea is: during pytest_runtest_setup, cases 1-3 are "flagged" with a secret marker, which is then checked for during pytest_runtest_call. Search secret_marker in code.
Since (4) had to go down like this, I leveraged the same trick to also add some checks on things like is the exception=??? actually valid, etc.
I wouldn't go as far as to say the implementation is particularly clean, but it works reliably and I really need it. This is the revisited implementation of what I was discussing in #9.
There is a small concern. Consider the newly added test_pytest_mark_setup_raises_unexpected_exception_fixture in test_raises.py:
import pytest
class SomeException(Exception):
pass
class AnotherException(Exception):
pass
@pytest.fixture
def raise_me():
raise AnotherException('the message')
@pytest.mark.setup_raises(exception=SomeException)
def test_pytest_mark_setup_raises_unexpected_exception_fixture(raise_me):
pass
In any case where setup_raises is used, the test function body should be empty (just a single pass to compile). Logically, I cannot think of any scenario where if you expect setup_raises you also want to have an implementation in the test function. But the implications are:
If you have testing code in the function body and are banking on your setup raising, this is a contradiction.
You therefore cannot mark a function with both @pytest.mark.raisesand@pytest.mark.setup_raises. Only one is allowed -- you either are expecting the setup to fail, in which case the test body should be empty, or you want the test body to fail, in which case the test setup better not raise since you need a valid test setup.
Anyway, this is a big change and should definitely be reviewed. I hesitated to put them all in one PR, but at the end of the day the docker / pylint fixes were needed for me to test things, and adding the match independently in the context of the changes for setup_raises would make for a guaranteed merge conflict between the two.
I look forward to updating / fixing / changing anything you need. If you don't want this change in this repo then I'll package my own because I can't fully test my code without setup_raises :scream:
sudo make test
does not require the docker image to rebuild every time source is updated.setup_raises
marker, re-factor logic to be reused.match=r"expr"
argument. Being able to supply regular expressions is incredibly useful.with pytest.raises(...)
, except explicit failure given if bothmessage
andmatch
are provided.match
andmessage
? How do you format thefailure_message
? Basically, forcemessage
ormatch
not both can be removed, but need counsel on what you want when both supplied.import
statements, likely because of howpytest
actually uses this plugin.setup.cfg
.98%
due to unknown coverage parsing, but hey that's pretty good :wink:Authentize
) need to do is enable the integration and I can add the badge and update.travis.yml
to actually do the reporting. Codecov isn't the only one, coveralls is popular, I just love the codecov interface / it's really easy to upload to.match
andsetup_raises
).pytest/#4175
How it All Works
This is also detailed thoroughly in the docstrings, but here's the overview of how it all works together.
raises
andsetup_raises
markers.pytest_runtest_setup
forsetup_raises
orpytest_runtest_call
forraises
like usual).item
,outcome
, and stringmarker_name
indicating whether its coming fromsetup
orcall
forwarded to_pytest_raises_validation
.setup_raises
orraises
are found, there are now three distinct scenarios tested.isinstance
of the expected exception class (either viaexception=ExceptionClass
or the default catch-allException
if unspecified.message
ormatch
are specified, the exception message is then validated.message
ormatch
do not check out,_pytest_fail_by_mark_or_set_excinfo
is called._pytest_fail_by_mark_or_set_excinfo
fails the test.setup_raises
orraises
test is failed using_pytest_fail_by_mark_or_set_excinfo
.setup_raises
andraises
. I don't understand the details (hence thepytest
issue linked above), but basically there are problems with settingoutcome.excinfo
or callingpytest.fail(...)
.pytest/#4175
for details discovered by trial-and-error.setup_raises
so that tests properly FAIL rather than ERROR. The general idea is: duringpytest_runtest_setup
, cases 1-3 are "flagged" with a secret marker, which is then checked for duringpytest_runtest_call
. Searchsecret_marker
in code.exception=???
actually valid, etc.I wouldn't go as far as to say the implementation is particularly clean, but it works reliably and I really need it. This is the revisited implementation of what I was discussing in #9.
There is a small concern. Consider the newly added
test_pytest_mark_setup_raises_unexpected_exception_fixture
intest_raises.py
:In any case where
setup_raises
is used, the test function body should be empty (just a singlepass
to compile). Logically, I cannot think of any scenario where if you expectsetup_raises
you also want to have an implementation in the test function. But the implications are:setup
raising, this is a contradiction.@pytest.mark.raises
and@pytest.mark.setup_raises
. Only one is allowed -- you either are expecting the setup to fail, in which case the test body should be empty, or you want the test body to fail, in which case the test setup better not raise since you need a valid test setup.Anyway, this is a big change and should definitely be reviewed. I hesitated to put them all in one PR, but at the end of the day the docker / pylint fixes were needed for me to test things, and adding the
match
independently in the context of the changes forsetup_raises
would make for a guaranteed merge conflict between the two.I look forward to updating / fixing / changing anything you need. If you don't want this change in this repo then I'll package my own because I can't fully test my code without
setup_raises
:scream: