search

nose-devs / nose

nose is nicer testing for python
http://readthedocs.org/docs/nose/en/latest/
1.36k stars 396 forks source link

Warning testing/catch_warnings context #1002

Closed seberg closed 8 years ago

seberg commented 8 years ago

Sorry, this is probably not the right place, had tried the google group nose-users first, but that did not work out (only tried to use it as a mailing list without google adress), but thought I would give it a shot anyway. Also I am not sure this is interesting to nose itself, but thought I would ask, so feel free to just instant close ;).

I was wondering about a broader interest in reliably testing warnings. (tl;dr was thinking of a new warning context for numpy, in https://github.com/numpy/numpy/pull/7763 and it might be interested to others also frustrated with warning testing?)

Basically, the story is that in NumPy I have been seriously annoyed in not being able to reliably test warnings. This means on the one hand, suppress/filter away warnings in tests where we do not care about them, on the other hand actually be able to check that they are given correctly elsewhere. Additionally, I was annoyed that when creating a new warning it is often hard to see what parts are actually affected, because it does not show up everywhere.

The biggest problem is that before some Python 3.4 version, this type of code is not reliable due to a bug (http://bugs.python.org/issue4180):

with warnings.catch_warnings():
    warnings.simplefilter("ignore", DeprecationWarning)
    function_causing_deprecation_warning()

with warnings.catch_warnings(record=True):
    warnings.simplefilter("always", DeprecationWarning)
    # Depending on the warning location, the warning may not be given
    same_or_other_function_causing_same_warning()

The same happens if a catch warnings context with recording catches an extra warning (which maybe should be set to error though, but in my opinion ideally not in release versions -- a new downstream deprecation warning, should not fail the tests, but would be good to be visible).

Another thing is that nesting of these contexts is not always cleanly possible. Also specific catching of one warning, but printing (not raising) of other warnings is difficult or impossible since the inner catch warning context will catch all warnings.

Because especially the first bug annoyed me so much; to get around these issues I implemented a new catch warnings context on steroids for possible inclusion in numpy in https://github.com/numpy/numpy/pull/7763 Now I have been wondering a bit whether there may even be more interest in such a hack/context manager, or am I the only one being frustrated by warnings in tests?

And of course, if we would want to put something like that into nose or elsewhere, we could use that in the future and not ship some weird context manager indefinitely. Sorry again for the spam.

seberg commented 8 years ago

Sorry for the spam, will just close this again (if anyone cares, feel free to comment at numpy/testing-in-python mailing list).