search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
12.1k stars 2.68k forks source link

`pytest -Werror` fails with ignore warnings in pytest.ini #8534

Open jieru-hu opened 3 years ago

jieru-hu commented 3 years ago

Hi, first of all, thanks for the wonderful library!!

My code raises UserWarnings and I want to ignore them when running python -Werror .. I've been following this doc.

Minimal repro:

# test_warning.py
from pytest import mark

import warnings

def test_warnings():
    warnings.warn(category=UserWarning, message=f"user warning")
# pytest.ini
[pytest]
filterwarnings =
    error
    ignore::UserWarning

Run:

$ pytest -Werror .
================================================================================================================================ test session starts ================================================================================================================================
platform darwin -- Python 3.8.8, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: test_warnings, configfile: pytest.ini
plugins: snail-0.1.0, typeguard-2.11.1, hydra-core-1.1.0.dev6
collected 1 item                                                                                                                                                                                                                                                                    

test_warnings.py F                                                                                                                                                                                                                                                            [100%]

===================================================================================================================================== FAILURES ======================================================================================================================================
___________________________________________________________________________________________________________________________________ test_warnings ___________________________________________________________________________________________________________________________________

    def test_warnings():
>       warnings.warn(category=UserWarning, message=f"user warning")
E       UserWarning: user warning

test_warnings.py:8: UserWarning
============================================================================================================================== short test summary info ==============================================================================================================================
FAILED test_warnings.py::test_warnings - UserWarning: user warning
================================================================================================================================= 1 failed in 0.24s =================================================================================================================================
(pytest38) jieru-mbp:test_warnings jieru$

If instead I decorate the test with @mark.filterwarnings("ignore::UserWarning"), pytest -Werror runs with no issue. I have many tests that need the warning filter, so adding decorate to each of them is really not ideal. Is it possible to control the filtering with pytest.ini?

pip list

$ pip list
Package                         Version             Location
------------------------------- ------------------- --------------------------------------------------------------------------------
absl-py                         0.12.0
aiohttp                         3.7.3
aiohttp-cors                    0.7.0
aioredis                        1.3.1
alembic                         1.5.4
antlr4-python3-runtime          4.8
appdirs                         1.4.3
argcomplete                     1.11.1
async-timeout                   3.0.1
attrs                           20.3.0
ax-platform                     0.1.20
bandit                          1.7.0
black                           20.8b1
bleach                          3.2.1
blessings                       1.7
boto3                           1.17.17
botocore                        1.20.17
botorch                         0.4.0
cachetools                      4.2.0
certifi                         2020.12.5
cfgv                            3.2.0
chardet                         3.0.4
click                           7.1.2
cliff                           3.7.0
cloudpickle                     1.6.0
cmaes                           0.8.1
cmd2                            1.5.0
colorama                        0.4.4
colorful                        0.5.4
colorlog                        4.1.0
coverage                        5.3.1
distlib                         0.3.0
docutils                        0.16
filelock                        3.0.12
flake8                          3.8.4
flake8-copyright                0.2.2
fsspec                          0.8.7
future                          0.18.2
gitdb                           4.0.5
GitPython                       3.1.12
google-api-core                 1.25.0
google-auth                     1.24.0
google-auth-oauthlib            0.4.3
googleapis-common-protos        1.52.0
gpustat                         0.6.0
gpytorch                        1.4.0
grpcio                          1.34.1
hiredis                         1.1.0
identify                        1.5.13
idna                            2.10
importlib-resources             5.1.0
incremental                     17.5.0
iniconfig                       1.1.1
isort                           5.5.2
Jinja2                          2.11.2
jmespath                        0.10.0
joblib                          1.0.0
jsonschema                      3.2.0
keyring                         21.8.0
Mako                            1.1.4
Markdown                        3.3.4
MarkupSafe                      1.1.1
mccabe                          0.6.1
mkl-fft                         1.3.0
mkl-random                      1.1.1
mkl-service                     2.3.0
msgpack                         1.0.2
multidict                       5.1.0
mypy                            0.790
mypy-extensions                 0.4.3
nodeenv                         1.5.0
nox                             2020.12.31
numpy                           1.19.5
nvidia-ml-py3                   7.352.0
oauthlib                        3.1.0
olefile                         0.46
omegaconf                       2.1.0.dev24
opencensus                      0.7.12
opencensus-context              0.1.2
optuna                          2.4.0
packaging                       20.8
pandas                          1.2.2
pathspec                        0.8.1
pbr                             5.5.1
pickle5                         0.0.11
Pillow                          8.1.0
pip                             21.0.1
pkginfo                         1.7.0
plotly                          4.14.3
pluggy                          0.13.1
pre-commit                      2.9.3
prettytable                     2.0.0
prometheus-client               0.9.0
protobuf                        3.14.0
psutil                          5.8.0
py                              1.10.0
py-spy                          0.3.4
pyasn1                          0.4.8
pyasn1-modules                  0.2.8
pycodestyle                     2.6.0
pyflakes                        2.2.0
Pygments                        2.7.4
pyparsing                       2.4.7
pyperclip                       1.8.1
pyrsistent                      0.17.3
pytest                          6.2.3
pytest-snail                    0.1.0
python-dateutil                 2.8.1
python-editor                   1.0.4
pytorch-lightning               1.2.3
pytz                            2020.5
PyYAML                          5.3.1
ray                             1.2.0
readme-renderer                 28.0
redis                           3.5.3
regex                           2020.11.13
requests                        2.25.1
requests-oauthlib               1.3.0
requests-toolbelt               0.9.1
retrying                        1.3.3
rfc3986                         1.4.0
rsa                             4.7
s3transfer                      0.3.4
scikit-learn                    0.24.1
scipy                           1.6.0
setuptools                      52.0.0.post20210125
six                             1.14.0
smmap                           3.0.4
SQLAlchemy                      1.3.23
stevedore                       3.3.0
submitit                        1.1.5
tensorboard                     2.4.1
tensorboard-plugin-wit          1.8.0
threadpoolctl                   2.1.0
toml                            0.10.2
torch                           1.7.1
torchaudio                      0.7.0a0+a853dff
torchvision                     0.8.2
towncrier                       19.2.0
tqdm                            4.56.0
twine                           3.3.0
typed-ast                       1.4.2
typeguard                       2.11.1
typing-extensions               3.7.4.3
urllib3                         1.25.11
virtualenv                      20.0.15
wcwidth                         0.2.5
webencodings                    0.5.1
Werkzeug                        1.0.1
wheel                           0.36.2
yamllint                        1.25.0
yarl                            1.6.3

pytest and operating system versions

versions are in the example output, I'm on MAC OS catalina 10.15

The-Compiler commented 3 years ago

Why do you need -Werror at all though, if you have filterwarnings = error in your config?

jieru-hu commented 3 years ago

Thanks for taking a look.

Having -Werror for all pytest is how the CI is setup for my project - I also tried removing error from pytest.ini, didn't help either.

xtofl commented 1 year ago

Same problem here, I can reproduce it very easily.

# pytest.ini 
[pytest]
filterwarnings =
    ignore::FutureWarning
# test_with_warning.py 
import warnings
from datetime import datetime

def foo():
    warnings.warn(message="this won't work after 2030", category=FutureWarning)
    if datetime.now() > datetime(year=2030, month=1, day=1):
        raise RuntimeError

import pytest
def test_foo_rely_on_pytest_ini():
    foo()
@pytest.mark.filterwarnings("ignore::FutureWarning")
def test_foo_marked():
    foo()
/tmp/tmp.oi4oeAN6If$ pytest -W error .
======================================== test session starts ========================================
platform linux -- Python 3.8.10, pytest-7.4.0, pluggy-1.3.0
rootdir: /tmp/tmp.oi4oeAN6If
configfile: pytest.ini
collected 2 items                                                                                   

test_with_warning.py F.                                                                       [100%]

============================================= FAILURES ==============================================
____________________________________ test_foo_rely_on_pytest_ini ____________________________________

    def test_foo_rely_on_pytest_ini():
>       foo()

test_with_warning.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def foo():
>       warnings.warn(message="this won't work after 2030", category=FutureWarning)
E       FutureWarning: this won't work after 2030

test_with_warning.py:5: FutureWarning
====================================== short test summary info ======================================
FAILED test_with_warning.py::test_foo_rely_on_pytest_ini - FutureWarning: this won't work after 2030
==================================== 1 failed, 1 passed in 0.02s ====================================
$ pip freeze
exceptiongroup==1.1.3
iniconfig==2.0.0
packaging==23.1
pluggy==1.3.0
pytest==7.4.0
tomli==2.0.1

-W error is great to find unexpected issues (e.g. the np.bool deprecation warning), so this is an important feature.