2.13.3: test_negative fails with "Error messages changed"

mtelka commented 1 year ago

Things to check first

[X] I have searched the existing issues and didn't find my bug already reported there
[X] I have checked that my bug is still present in the latest release

Typeguard version

2.13.3

Python version

3.9.16

What happened?

While packaging typeguard for OpenIndiana I ran tests and one test failed:

________________________________ test_negative _________________________________
$(BUILD_DIR)/tests/mypy/test_type_annotations.py:114: in test_negative
    raise RuntimeError("Error messages changed")
E   RuntimeError: Error messages changed
----------------------------- Captured stdout call -----------------------------
Line 11
        Expected: error: Incompatible return value type (got "str", expected "int")
        Got:      error: Incompatible return value type (got "str", expected "int")  [return-value]
Line 16
        Expected: error: Incompatible return value type (got "str", expected "int")
        Got:      error: Incompatible return value type (got "str", expected "int")  [return-value]
Line 20
        Expected: error: Incompatible return value type (got "int", expected "str")
        Got:      error: Incompatible return value type (got "int", expected "str")  [return-value]
Line 24
        Expected: error: Incompatible return value type (got "bool", expected "str")
        Got:      error: Incompatible return value type (got "bool", expected "str")  [return-value]
Line 28
        Expected: error: Incompatible return value type (got "bool", expected "str")
        Got:      error: Incompatible return value type (got "bool", expected "str")  [return-value]
Line 31
        Expected: error: Argument 1 to "arg_type" has incompatible type "Callable[[int], int]"; expected "int"
        Got:      error: Argument 1 to "arg_type" has incompatible type "Callable[[int], int]"; expected "int"  [arg-type]
Line 32
        Expected: error: Argument 1 to "foo" has incompatible type "str"; expected "int"
        Got:      error: Argument 1 to "foo" has incompatible type "str"; expected "int"  [arg-type]
Line 52
        Expected: error: Argument 1 to "get_value" has incompatible type "str"; expected "MyClass"
        Got:      error: Argument 1 to "get_value" has incompatible type "str"; expected "MyClass"  [arg-type]
Line 53
        Expected: error: Argument 1 to "MyClass" has incompatible type "str"; expected "int"
        Got:      error: Argument 1 to "MyClass" has incompatible type "str"; expected "int"  [arg-type]

How can we reproduce the bug?

Likely by running tests with the following modules installed:

$ pip3.9 freeze
aiosignal==1.3.1
alabaster==0.7.12
ansible==2.9.27
appdirs==1.4.4
argcomplete==2.0.0
argh==0.27.2
asgiref==3.6.0
asn1crypto==1.5.1
astor==0.8.1
astroid==2.13.2
asttokens==2.2.1
async-generator==1.10
async-timeout==4.0.2
atomicwrites==1.4.1
attrs==22.2.0
autocommand==2.2.1
Automat==22.10.0
Babel==2.11.0
backcall==0.2.0
backports-abc==0.5
backports.cached-property==1.0.2
backports.entry-points-selectable==1.2.0
backports.functools-lru-cache==1.6.4
backports.ssl-match-hostname==3.7.0.1
backports.unittest-mock==1.5
barman==3.4.0
bcrypt==3.2.2
beautifulsoup4==4.11.2
black==23.1.0
bleach==6.0.0
blinker==1.5
borgbackup==1.2.3
bottle==0.12.23
build==0.10.0
CacheControl==0.12.11
cached-property==1.5.2
cachetools==5.3.0
calver==2022.6.26
ccsm==0.8.18
certifi==2022.12.7
cffi==1.15.1
cfgv==3.3.1
Chameleon==3.10.2
characteristic==14.3.0
chardet==5.0.0
charset-normalizer==3.0.1
check-manifest==0.49
cheroot==9.0.0
CherryPy==18.8.0
chevron==0.14.0
CJKwrap==2.2
click==8.1.3
cloudpickle==2.2.1
cmarkgfm==2022.10.27
codecov==2.1.12
colorama==0.4.6
colorlog==6.7.0
commonmark==0.9.1
configobj==5.0.8
constantly==15.1.0
contextlib2==21.6.0
covdefaults==2.2.2
coverage==7.0.5
coveralls==3.3.1
crashtest==0.4.1
crmsh==4.0.0
cryptography==3.2.1
cryptography-vectors==39.0.1
cupshelpers==1.0
curio==1.6
Cython==0.29.33
cython-test-exception-raiser==1.0.2
dbus-python==1.2.18
ddt==1.6.0
decorator==5.1.1
defusedxml==0.7.1
diff-cover==7.3.0
dill==0.3.6
distlib==0.3.6
distro==1.8.0
dnspython==2.2.1
docopt==0.6.2
docutils==0.19
dogpile.cache==1.1.8
editables==0.3
elastic-transport==8.4.0
elasticsearch==8.5.3
elementpath==4.0.1
ephemeral-port-reserve==1.1.4
exceptiongroup==1.1.0
execnet==1.9.0
executing==1.2.0
extras==1.0.0
fastjsonschema==2.16.3
fields==5.0.0
filelock==3.9.0
findpython==0.2.2
fixtures==4.0.1
flake8==5.0.4
flake8-2020==1.7.0
flake8-noqa==1.3.0
flake8-typing-imports==1.14.0
flaky==3.7.0
flamegraph==0.1
flit_core==3.8.0
flit_scm==1.7.0
flufl.flake8==0.10
freezegun==1.2.2
frozendict==2.3.5
frozenlist==1.3.3
fs==2.4.16
func-timeout==4.3.5
future==0.18.2
ghp-import==2.1.0
gi-docgen==2023.1
gprof2dot==2022.7.29
graphviz==0.20.1
greenlet==2.0.2
hatch-fancy-pypi-readme==22.8.0
hatch-nodejs-version==0.3.1
hatch-vcs==0.3.0
hatchling==1.13.0
http-parser==0.9.0
hupper==1.11
hyperlink==21.0.0
hypothesis==6.68.2
icecream==2.1.3
identify==2.5.9
idna==3.4
imagesize==1.4.1
importlib-metadata==6.0.0
importlib-resources==5.12.0
incremental==22.10.0
inflect==6.0.2
iniconfig==2.0.0
installer==0.6.0
iocapture==0.1.2
ipython==7.34.0
iso8601==1.1.0
isort==5.12.0
itsdangerous==2.1.2
jaraco.classes==3.2.3
jaraco.collections==3.8.0
jaraco.context==4.2.0
jaraco.envs==2.4.0
jaraco.functools==3.5.2
jaraco.itertools==6.2.1
jaraco.path==3.4.0
jaraco.test==5.3.0
jaraco.text==3.11.1
jedi==0.18.2
Jinja2==3.0.3
jsonrpclib-pelix==0.4.3.2
jsonschema==4.17.3
jupyter_core==5.1.1
keyring==23.13.1
keyrings.alt==4.2.0
kgb==7.1.1
lazy==1.5
lazy-object-proxy==1.9.0
lingua==4.15.0
littleutils==0.2.2
louis==3.24.0
lxml==4.9.2
Mako==1.2.4
Markdown==3.4.1
markdown2==2.4.7
MarkupSafe==2.1.2
matplotlib-inline==0.1.6
mccabe==0.7.0
mercurial==5.9.3
mergedeep==1.3.4
meson==1.0.1
mkdocs==1.4.2
mkdocs-bootstrap==1.1
mkdocs-bootswatch==1.1
mock==5.0.0
more-itertools==9.0.0
msgpack==1.0.4
multidict==6.0.4
mypy==1.0.1
mypy-extensions==1.0.0
nbformat==5.7.2
netsnmp-python==1.0a1
nodeenv==1.7.0
nose==1.3.7
notify2==0.3.1
nox==2022.11.21
objgraph==3.5.0
olefile==0.46
outcome==1.2.0
packaging==23.0
parameterized==0.8.1
paramiko==2.12.0
parso==0.8.3
Paste==3.5.2
PasteDeploy==3.0.1
path==16.6.0
path.py==12.5.0
pathlib2==2.3.7.post1
pathspec==0.11.0
pbr==5.11.0
pdm-pep517==1.0.6
pep440==0.1.2
pep517==0.13.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.4.0
pip-run==10.0.5
pipdeptree==2.3.3
pkg==0.1
pkgconfig==1.5.5
pkginfo==1.8.3
pkgutil_resolve_name==1.3.10
plaster==1.1.2
plaster-pastedeploy==1.0.1
platformdirs==3.0.0
pluggy==1.0.0
ply==3.11
poetry-core==1.5.0
polib==1.1.1
portend==3.1.0
powerline-status==2.8.2
pre-commit==2.21.0
pretend==1.0.9
prettytable==3.6.0
process-tests==2.1.2
prompt-toolkit==3.0.36
psutil==5.9.0
psycopg2==2.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
py==1.11.0
py-cpuinfo==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.9.2
pybonjour==1.1.1
pycairo==1.23.0
pycmd==1.2
pycodestyle==2.10.0
PyContracts==2.0.1
pycparser==2.21
pycryptodome-test-vectors==1.0.12
pycups==2.0.1
pycurl==7.44.1
pydantic==1.10.2
pydiffx==1.1
pydocstyle==6.1.1
pyfakefs==5.0.0
pyflakes==3.0.1
pyftpdlib==1.5.7
Pygments==2.13.0
PyGObject==3.42.0
PyHamcrest==2.0.4
pylint==2.12.2
PyNaCl==1.5.0
pyOpenSSL==19.0.0
pyparsing==3.0.9
pyproject-installer==0.4.0
pyproject-metadata==0.7.1
pyproject_api==1.5.0
pyproject_fmt==0.4.1
pyproject_hooks==1.0.0
PyQt-builder==1.12.2
PyQt5==5.15.6
PyQt5-sip==12.10.1
Pyro4==4.82
pyroma==4.1
pyrsistent==0.19.3
PySocks==1.7.1
pytest==7.2.1
pytest-asyncio==0.20.3
pytest-benchmark==4.0.0
pytest-black==0.3.12
pytest-black-multipy==1.0.1
pytest-checkdocs==2.9.0
pytest-console-scripts==1.3.1
pytest-cov==4.0.0
pytest-datadir==1.4.1
pytest-enabler==2.0.0
pytest-env==0.8.1
pytest-expect==1.1.0
pytest-fixture-config==1.7.0
pytest-flake8==1.1.1
pytest-forked==1.6.0
pytest-freezegun==0.4.2
pytest-lazy-fixture==0.6.3
pytest-mock==3.7.0
pytest-mypy==0.10.3
pytest-mypy-plugins==1.10.1
pytest-perf==0.12.0
pytest-randomly==3.12.0
pytest-reporter==0.5.2
pytest-socket==0.6.0
pytest-subtests==0.9.0
pytest-timeout==2.0.2
pytest-travis-fold==1.3.0
pytest-xdist==3.1.0
python-dateutil==2.8.2
python-dotenv==0.21.1
python-magic==0.4.27
python-rapidjson==1.9
python-xlib==0.33
pytz==2022.7.1
pyxdg==0.28
PyYAML==6.0
pyyaml_env_tag==0.1
pyzmq==25.0.0
qualname==0.1.0
RBTools==4.0
re-assert==1.1.0
readme-renderer==37.3
readthedocs-sphinx-ext==2.2.0
redis==3.5.3
regex==2022.10.31
repoze.lru==0.7
repoze.sphinx.autointerface==1.0.0
requests==2.28.2
requests-toolbelt==0.10.1
resolvelib==0.9.0
rfc3986==2.0.0
rich==12.6.0
rrdtool==0.1.10
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.7
schema==0.7.5
SCons==4.3.0
semantic-version==2.10.0
serpent==1.41
service-identity==21.1.0
setuptools-rust==1.5.2
setuptools-scm==7.1.0
setuptools-scm-git-archive==1.4
sh==1.14.3
shellingham==1.5.0
simplegeneric==0.8.1
simplejson==3.18.2
singledispatch==4.0.0
sip==6.6.1
six==1.16.0
smartypants==2.0.1
smmap==5.0.0
sniffio==1.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
Sphinx==5.3.0
sphinx-issues==3.0.1
sphinx-rtd-theme==1.1.1
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sqlparse==0.4.3
stevedore==5.0.0
strictyaml==1.6.2
tabulate==0.9.0
teamcity-messages==1.32
tempora==5.2.1
termcolor==2.1.1
terminator==2.1.1
termstyle==0.1.11
testpath==0.6.0
testresources==2.0.1
testscenarios==0.5.0
testtools==2.5.0
texttable==1.6.7
time-machine==2.9.0
tokenize-rt==5.0.0
toml==0.10.2
tomli==2.0.1
tomli_w==1.0.0
tomlkit==0.11.6
tornado==6.2
tox==4.4.6
tox-current-env==0.0.11
tox-gh-actions==3.0.0
tqdm==4.64.1
traitlets==5.5.0
translationstring==1.4
trove-classifiers==2022.12.1
trustme==0.9.0
twine==4.0.2
Twisted==22.10.0
typed-ast==1.5.4
types-docutils==0.19.1.1
types-freezegun==1.1.10
types-mock==4.0.15.2
types-psutil==5.9.5.5
types-pytz==2022.6.0.1
types-setuptools==65.7.0.3
types-toml==0.10.8.1
types-typed-ast==1.5.8.3
typing_extensions==4.5.0
typogrify==2.0.7
tzdata==2022.7
u-msgpack-python==2.7.2
Unidecode==1.3.6
urllib3==1.26.14
validate-pyproject==0.10.1
validators==0.20.0
venusian==3.0.0
virtualenv==20.19.0
voluptuous==0.13.1
waitress==2.1.2
watchdog==2.2.1
wcag-contrast-ratio==0.9
wcwidth==0.2.6
webencodings==0.5.1
wrapt==1.14.1
xmlschema==1.9.2
youtube-dl==2021.1.8
zc.lockfile==2.0
zest.releaser==7.3.0
zipp==3.14.0
zope.deprecation==4.4.0
zope.event==4.5.0
zope.exceptions==4.6
zope.interface==5.5.2
zope.testing==5.0.1
zope.testrunner==5.6
zstd==1.5.2.6
$

agronholm commented 1 year ago

Probably due to mypy upgrades. I am actively working on v3.0 which does pass these tests. What you're seeing is not a bug per se, so unless something else is failing too, it's not indicative of a bug. If you're not in a hurry to package typeguard, then you may want to wait until v3.0 final. I expect that to be released within a few weeks.

mtelka commented 1 year ago

@agronholm, thanks for explanation.

agronholm commented 1 year ago

I've just released v3.0.0rc1. It contains a lot of changes from the previous pre-release version. Let me know if you experience test failures with it.

mtelka commented 1 year ago

I just tested 3.0.0rc1 and test_negative passed.

agronholm / typeguard