Incorrect introspection for MagicMock.assert_has_calls

progval commented 3 years ago

Hi,

I wrote a test that calls a MagicMock() with a list of bytes, then uses assert_has_calls with a different argument (list of strings).

Here is pytest's output:

    def test_foo():
        from unittest.mock import MagicMock, call

        f = MagicMock()

        f("foo", [b"bar"])
        f("foo", [b"baz"])

        f.assert_has_calls(
            [
                call("foo", ["bar"]),
                call("foo", ["baz"]),
            ],
>           any_order=True
        )
E       AssertionError: (call('foo', ['bar']), call('foo', ['baz'])) not all found in call list
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('foo', [b'baz']) == ([call('foo', ['bar']), call('foo', ['baz'])],)
E         At index 0 diff: 'foo' != [call('foo', ['bar']), call('foo', ['baz'])]
E         Left contains one more item: [b'baz']
E         Full diff:
E         - ([call('foo', ['bar']), call('foo', ['baz'])],)
E         + ('foo', [b'baz'])
E       Kwargs:
E       assert {} == {'any_order': True}
E         Right contains 1 more item:
E         {'any_order': True}
E         Full diff:
E         - {'any_order': True}
E         + {}

The first line of the error(AssertionError) is correct, but the other ones aren't:

('foo', [b'baz']) == ([call('foo', ['bar']), call('foo', ['baz'])],) is not exactly what was asserted; instead the assertion was that the left side is inside the right side's first element
assert {} == {'any_order': True} this is not part of the assertion, any_order is just an argument of the function

Instead, I would probably expect something like this:

E       AssertionError: (call('foo', ['bar']), call('foo', ['baz'])) not all found in call list
E       
E       pytest introspection follows:
E       
E       assert call('foo', [b'baz']) in [call('foo', ['bar']), call('foo', ['baz'])]

OS: Debian 10

Python version: 3.7.3

pytest version: 6.2.2

pip list

``` Package Version Location ----------------------------- ------------------------------- --------------------------------------------------------------------------- aiohttp 3.5.1 aiohttp-utils 3.1.1 aiosqlite 0.16.0 alabaster 0.7.12 amqp 2.5.2 ansible 2.7.7 anyjson 0.3.3 apache-libcloud 2.4.0 apipkg 1.5 appdirs 1.4.3 argcomplete 1.8.1 arrow 0.12.1 asgiref 3.2.10 asn1crypto 0.24.0 aspy.yaml 1.3.0 async-generator 1.10 async-timeout 3.0.1 atomicwrites 1.1.5 attrs 19.3.0 attrs-strict 0.1.0 azure-common 1.1.24 azure-nspkg 3.0.2 azure-storage 0.36.0 Babel 2.7.0 bcrypt 3.1.6 beautifulsoup4 4.8.1 billiard 3.6.3.0 binwalk 2.1.2 black 19.10b0 bleach 3.1.2 blessed 1.15.0 blinker 1.4 borgbackup 1.1.9 Brlapi 0.6.7 Brotli 1.0.7 cached-property 1.5.2 cachetools 4.2.0 cassandra-driver 3.20.1 celery 4.4.0 certifi 2018.8.24 cffi 1.14.3 cfgv 2.0.1 chardet 3.0.4 click 7.1.2 colorama 0.3.7 commonmark 0.9.1 confluent-kafka 1.5.0 coreapi 2.3.3 coreschema 0.0.4 coverage 5.0 cryptography 2.6.1 cupshelpers 1.0 cycler 0.10.0 daiquiri 2.1.1 dash 1.12.0 dash-bootstrap-components 0.10.2 dash-core-components 1.10.0 dash-html-components 1.0.3 dash-renderer 1.4.1 dash-table 4.7.0 decorator 4.3.0 defusedxml 0.5.0 Deprecated 1.2.6 devscripts 2.19.5+deb10u1 diffoscope 113 distro 1.5.0 distro-info 0.21 Django 2.2.15 django-cors-headers 3.2.0 django-extensions 2.2.5 django-js-reverse 0.9.1 django-stubs 1.7.0 django-test-migrations 1.1.0 django-webpack-loader 0.6.0 djangorestframework 3.10.3 djangorestframework-stubs 1.4.0 docker 4.3.1 docker-compose 1.27.4 dockerpty 0.4.1 docopt 0.6.2 docutils 0.15.2 dulwich 0.19.13 ecdsa 0.15 EditorConfig 0.12.1 elasticsearch 7.0.5 elementpath 2.0.4 enjarify 1.0.3 entrypoints 0.3 ephem 3.7.6.0 execnet 1.7.1 extras 1.0.0 fastimport 0.9.8 feedparser 5.2.1 filelock 3.0.10 fixtures 3.0.0 flake8 3.8.3 Flask 1.1.1 Flask-Compress 1.5.0 fossology 0.2.0 frozendict 1.2 future 0.18.2 gbp 0.9.14 geomet 0.1.2 glob2 0.7 gpg 1.12.0 gunicorn 19.9.0 html5lib 1.0.1 htmlmin 0.1.12 httplib2 0.11.3 humanize 3.2.0 hypothesis 5.49.0 identify 1.4.7 idna 2.6 imagesize 1.1.0 importlib-metadata 0.23 iniconfig 1.0.1 iotop 0.6 ipykernel 4.9.0 ipython 5.8.0 ipython-genutils 0.2.0 ipywidgets 6.0.0 iso8601 0.1.12 isodate 0.6.0 isort 5.5.3 itsdangerous 0.24 itypes 1.2.0 Jinja2 2.10.3 jmespath 0.9.4 jsbeautifier 1.6.4 jsondiff 1.1.1 jsonschema 2.6.0 junit-xml 1.8 jupyter-client 5.2.3 jupyter-core 4.4.0 kafka-python 2.0.0 keyring 17.1.1 keyrings.alt 3.1.1 kiwisolver 1.0.1 kombu 4.6.7 launchpadlib 1.10.13 lazr.restfulclient 0.14.3 lazr.uri 1.0.3 libarchive-c 2.8 linecache2 1.0.0 llfuse 1.3.6 lockfile 0.12.2 louis 3.8.0 lxml 4.4.1 markdown-it-py 0.5.6 MarkupSafe 1.1.0 matplotlib 3.0.2 mccabe 0.6.1 mercurial 5.7 meson 0.49.2 mirakuru 2.3.0 mistune 0.8.4 more-itertools 4.2.0 msgpack 1.0.0 multidict 4.5.2 musicbrainzngs 0.6 mutagen 1.40.0 mutmut 2.0.0 mypy 0.800 mypy-extensions 0.4.3 myst-parser 0.12.10 nagiosplugin 1.3.2 nbconvert 5.4.0 nbformat 4.4.0 ndjson 0.3.1 netaddr 0.7.19 nodeenv 1.3.3 nose 1.3.7 notebook 5.7.8 ntlm-auth 1.1.0 numexpr 2.6.9 numpy 1.16.2 oauthlib 3.1.0 olefile 0.46 outcome 1.1.0 packaging 19.2 pandas 0.23.3+dfsg pandocfilters 1.4.2 paramiko 2.4.2 parso 0.7.0 pathspec 0.7.0 patool 1.12 pbr 5.4.5 pexpect 4.6.0 pg-activity 2.1.0 pg8000 1.13.2 pickleshare 0.7.5 pifpaf 3.0.0 pika 1.1.0 Pillow 5.4.1 pip 21.0.1 pkginfo 1.5.0.1 plotille 3.7.2 plotly 4.5.4 pluggy 0.13.1 pony 0.7.13 port-for 0.4 powerline-status 2.7 pre-commit 1.20.0 pristine-zip 0.1.dev73+gefd1d06.d20200805 /home/dev/pristine_zip progressbar 2.5 prometheus-client 0.6.0 prompt-toolkit 1.0.15 psutil 5.5.1 psycopg2 2.8.6 py 1.9.0 py4j 0.10.8.1 pyasn1 0.4.2 pybadges 2.2.1 pybloomfiltermmap3 0.5.3 pycairo 1.16.2 pycodestyle 2.6.0 pycparser 2.20 pycrypto 2.6.1 pycups 1.9.73 pycurl 7.43.0.2 pyflakes 2.2.0 pyfuse3 3.1.1 Pygments 2.3.1 PyGObject 3.30.4 pyinotify 0.9.6 pykerberos 1.1.14 PyLD 2.0.3 PyNaCl 1.3.0 PyOpenGL 3.1.0 pyOpenSSL 19.0.0 pypandoc 1.4 pyparsing 2.2.0 PyPDF2 1.26.0 pyqtgraph 0.10.0 PySimpleSOAP 1.16.2 pysmbc 1.0.15.6 pytest 6.2.2 pytest-cov 2.10.0 pytest-cover 3.0.0 pytest-coverage 0.0 pytest-django 3.7.0 pytest-forked 1.1.3 pytest-kafka 0.4.0 pytest-mock 1.13.0 pytest-parallel 0.0.10 pytest-postgresql 2.5.3 pytest-xdist 1.31.0 python-apt 1.8.4.3 python-cephlibs 0.94.5.post1 python-daemon 2.2.4 python-dateutil 2.8.1 python-debian 0.1.35 python-debianbts 2.8.2 python-dotenv 0.15.0 python-hglib 2.6.1 python-jose 3.1.0 python-json-logger 0.1.11 python-keycloak 0.20.0 python-magic 0.4.16 python-memcached 1.58 python-mimeparse 1.6.0 pytz 2019.1 pywinrm 0.3.0 pyxattr 0.6.1 pyxdg 0.25 PyYAML 3.13 pyzmq 17.1.2 quodlibet 4.2.1 rados 2.0.0 rdflib 5.0.0 rdflib-jsonld 0.5.0 recommonmark 0.6.0 redis 3.3.11 regex 2020.2.20 reportbug 7.5.3-deb10u1 reportlab 3.5.13 requests 2.25.1 requests-ftp 0.3.1 requests-kerberos 0.11.0 requests-mock 1.8.0 requests-ntlm 1.1.0 retrying 1.3.3 rsa 4.0 scipy 1.1.0 scour 0.37 scramp 1.1.0 SecretStorage 2.3.1 Send2Trash 1.5.0 sentry-sdk 0.13.5 setuptools 40.8.0 simplegeneric 0.8.1 simplejson 3.16.0 simpy 3.0.13 six 1.12.0 sniffio 1.2.0 snowballstemmer 2.0.0 sortedcontainers 2.1.0 soupsieve 1.9.5 Sphinx 3.3.1 sphinx-click 2.5.0 sphinx-rtd-theme 0.5.0 sphinx-tabs 1.3.0 sphinxcontrib-applehelp 1.0.1 sphinxcontrib-devhelp 1.0.1 sphinxcontrib-htmlhelp 1.0.2 sphinxcontrib-httpdomain 1.7.0 sphinxcontrib-images 0.9.2 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-programoutput 0.16 sphinxcontrib-qthelp 1.0.2 sphinxcontrib-serializinghtml 1.1.3 SQLAlchemy 1.3.10 sqlalchemy-stubs 0.2 sqlitedict 1.6.0 sqlparse 0.3.1 subvertpy 0.10.1 swh.clearlydefined 0.1.dev5+g430c67f /home/dev/swh-environment/swh-clearlydefined swh.core 0.12.0 /home/dev/.local/lib/python3.7/site-packages swh.deposit 0.6.1.dev24+g22172d4e /home/dev/swh-environment/swh-deposit swh.docs 0.0.1.dev287+g54fe755.d20210325 /home/dev/swh-environment/swh-docs swh.fuse 0.9.2 /home/dev/.local/lib/python3.7/site-packages swh.graph 0.2.10.dev45+g5a987aa /home/dev/swh-environment/swh-graph swh.icinga-plugins 0.2.2.dev1+g441faf2 /home/dev/swh-environment/swh-icinga-plugins swh.indexer 0.6.4.dev4+g3baf8bb /home/dev/swh-environment/swh-indexer swh.journal 0.7.2.dev3+g458a405.d20210322 /home/dev/swh-environment/swh-journal swh.lister 0.4.1.dev9+gc782275 /home/dev/swh-environment/swh-lister swh.loader.core 0.20.1.dev4+gcf8fc9e /home/dev/swh-environment/swh-loader-core swh.loader.git 0.3.6 /home/dev/swh-environment/swh-loader-git swh.loader.mercurial 0.4.0 /home/dev/swh-environment/swh-loader-mercurial swh.loader.svn 0.0.54.post2 /home/dev/swh-environment/swh-loader-svn swh.mirror.forge 0.0.5.post20 /home/dev/swh-environment/swh-mirror-forge swh.model 2.3.0 /home/dev/.local/lib/python3.7/site-packages/swh.model-2.3.0-py3.7.egg swh.objstorage 0.2.2 /home/dev/.local/lib/python3.7/site-packages/swh.objstorage-0.2.2-py3.7.egg swh.objstorage.replayer 0.2.1 /home/dev/swh-environment/swh-objstorage-replayer swh.scanner 0.3.1.dev0+g33a9cd4.d20210301 /home/dev/swh-environment/swh-scanner swh.scheduler 0.10.0 /home/dev/.local/lib/python3.7/site-packages swh.search 0.3.3.dev1+g4390bea.d20201211 /home/dev/swh-environment/swh-search swh.storage 0.26.0 /home/dev/swh-environment/swh-storage swh.vault 0.0.35.dev3+gd64687d.d20200904 /home/dev/swh-environment/swh-vault swh.web 0.0.284.dev6+g0b974c89 /home/dev/swh-environment/swh-web swh.web.client 0.2.2 /home/dev/.local/lib/python3.7/site-packages systemd-python 234 tables 3.4.4 tblib 1.6.0 tenacity 6.0.0 terminado 0.8.1 testinfra 4.0.0 testing.common.database 2.0.3 testing.postgresql 1.3.0 testpath 0.4.2 testresources 2.0.1 testtools 2.4.0 texttable 1.6.3 tlsh 0.2.0 toml 0.10.2 tornado 5.1.1 tox 3.7.0 traceback2 1.4.0 traitlets 4.3.2 trio 0.17.0 typed-ast 1.4.0 typing-extensions 3.7.4.1 unattended-upgrades 0.1 unidiff 0.5.4 unittest2 1.1.0 uritemplate 3.0.1 urllib3 1.24.1 vcversioner 2.16.0.0 vine 1.3.0 virtualenv 16.7.7 wadllib 1.3.4 wcwidth 0.1.7 webencodings 0.5.1 websocket-client 0.57.0 Werkzeug 0.16.0 wheel 0.32.3 widgetsnbextension 2.0.0 wrapt 1.11.2 xattr 0.9.7 xmlschema 1.3.1 xmltodict 0.11.0 yarl 1.3.0 zipp 0.6.0 ```

Thanks!

asottile commented 3 years ago

'pytest introspection follows" doesn't appear in our codebase, this must be something from a faulty plugin -- appears to be pytest-mock

nicoddemus commented 3 years ago

Hi @ProgVal,

Indeed this is a bug in the introspection done by pytest-mock, thanks for reporting.

The problem seems to be how the internal assert_wrapper works, it can be followed from here:

https://github.com/pytest-dev/pytest-mock/blob/be33132c4dbf7dcd0415b5595b8acb8ae2b0cec4/src/pytest_mock/plugin.py#L442-L444

I probably won't get to this any time soon, but others feel free to give it a shot. 👍

cc @asfaltboy

TommyDew42 commented 2 years ago

Ran into the same error today too! The pytest-mock version is 3.6.1

idow09 commented 1 year ago

Me too! with pytest-mock version 3.10.0

Astral1020 commented 1 year ago

I created this PR #359

pytest-dev / pytest-mock

Incorrect introspection for MagicMock.assert_has_calls #234