nedbat / coveragepy

The code coverage tool for Python
https://coverage.readthedocs.io
Apache License 2.0
2.89k stars 419 forks source link

PyPy3.9/3.8 + Django 4.1 + Coverage Failure with LazyObject #1382

Open DylanYoung opened 2 years ago

DylanYoung commented 2 years ago

Describe the bug Odd failure on PyPy, but not CPython. See PyPy bug for details: https://foss.heptapod.net/pypy/pypy/-/issues/3751

Traceback (most recent call last):
  File "django-csp/.tox/pypy3.9-main/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 187, in console_main
    code = main()
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 145, in main
    config = _prepareconfig(args, plugins)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 324, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 1016, in pytest_cmdline_parse
    self.parse(args)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 1304, in parse
    self._preparse(args, addopts=addopts)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 1206, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pytest_django/plugin.py", line 353, in pytest_load_initial_conftests
    _setup_django()
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/pytest_django/plugin.py", line 236, in _setup_django
    django.setup()
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/django/__init__.py", line 25, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/django/apps/registry.py", line 124, in populate
    app_config.ready()
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/django/contrib/admin/apps.py", line 27, in ready
    self.module.autodiscover()
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/django/contrib/admin/__init__.py", line 50, in autodiscover
    autodiscover_modules("admin", register_to=site)
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/django/utils/module_loading.py", line 58, in autodiscover_modules
    import_module("%s.%s" % (app_config.name, module_to_search))
  File ".pyenv/versions/pypy3.9-7.3.9/lib/pypy3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<builtin>/frozen importlib._bootstrap_external", line 863, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/django/contrib/auth/admin.py", line 29, in <module>
    class GroupAdmin(admin.ModelAdmin):
  File "django-csp/.tox/pypy3.9-main/lib/pypy3.9/site-packages/django/contrib/admin/decorators.py", line 102, in _model_admin_wrapper
    raise ValueError("site must subclass AdminSite")
ValueError: site must subclass AdminSite

To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

  1. What version of Python are you using?
Python 3.9.12 (05fbe3aa5b0845e6c37239768aa455451aa5faba, Mar 29 2022, 09:54:47)
[PyPy 7.3.9 with GCC Apple LLVM 13.0.0 (clang-1300.0.29.30)]

Also fails on 3.8.

  1. What version of coverage.py shows the problem? The output of coverage debug sys is helpful.

Output redacted for sensitive info.

-- sys -------------------------------------------------------
               coverage_version: 6.4
                         tracer: -none-
                        CTracer: unavailable
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
                    config_file: None
                config_contents: -none-
                      data_file: -none-
                         python: 3.9.12 (05fbe3aa5b0845e6c37239768aa455451aa5faba, Mar 29 2022, 09:54:47)[PyPy 7.3.9 with GCC Apple LLVM 13.0.0 (clang-1300.0.29.30)]
                       platform: macOS-10.14.6-x86_64-i386-64bit
                 implementation: PyPy
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 38110
                   command_line: coverage debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.24.0
             sqlite3_temp_store: 0
        sqlite3_compile_options: BUG_COMPATIBLE_20160819, COMPILER=clang-10.0.1, DEFAULT_CACHE_SIZE=2000,
                                 DEFAULT_CKPTFULLFSYNC, DEFAULT_JOURNAL_SIZE_LIMIT=32768,
                                 DEFAULT_PAGE_SIZE=4096, DEFAULT_SYNCHRONOUS=2, DEFAULT_WAL_SYNCHRONOUS=1,
                                 ENABLE_API_ARMOR, ENABLE_COLUMN_METADATA, ENABLE_DBSTAT_VTAB, ENABLE_FTS3,
                                 ENABLE_FTS3_PARENTHESIS, ENABLE_FTS3_TOKENIZER, ENABLE_FTS4, ENABLE_FTS5,
                                 ENABLE_JSON1, ENABLE_LOCKING_STYLE=1, ENABLE_PREUPDATE_HOOK, ENABLE_RTREE,
                                 ENABLE_SESSION, ENABLE_SNAPSHOT, ENABLE_SQLLOG,
                                 ENABLE_UNKNOWN_SQL_FUNCTION, ENABLE_UPDATE_DELETE_LIMIT,
                                 HAS_CODEC_RESTRICTED, HAVE_ISNAN, MAX_LENGTH=2147483645,
                                 MAX_MMAP_SIZE=1073741824, MAX_VARIABLE_NUMBER=500000, OMIT_AUTORESET,
                                 OMIT_LOAD_EXTENSION, STMTJRNL_SPILL=131072, THREADSAFE=2, USE_URI
  1. What versions of what packages do you have installed? The output of pip freeze is helpful.

Also fails on Django 4.1a1

asgiref==3.5.2
attrs==21.4.0
cffi==1.15.0
coverage==6.4
Django @ https://github.com/django/django/archive/main.tar.gz
greenlet==0.4.13
hpy==0.0.3
iniconfig==1.1.1
Jinja2==3.1.2
MarkupSafe==2.1.1
mock==1.0.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pycodestyle==2.8.0
pyflakes==2.4.0
pyparsing==3.0.9
pytest==7.1.2
pytest-cov==3.0.0
pytest-django==4.5.2
pytest-flakes==4.0.5
pytest-pycodestyle==2.2.1
readline==6.2.4.1
sqlparse==0.4.2
tomli==2.0.1
  1. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.

See linked PyPy issue: https://foss.heptapod.net/pypy/pypy/-/issues/3751 And related Django thread: https://code.djangoproject.com/ticket/28358#comment:13 And Django commit that introduced the issue: https://github.com/django/django/commit/97d7990abde3fe4b525ae83958fd0b52d6a1d13f

  1. What commands did you run?
pytest --cov=./csp ./csp
coverage run -m pytest ./csp

On this branch: https://github.com/DylanYoung/django-csp/tree/GH-36

Both exhibit the problem. No error when coverage is not used.

Expected behavior

No error.

Additional Info

See this comment for how to repro: https://foss.heptapod.net/pypy/pypy/-/issues/3751#note_185362

nedbat commented 1 year ago

I can reproduce this with:

git clone https://github.com/DylanYoung/django-csp
cd django-csp
git checkout GH-36
pip install tox
tox -e pypy3.9-main

But there is a lot of stuff going on here that I don't understand. I don't see why coverage would affect things, or why PyPy would behave differently, but they do.

nedbat commented 1 year ago

A clue: Python 3.9 also fails if you use the Python tracer instead of the C tracer (the --timid option).

ionelmc commented 1 year ago

A simpler reproducer:

class Foo:
    pass

from django.utils.functional import LazyObject

class LazyFoo(LazyObject):
    def _setup(self):
        self._wrapped = Foo()

assert isinstance(LazyFoo(), Foo)

Assertion fails with coverage on.

ionelmc commented 1 year ago

So I've tried this and it happens with hunter too 🙃 This fails:

import hunter
hunter.trace()
from django.utils.functional import LazyObject
class Foo:
    pass
class LazyFoo(LazyObject):
    def _setup(self):
        self._wrapped = Foo()
assert isinstance(LazyFoo(), Foo)

This works:

import hunter
from django.utils.functional import LazyObject
hunter.trace()
class Foo:
    pass
class LazyFoo(LazyObject):
    def _setup(self):
        self._wrapped = Foo()
assert isinstance(LazyFoo(), Foo)

I believe something goes wrong in the LazyObject class definition when it's traced.

ionelmc commented 1 year ago

Ok this is the minimal reproducer (seems to fail both for pypy and cpython):

import sys

def tracer(frame, event, _):
    return tracer

sys.settrace(tracer)

class Foo:
    pass

class FancyFoo:
    def bug(self):
        super()

    @property
    def __class__(self):
        return Foo

assert isinstance(FancyFoo(), Foo)
LilyFoote commented 2 months ago

The PyPy issue has moved to https://github.com/pypy/pypy/issues/3750.