Infinite recursion running coverage with django unit tests on circleCI

[lee-hodg]

Describe the bug

I keep getting an infinite recursion exception when running coverage on circleCI, and with unit tests that use python sure

The exception looks like

TOTAL                                                            6423   2386    63%
Traceback (most recent call last):
  File "app/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 55, in handle
    failures = test_runner.run_tests(test_labels)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django_nose/runner.py", line 308, in run_tests
    result = self.run_suite(nose_argv)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/django_nose/runner.py", line 244, in run_suite
    nose.core.TestProgram(argv=nose_argv, exit=False,
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/nose/core.py", line 118, in __init__
    unittest.TestProgram.__init__(
  File "/usr/local/lib/python3.8/unittest/main.py", line 101, in __init__
    self.runTests()
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/nose/core.py", line 207, in runTests
    result = self.testRunner.run(self.test)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/nose/core.py", line 66, in run
    result.printErrors()
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/nose/result.py", line 110, in printErrors
    self.config.plugins.report(self.stream)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/nose/plugins/manager.py", line 99, in __call__
    return self.call(*arg, **kw)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/nose/plugins/manager.py", line 167, in simple
    result = meth(*arg, **kw)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/nose/plugins/cover.py", line 183, in report
    self.coverInstance.stop()
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/coverage/control.py", line 1091, in html_report
    return reporter.report(morfs)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/coverage/html.py", line 128, in report
    m.update(self.config)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/coverage/misc.py", line 228, in update
    self.update(a)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/coverage/misc.py", line 228, in update
    self.update(a)
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/coverage/misc.py", line 228, in update
    self.update(a)
  [Previous line repeated 966 more times]
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/coverage/misc.py", line 221, in update
    for k in dir(v):
  File "/home/circleci/repo/venv/lib/python3.8/site-packages/sure/__init__.py", line 1154, in _new_dir
    import pytest
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1342, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1314, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1466, in find_spec
  File "<frozen importlib._bootstrap_external>", line 64, in _path_join
  File "<frozen importlib._bootstrap_external>", line 64, in <listcomp>
RecursionError: maximum recursion depth exceeded while calling a Python object

Exited with code exit status 1
CircleCI received exit code 1

I've tried multiple versions of coverage from coverage==4.5.4 to 6.1.2 and 6.2 and it's the same (sure==1.4.78 and django-nose==1.4.7).

The command I run is with Django like python app/manage.py test tests --cover-html --cover-html-dir=tests/test-reports and locally (on my laptop virtualenv) it seems to work just fine.

To Reproduce How can we reproduce the problem? Please be specific. Don't just link to a failing CI job. Answer the questions below: . What version of Python are you using?

3.8.5

What version of coverage.py shows the problem? The output of coverage debug sys is helpful. 4.5.4, 6.1.2, 6.2 showed this problem (maybe others...)

What versions of what packages do you have installed? The output of pip freeze is helpful.

apiclient==1.0.4
asgiref==3.4.1; python_full_version >= "3.6.2" and python_full_version < "4.0.0" and python_version >= "3.6"
aws-xray-sdk==0.95
babel==2.8.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
beautifulsoup4==4.10.0; python_full_version > "3.0.0"
blinker==1.4
boto3==1.18.65; python_version >= "3.6"
boto==2.49.0
botocore==1.21.65; python_version >= "3.6"
cachetools==3.1.1
certifi==2020.12.5
cffi==1.14.0
chardet==3.0.4
charset-normalizer==2.0.9; python_full_version >= "3.6.2" and python_version >= "3" and python_full_version < "4.0.0" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.6.0")
coreapi==2.3.3
coreschema==0.0.4
coverage==4.5.4; (python_version >= "2.6" and python_full_version < "3.0.0") or (python_full_version >= "3.3.0" and python_version < "4")
cryptography==36.0.1; python_version >= "3.6"
defusedxml==0.6.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
dj-stripe==2.5.0; python_full_version >= "3.6.2" and python_full_version < "4.0.0"
django-admin-rangefilter==0.8.3; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
django-allauth==0.47.0
django-cors-headers==3.10.1; python_version >= "3.6"
django-countries==7.2.1
django-elasticsearch-dsl-drf==0.22.2; python_version >= "2.7"
django-elasticsearch-dsl==7.2.1
django-filter==2.4.0; python_version >= "3.5"
django-hashid-field==3.3.3
django-jsonfield==1.4.1
django-localflavor==3.1
django-model-utils==4.2.0
django-nine==0.2.5; python_version >= "2.7"
django-nose==1.4.7
django-rest-auth==0.9.5
django-rest-swagger==2.2.0
django-safedelete @ git+https://github.com/MarcelForArt/django-safedelete.git@marcelv2.2
django-storages==1.12.3; python_version >= "3.5"
django-user-agents==0.4.0
django==3.2.10; python_version >= "3.6"
djangorestframework-composed-permissions==0.2.1
djangorestframework==3.13.0; python_version >= "3.6"
docopt==0.6.2; python_version >= "3"
drf-nested-routers==0.93.3; python_version >= "3.5"
drfpasswordless @ git+https://github.com/MarcelForArt/django-rest-framework-passwordless.git@marcelv2.2 ; python_version >= "3"
ecdsa==0.17.0; (python_version >= "2.6" and python_full_version < "3.0.0") or (python_full_version >= "3.3.0")
elasticsearch-dsl==7.4.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "2.7"
elasticsearch==7.16.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0" and python_version < "4")
exifread==2.3.2
factory-boy==3.2.1; python_version >= "3.6"
faker==11.1.0; python_version >= "3.6"
google-api-core==2.3.2; python_version >= "3.6"
google-api-python-client==2.33.0; python_version >= "3.6"
google-auth-httplib2==0.1.0
google-auth==2.3.3; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
googleapis-common-protos==1.54.0; python_version >= "3.6"
googlemaps==3.1.3
hashids==1.3.1; python_version >= "2.7"
httplib2==0.20.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
idna==2.6
itypes==1.1.0
jinja2==2.11.3; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
jmespath==0.9.3
jsonfield2==4.0.0.post0
jsonfield==3.1.0; python_full_version >= "3.6.2" and python_full_version < "4.0.0" and python_version >= "3.6"
jsonpickle==1.2
lxml==4.1.1
mandrill-really-maintained==1.2.4
markupsafe==1.1.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
mixpanel-utils==2.1.0; python_version >= "3"
mixpanel==4.9.0; (python_version >= "2.7" and python_full_version < "3.4.0") or (python_full_version >= "3.5.0")
mock==3.0.5; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
moto==2.3.0
neverbounce-sdk==4.3.0
nose-timer==1.0.1
nose==1.3.7
numpy==1.17.3; python_version >= "3.5"
oauth2client==4.1.3
oauthlib==3.1.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
openapi-codec==1.3.2
package-name==0.1
parameterized==0.6.1
paypal-checkout-serversdk==1.0.1
paypalhttp==1.0.1
paypalrestsdk==1.13.1
pillow==8.4.0; python_version >= "3.6"
protobuf==3.19.1; python_version >= "3.6"
psycopg2==2.9.2; python_version >= "3.6"
pyaml==21.10.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycountry==20.7.3
pycparser==2.21; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
pycryptodome==3.12.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
pyjwt==2.3.0; python_version >= "3.6"
pyopenssl==21.0.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.6.0"
pyparsing==3.0.6; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
pysha3==1.0.2
python-amplitude-export==0.0.2
python-dateutil==2.8.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
python-jose==3.3.0
python-magic==0.4.24; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
python-memcached==1.59
python-mimeparse==1.6.0
python-stdnum==1.17
python3-openid==3.2.0
pytz==2021.3
pyyaml==5.4.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
requests-oauthlib==1.3.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
requests==2.26.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
responses==0.16.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
rsa==4.8; python_version >= "3.6" and python_version < "4"
s3transfer==0.5.0; python_version >= "3.6"
sentry-sdk==1.5.1
simplejson==3.17.6; (python_version >= "2.5" and python_full_version < "3.0.0") or (python_full_version >= "3.3.0")
six==1.14.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
soupsieve==2.3.1; python_version >= "3.6" and python_full_version > "3.0.0"
sqlparse==0.4.2; python_version >= "3.5"
stripe==2.64.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
sure==2.0.0
text-unidecode==1.3
ua-parser==0.10.0
uritemplate==4.1.1; python_version >= "3.6"
urllib3==1.26.7; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0" and python_version < "4")
user-agents==2.2.0
websocket-client==1.2.3; python_version >= "3.6"
werkzeug==2.0.2; python_version >= "3.6"
wrapt==1.13.3; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
xmltodict==0.12.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")

What commands did you run?

Django unit tests on circleCI

python app/manage.py test tests --cover-html --cover-html-dir=tests/test-reports

Expected behavior

A coverage report is generated without hitting an infinite recursion issue.

Additional context

[nedbat]

This sounds weird! Somehow, the coverage configuration object has itself (or some other value) recursively. I have no idea why that would be.

Can you run the tests again with a new branch of coverage.py? I added some debugging to see what the configuration object is like. Install coverage with:

pip install git+https://github.com/nedbat/coveragepy.git@nedbat/debug-1298#egg=coverage==0.0

and run the tests. It should still fail with a recursion error, but we'll have a bit of debug output to look at.

[jaredhobbs]

I'm running into a similar issue. Coverage works fine to run my tests but then running coverage html results in RecursionError: maximum recursion depth exceeded:

root@52d7bb691933:/src# coverage html
Traceback (most recent call last):
  File "/usr/local/bin/coverage", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/coverage/cmdline.py", line 880, in main
    status = CoverageScript().command_line(argv)
  File "/usr/local/lib/python3.10/site-packages/coverage/cmdline.py", line 638, in command_line
    total = self.coverage.html_report(
  File "/usr/local/lib/python3.10/site-packages/coverage/control.py", line 1003, in html_report
    ret = reporter.report(morfs)
  File "/usr/local/lib/python3.10/site-packages/coverage/html.py", line 208, in report
    self.html_file(fr, analysis)
  File "/usr/local/lib/python3.10/site-packages/coverage/html.py", line 272, in html_file
    file_data = self.datagen.data_for_file(fr, analysis)
  File "/usr/local/lib/python3.10/site-packages/coverage/html.py", line 73, in data_for_file
    for lineno, tokens in enumerate(fr.source_token_lines(), start=1):
  File "/usr/local/lib/python3.10/site-packages/coverage/phystokens.py", line 111, in source_token_lines
    match_case_lines = MatchCaseFinder(source).match_case_lines
  File "/usr/local/lib/python3.10/site-packages/coverage/phystokens.py", line 76, in __init__
    self.visit(ast.parse(source))
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 418, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 418, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 418, in generic_visit
...
RecursionError: maximum recursion depth exceeded

This started happening after I updated to Python 3.10.1 from 3.8.

[nedbat]

@jaredhobbs that looks like a different stack trace. Would you mind opening a new issue, with as many detailed steps as you can give me to reproduce the issue?

[nedbat]

@lee-hodg @jaredhobbs Can you provide any more information?

[jaredhobbs]

I wasn't able to create a minimal example unfortunately. I was, however, able to get it working by increasing the recursion limit to 1500.

[nedbat]

@jaredhobbs is your code public? Or would you be able to add a bit of debugging to the coverage.py source to find the source file that caused it? I would really like to understand what happened. Also, I am very surprised that increasing the limit from 1000 to 1500 made it work.

[jaredhobbs]

It's not public, sorry. The code includes a bunch of very large files that were auto-generated from xsd schema files. I imagine the problematic source file is one of those.

That being said, I'd be happy to add some debugging to coverage.py to find the source that created it! Let me know how to do that!

[nedbat]

Thanks for persisting with me! :)

Do you have the end of the RecursionError stack trace? I can see we are deep in the ast module, but I'm hoping the final few frames indicate something about coverage.py code.

Do your generated source files use the new 3.10 match/case feature?

You mentioned the files are very large. Can you put some numbers to it? How many lines, and how deep are the nested structures (classes, functions, if-statements, etc, etc)? What's the deepest indentation level?

To find the particular file, you can add print(fr) just before this line in html.py:

File "/usr/local/lib/python3.10/site-packages/coverage/html.py", line 73, in data_for_file
    for lineno, tokens in enumerate(fr.source_token_lines(), start=1):

[jaredhobbs]

The end of the stack trace is:

  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 420, in generic_visit
    self.visit(value)
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 420, in generic_visit
    self.visit(value)
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 414, in generic_visit
    for field, value in iter_fields(node):
RecursionError: maximum recursion depth exceeded

The generated source files don't use the match/case feature; they were generated with Python 3.8.

After printing out the file names, I found the file that is being processed before the recursion error. That file is 15,571 lines long. It was generated with generateds version 2.40.6. I didn't see any deeply nested structures in the file. Seems like the deepest nesting was about 4 levels deep.

[nedbat]

I'm trying to think how to debug this without the source. If you can email it to me to keep from posting it here publicly, that will work too: ned@nedbatchelder.com

Can you try this program on it?

import ast

class MatchCaseFinder(ast.NodeVisitor):
    """Helper for finding match/case lines."""
    def __init__(self, source):
        # This will be the set of line numbers that start match or case statements.
        self.match_case_lines = set()
        self.visit(ast.parse(source))

    def visit_Match(self, node):
        """Invoked by ast.NodeVisitor.visit"""
        raise Exception("visit_Match shouldn't have been called!")

with open("THAT_FILE") as f:
    source = f.read()

print(MatchCaseFinder(source).match_case_lines)

[jaredhobbs]

That program also triggers the RecursionError. Here's the output from the top and bottom:

root@43ac68281994:/src# python3 tmp.py
Traceback (most recent call last):
  File "/src/tmp.py", line 17, in <module>
    print(MatchCaseFinder(source).match_case_lines)
  File "/src/tmp.py", line 8, in __init__
    self.visit(ast.parse(source))
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 418, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 418, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
...
  File "/usr/local/lib/python3.10/ast.py", line 420, in generic_visit
    self.visit(value)
  File "/usr/local/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/usr/local/lib/python3.10/ast.py", line 414, in generic_visit
    for field, value in iter_fields(node):
  File "/usr/local/lib/python3.10/ast.py", line 254, in iter_fields
    yield field, getattr(node, field)
RecursionError: maximum recursion depth exceeded while calling a Python object

[nedbat]

It sounds like a bug in the ast module, since my code isn't even in the stack there. But unless you can share the file with someone we'll never know what's going wrong.