search

ronaldoussoren / modulegraph

modulegraph determines a dependency graph between Python modules primarily by bytecode analysis for import statements. modulegraph uses similar methods to modulefinder from the standard library, but uses a more flexible internal representation, has more extensive knowledge of special cases, and is extensible.
MIT No Attribution
38 stars 4 forks source link

modulefinder breaks on `idna` package #25

Closed ronaldoussoren closed 9 years ago

ronaldoussoren commented 9 years ago

Original report by Glyph (Bitbucket: Glyph, GitHub: Glyph).

$ cat importidna.py 
import idna
$ pip freeze | grep idna
idna==1.1
$ python -m modulegraph importidna.py 2>&1 | tail
    self.visit(value)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 245, in generic_visit
    for field, value in iter_fields(node):
RuntimeError: maximum recursion depth exceeded
$
ronaldoussoren commented 9 years ago

Original comment by Glyph (Bitbucket: Glyph, GitHub: Glyph).

This is causing problems now because cryptography now has a hard dependency on idna, and Mimic, which wants to package an app that uses Twisted (which uses Cryptography, which uses IDNA).

ronaldoussoren commented 9 years ago

Original comment by Glyph (Bitbucket: Glyph, GitHub: Glyph).

Particularly, the file in question seems to be idna.idnadata.

ronaldoussoren commented 9 years ago

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

Thanks for a given a specific example. That should make it a lot easier to find a fix.

ronaldoussoren commented 9 years ago

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

This is pretty annoying, idna.idnadata defines a data structure using large expression. Using ast.NodeVisitor on the AST for that expression causes too deep recursion, hence the runtime error.

The code in py2app that uses the NodeVisitor needs to be rewritten to avoid using that much stack space.

ronaldoussoren commented 9 years ago

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

Just avoiding recursion for expressions should be good enough. I'll push a fix later today.

ronaldoussoren commented 9 years ago

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

Modules that have deeply nested AST trees caused RuntimeErrors due to too much recursion.

Fixes #25

ronaldoussoren commented 9 years ago

Original comment by Glyph (Bitbucket: Glyph, GitHub: Glyph).

Wow, thanks for the quick fix! Will this be in a release soon?