Closed pmneve closed 3 years ago
I also encountered this error. Maybe it's because line 1307 in get_node, sometimes it enlarges the dictionary.
Have the same issue - did you find a work around?
Good catch. I agree with @zkkzhangkangkang's static analysis, get_node
seems to be the only place called by collapse_inner
that could cause this.
The reason then must be that the parent node being looked up by get_parent_node
does not already exist - and no other nodes with the same short name exist, since a new entry is being added to self.nodes
, changing the dictionary size. This shouldn't happen for the use case of collapse_inner
, but apparently does. :)
For testing, do any of you have a minimal working example, or a public project, that reproduces the bug? This would help me see why the parent node isn't getting defined.
As a fallback solution, it is possible to postpone the update of self.nodes
until the loop in collapse_inner
has completed, but that complicates the code, and makes the implementation hard to explain (so by ZoP §17, probably a bad idea). We would need something like this pattern:
def get_node(..., stash=None):
...
target = stash or self.nodes
if name in target:
target[name].append(n)
else:
target[name] = [n]
return n
...
def collapse_inner(...):
stash = {}
def unstash():
nonlocal stash
for name in stash:
if name in self.nodes:
self.nodes[name].extend(stash[name])
# TODO: probably need to drop duplicates - list(set(...))?
else:
self.nodes[name] = stash[name]
stash = {}
handled = set()
while True:
for name in self.nodes:
if name in ('lambda', 'listcomp', 'setcomp', 'dictcomp', 'genexpr'):
for n in self.nodes[name]:
if n not in handled:
....
# use the stash option here
handled.add(n)
if not stash:
break
unstash()
@Technologicat, I was using a large DNN code base, but here is one minimal set that causes this. It seems a combination of __init__.py
and this additional file causes the error.
Notes:
a) If I rename __init__.py
to a.py
it works
b) If I change the 2nd argument of f.py to not involve the for iteration it works
c) The example is nonsensical - I just made sure it runs
command and output:
find . -iname "*.py" | xargs pyan3 --colored --no-defines
Traceback (most recent call last):
File "/Users/pp/anaconda3/envs/ml/bin/pyan3", line 11, in <module>
sys.exit(main())
File "/Users/pp/anaconda3/envs/ml/lib/python3.7/site-packages/pyan/main.py", line 109, in main
v = CallGraphVisitor(filenames, logger)
File "/Users/pp/anaconda3/envs/ml/lib/python3.7/site-packages/pyan/analyzer.py", line 77, in __init__
self.process()
File "/Users/pp/anaconda3/envs/ml/lib/python3.7/site-packages/pyan/analyzer.py", line 87, in process
self.postprocess()
File "/Users/pp/anaconda3/envs/ml/lib/python3.7/site-packages/pyan/analyzer.py", line 154, in postprocess
self.collapse_inner()
File "/Users/pp/anaconda3/envs/ml/lib/python3.7/site-packages/pyan/analyzer.py", line 1562, in collapse_inner
for name in self.nodes:
RuntimeError: dictionary changed size during iteration
Another, more legitimate example that is equally nonsensical, but is structured correctly. Same issue. pyan.zip
@pliablepixels As you may have guessed, I'm busy right now, but I'll have a look as soon as I have time. Having an example that reproduces the bug should make this much easier. Thanks for the MWE!
It looks like this PR fixes the issue: https://github.com/Technologicat/pyan/pull/24/files
Yes, it should work now.
(collapse_inner
now iterates over 'list(self.nodes), not
self.nodes`.)
Getting this on a specific directory (not seeing it elsewhere) What do you need to dig into this?
Traceback (most recent call last): File "/Users/nevep/.pyenv/bin/versions/3.7.4/envs/egs-api/bin/pyan3", line 7, in
exec(compile(f.read(), file, 'exec'))
File "/Users/nevep/bepress/pyan/pyan3", line 11, in
sys.exit(main())
File "/Users/nevep/bepress/pyan/pyan/main.py", line 109, in main
v = CallGraphVisitor(filenames, logger)
File "/Users/nevep/bepress/pyan/pyan/analyzer.py", line 77, in init
self.process()
File "/Users/nevep/bepress/pyan/pyan/analyzer.py", line 87, in process
self.postprocess()
File "/Users/nevep/bepress/pyan/pyan/analyzer.py", line 154, in postprocess
self.collapse_inner()
File "/Users/nevep/bepress/pyan/pyan/analyzer.py", line 1562, in collapse_inner
for name in self.nodes:
RuntimeError: dictionary changed size during iteration