Crash with genexpr nested in list comprehension

[Technologicat]

To reproduce: feed pyan stage2.py (commit c614ebf) from https://github.com/TUTElectromechanics/mm-codegen

Log attached.

pyan_crash.txt

[Technologicat]

Specifically, the crash occurs when the genexpr appears as the collection to iterate over. This is the offending code fragment from stage2.py in the mm-codegen project:

        return [([(name, allargs) for name,_,_,allargs,_ in recs],
                  {name: inargs for name,inargs,_,_,_ in recs},
                  {name: meta for name,_,_,_,meta in recs})
                for recs in (results[key] for key in sorted(results.keys()))]

It seems the stage2.CodeGenerator.analyze_interface.listcomp.genexpr scope is not picked up by Python's symtable.symtable(). See output from pyan -cugan -V stage2.py, search for "Scopes now:". Look at analyze_scopes() in analyzer.py; it's not doing much except calling symtable and recursively extracting the results.

For now, waiting to upgrade Python to see if the issue has been fixed.

[Technologicat]

Digging deeper, there is a bug in Pyan after all: the outermost iterator in a comprehension (including genexprs) should be evaluated in the current scope (it is evaluated before the comprehension starts), but Pyan treats it as being part of the inner scope.

See function symtable_handle_comprehension() in https://github.com/python/cpython/blob/master/Python/symtable.c

For how it works, see https://stackoverflow.com/questions/48753060/what-are-these-extra-symbols-in-a-comprehensions-symtable

For related discussion, see https://bugs.python.org/issue10544