Generator + regular-function stack, or, elaboration caching aint right

[dan-fritchman]

Test case:

def test_generator_function_combo():
    """ Test instantiating hierarchies that combine `@h.generator`s and regular old functions. """

    @h.generator
    def Gen(_: h.HasNoParams) -> h.Module:
        @h.module
        class HasDiff:
            d = h.Diff(port=True)
        return HasDiff

    def NonGen() -> h.Module:
        @h.module
        class HasTwoHasDiffs:
            d = h.Diff(port=False)
            a = Gen()(d=d)
            b = Gen()(d=d)
        return HasTwoHasDiffs

    h.elaborate(NonGen())
    h.elaborate(NonGen())

Produces:

E       RuntimeError: Elaboration Error at hierarchical path: 
E         Module        HasTwoHasDiffs              hdl21/tests/test_hdl21.py:1511
E         Instance      a                           hdl21/tests/test_hdl21.py:1513
E       Invalid connections `{'d_p': Missing connection to Port `d_p`, 'd_n': Missing connection to Port `d_n`, 'd': Connection to non-existent Port `d`}` on Instance `a` in Module `HasTwoHasDiffs`

hdl21/elab/elaborators/base.py:186: RuntimeError

[dan-fritchman]

A few notes:

• If both the outer and inner functions are generators, that works
• If both are not generators - i.e. they're both regular-old Python functions, that also works
• If the outer is a generator and the inner is a regular-old-function, that also works
• It's solely the combination of "generator instance inside regular-old function" that fails

[dan-fritchman]

The fact that this happens is tied to the (generally fraught) topic of flattening bundle-valued ports. More generally, of doing anything that modifies ports during an elaboration pass.

• We check whether each instance should verify connections against "flattened" or "pre-flattening" IOs, based on its elaboration status, and its parents' elaboration status
• We move any Bundle-ports to a _pre_flattening_io field after flattening them
• ... But sadly it's getting overridden, shortly thereafter, by the recently-flattened IOs

Why:

• Elaboration's "is this module already done" check is for the entirety of elaboration, i.e. have all passes completed
• There's no real per-pass cache
• The BundleFlattener pass encounters HasDiff twice - once per instance. The first time it flattens and write _pre_flattening_io. The second time, it errantly overwrites _pre_flattening_io, with the just-flattened IOs.

I think this bug's been there forever. The sole answer would seem to be per-pass elaboration "caches"/ "done-lists", particularly for elaboration passes which are not idempotent. I think that is only bundle-flattening to date, but there are plenty more possible (e.g. "infer unspecified port directions").

And why do the generator versions work? We can just more aggressively identify, cache, and skip re-running them. I suspect that's papering over this bug, perhaps other similar ones.