It's probably time for immutable hierarchy walking, compiling, elaboration, etc.

[dan-fritchman]

Hdl21 has many things that walk a hierarchical HDL-data-model tree, checking or manipulating each node along the way.
Notable examples include:

• PDK Compilers
• VLSIR protobuf export
• Elaboration itself

The way the "mutable" versions have always worked is by modifying their input modules inline. This has always helped streamline their implementations, largely because... we don't have to track what's in the old, input objects vs the new, compiled ones. They're the same things.

It's pretty much always been a concern that this would become a pain. Now it has. Particularly since:

• One of the key value propositions of Generator is to track unique parameter-calls, so that it does not export repeat modules. Doing so would generally fail in downstream tools, particularly legacy EDA formats we netlist into.
• This manifests itself as a global "generator call cache", which also helps speed up running slow ones.
• We now have a handful of applications along the lines of:
  • 1. Run some generator
  • 2. Compile its result into some context (e.g. PDK)
  • 3. Run that generator again
  • 4. Compile it into some other context

Most prior applications of Hdl21 had essentially just done (1), (2), end program, run a separate Python program/ process. That still works just fine.

We added band-aids to help aid the (1-4) loop, notably including "public" access to the GeneratorCache, and a capacity to reset it. That also can work. It's just very hard to communicate (and in fact, to remember) when one needs to do it. And in some cases it doesn't have any effect. Particularly when the programs take references to the result modules, instead of re-calling the generator functions.

Ideas:

• We want stuff that is compiled to be as close to immutable as possible. (With some caveats below.)
• Before the first, biggest compiler (elaboration), modules need to be mutable. (That's how they get built.)
• Modules do not disappear when compiled. (They can't.)
• Modules get some background metadata indicating the things they've been compiled into.
  • Maybe like Dict[int, Module] from id(compiler) to the compiled Module
• Instances get their of targets replaced by the newly compiled Module.
• If a Module gets used in more than one in-memory hierarchy, replace only the Instances in the compilation tree. Others continue to use the original.
• All the compilers should probably return the compiled form. Otherwise it'll be a hassle to dig out of that metadata.

Questions:

• Do we clone non-hierarchical things like Signals?
  • Case for: yeah, we want them to be immutable too.
  • Case against: it's a headache?
• Do we want some kinda "copy on write" behavior, such that if a compiler doesn't actually make modifications, it returns its input as-is?
• Does elaboration play by the same rules, or is it special? Can see pros either way.
• Do we want a separate type for the result of elaboration - e.g. an ElaboratedModule.
  • This would remove all the stuff that can't exist after elaboration, e.g. Bundle instances.
  • It could be a nice mechanism to enforce the (mostly) immutability.