RFC: create Modules/ExternalModules from on-disk protos

[growly]

It strikes me as convenient to be able to import a module by doing this:

import hdl21 as h

mod = h.ExternalModule(from_proto="/path/to/proto.pb")
# or
mod2 = h.Module(from_proto="/path/to/proto.pb")

It strikes me as such because I want to write a testbench for an externally-generated module in hdl21.

Obviously the long way is to the use the import/export functions already there, but they're more verbose. This just feels more sugary.

Thoughts?

[growly]

After thinking about it for literally 1 more minute I realise that this is slightly more complicated; we haven't imagined writing Modules to disk one at a time, they're usually in a Package with the transitive closure of their dependency graph neatly ordered before them.

Still, it would be might convenient. Maybe instead it's

import hdl21 as h
mod = h.Module(name="my_thing", in_package="/path/to/package.pb")

[dan-fritchman]

So what would doing this look like now?
Isn't it, like, one additional line in front of h.from_proto() ?

[growly]

I dunno. Let's leave this half-baked thought to prove for a while. I guess I try it both ways for a bit first.

[dan-fritchman]

Yep, good to see how it looks without dedicated/ built-in support.

Couple thoughts if we do add such a thing:

• Make it a function, not an expansion on the Module constructor args
• When importing from VLSIR/ protobufs, Hdl21 tries to recreate a "namespace hierarchy" similar to what it exports, via nested SimpleNamespaces. Probably easiest by example. If you have files like:

__init__.py
pymod1.py
pymod2.py

# in `pymod1.py`

import hdl21 as h
m = h.Module(name="FlipFlop")

# in `pymod2.py`

import hdl21 as h
m = h.Module(name="FlipFlop") # <= the point - same name 

Here there are two (Hdl21) Modules named FlipFlop. In Python this is fine; they are in different namespaces. In typical HDL tools which lack namespaces, they would be definitively not fine.

So Hdl21 tracks their defining (python) module, and exports them as something like

• vlsir.Module(name="pymod1_FlipFlop"), and
• vlsir.Module(name="pymod2_FlipFlop")

(Note this naming is the subject of a couple open bugs - #142 here and VLSIR #65.)

[growly]

Ok to not reusing the constructor, but I'm not sure what the point about namespaces is?

It looks like I just want a convenient way to do all of e.g.

package = vlsir_circuit.Package()
with open('../build/sky130_mux.package.pb', 'rb') as f:
    package.ParseFromString(f.read())
ns = h.from_proto(package)
mux = ns.sky130_mux

[dan-fritchman]

Looks pretty good!

Point is: depending how sky130_mux was defined, you may need to look for it several namespaces down, in something like ns.sub_namespace1.sub_namespace2.etc.

E.g. any of the unit tests around here:
https://github.com/dan-fritchman/Hdl21/blob/524373b7f4b8f14a47f2ee5287863fe0ff58d630/hdl21/tests/test_exports.py#L502