Set up a Python<-->CIRCT interface and add some tests

[teqdruid]

The MLIR python bindings documentation make bindings to Python sound easy to set up, though somehow I suspect differently. @GeorgeLyon are you working on the C API for Python integration? Or was it Swift?

I'm thinking it may be interesting to use Python as a design entry language a la MyHDL. It may also be interesting to let hardware guys play around with various algorithms in Python before we implement them in C++. Hardware and EDA people are more likely to be familiar and comfortable with Python than C++.

[seldridge]

I do think having a Python interface to write simple transforms would be immensely powerful. A Verilog engineer might envision some simple transform (e.g., hierarchy manipulation or batch name changes), but it is prohibitive to expect a Verilog engineer to go hack up a C++ MLIR transform.

Analogously, pandoc (Haskell document conversion project structured like a compiler running passes over a document AST) has a Python package that lets you write passes (which they call filters) for the "But I don't want to learn Haskell!" crowd.

[GeorgeLyon]

Yup, I'm working on Swift, and I'm betting we can make development pretty nice for both the entry-level use case as well as more advanced usages. Currently I'm only focusing on emitting MLIR from an embedded DSL, but passes are a somewhat straightforward generalization of this.

On the Python side, @stellaraccident and her team have been doing some great work, and those bindings are far further along than what I have.

[mikeurbach]

I set up Python bindings to CIRCT locally, and it kind of just worked. I agree they can be super useful for a variety of reasons.

For the use case of writing transformations in Python, I think we still have a ways to go to make this palatable to end users, but I'm definitely hoping to getting there. One of the more interesting things I've heard on this topic was this in this thread about PDL, but I'm not sure where that ended up. I am also interested in testing out the Swift bindings some day, but I'm much more familiar with Python.

For my use case, I just wanted to register the CIRCT dialects and passes from Python, along the lines of what is exposed by NPComp here: https://github.com/llvm/mlir-npcomp/blob/main/python/NpcompModule.cpp#L57-L58.

What is the minimal Python interface we'd want for starters? So long as all we're doing is binding existing the C API to Python, it should be straightforward to do with pybind11, similarly to how it's done in NPComp. I'd be happy to contribute this to CIRCT.

[stellaraccident]

I'd be happy to offer help/advice on getting the python bindings usable in your setup. I'm (slowly) working through last mile usability issues to make them a good place to hack on higher level ml dialects and there are still a couple of disconnected pieces. So far, I've been able to express everything needed but not at the level of simplicity I expect in all cases.

The real weak point right now is cross project support for simple out of tree dialects: it's relatively clunky how projects for together now and really only works for "heavy" projects like npcomp or circt that can afford to add some API shims to register everything (and are not trying to do exotic cross project dynamic linking). I feel like it should be fine for your immediate use cases, though, if you copy how npcomp set it up (modulo: base it on BUILD_SHARED_LIBS vs libMLIR, which has some bad issues until I get some of the dynamic linking bits fixed upstream).

Lmk how it goes.

[stellaraccident]

And on the transformation side, sounds like you want the PDL work to go somewhere. I think River is working on that but haven't checked recently.

[teqdruid]

What is the minimal Python interface we'd want for starters? So long as all we're doing is binding existing the C API to Python, it should be straightforward to do with pybind11, similarly to how it's done in NPComp. I'd be happy to contribute this to CIRCT.

I think it's fair to start with what you've got. Can you get it into a form suitable for PR'ing?

As for an MVP, I don't know what's reasonable since I haven't looked into it. For me, I'm looking for a Python API which allows my internal customers to migrate their existing ad-hoc verilog generation scripts into CIRCT. At first it'd be creating hw/comb/seq/sv ops, then we'd move higher into the stack gradually hardening what they're doing (contributing as much as possible into upstream CIRCT). I don't want them to have to worry about types, so ideally the Python API would be able to infer types based on the Values going into the builders.

I'd be happy to offer help/advice on getting the python bindings usable in your setup. I'm (slowly) working through last mile usability issues to make them a good place to hack on higher level ml dialects and there are still a couple of disconnected pieces. So far, I've been able to express everything needed but not at the level of simplicity I expect in all cases.

Thanks! What have you not been able to simplify to your satisfaction?

And on the transformation side, sounds like you want the PDL work to go somewhere. I think River is working on that but haven't checked recently.

I looked into PDL a while back for some of the lowerings I was doing. Specifically, I was looking for a way to write a bunch of replacement ops which only had to be modified slightly. I couldn't grok how to integrate PDL as a normal pass. It also looked like it was incomplete. I didn't have much time to look into it, however, so that's just the feeling I got.

[mikeurbach]

I think it's fair to start with what you've got. Can you get it into a form suitable for PR'ing?

What I have is really just a simple pybind11 module that exposes the registerAllDialects and registerAllPasses C API so I can have a circt-opt-like experience from Python. If that's useful, I can start there for sure.

It sounds like setting up dialect and operation wrappers in Python is what people are most interested in. That would allow us to start playing with the builders from Python. I have only really read the docs on this, but since it sounds like that is a direction we want to go, I can take a look at that too.

[teqdruid]

It sounds like setting up dialect and operation wrappers in Python is what people are most interested in. That would allow us to start playing with the builders from Python. I have only really read the docs on this, but since it sounds like that is a direction we want to go, I can take a look at that too.

I've really only done the same and it sounds like this is what we want. I'd appreciate if you were to do this.

[mikeurbach]

I'm at the point where I can say import circt, and should have a PR for that groundwork tomorrow or over the weekend. I'll set up the Python wrappers for RTL and SV after that, as well as an ExportVerilog binding.

[stellaraccident]

Nice! Glad/hope it proves useful. So you know, we've been doing the tablegen work for the dialect and operation wrappers on an as needed basis and there are still gaps. It's got some mileage on it, though, and works for a decent swath of things, give or take occasional hacks still needed. Let me know if you run up against limitations.

[mikeurbach]

The initial PR is in. Next up, going to need to bump LLVM to pick up the latest and greatest w.r.t. generating dialect bindings. Then, I will try to actually set those up so we can build RTL/Comb/SV ops from Python.

[mikeurbach]

The RTL dialect bindings are in.

I think the larger strategy is being discussed in #805, but I had some next steps on the last PR that I'll mention here:

1. add a Python extension to customize RTL module building, and add that to the test
2. add bindings to the SV dialect. this should be nice and clean after some of the boilerplate in this PR, and could serve as an example for how to add bindings to other dialects (as we add more C APIs for them)
3. add bindings to ExportVerilog, including tests

Each of these should be a relatively simple, isolated patch. I plan to submit them in the next week+, but if someone else wants to jump in, the foundation is there now.

That's about as far as we can get in the "add Python bindings to the C API" department, which I consider to be the scope of this issue.