search

The-OpenROAD-Project / OpenROAD

OpenROAD's unified application implementing an RTL-to-GDS Flow. Documentation at https://openroad.readthedocs.io/en/latest/
https://theopenroadproject.org/
BSD 3-Clause "New" or "Revised" License
1.38k stars 485 forks source link

odb should be distributed as a regular python package #1424

Open proppy opened 2 years ago

proppy commented 2 years ago

Currently the documented way to interact with the openroad python package is thru openroad -python (thru https://github.com/The-OpenROAD-Project/OpenROAD/blob/master/src/odb/src/swig/python/main.cpp?), it would be great if it was also distributed as a regular python package on PyPI and/or conda-eda so that developers can import it from a regular python interpreter without relying on the openroad binary, ex:

$ python
>>> import openroad
maliberty commented 2 years ago

We used to support odb in python as you describe but have never tried to support all of openroad. I suspect it is possible but a fair amount of work to do and support (e.g. we don't build shared libraries currently). Would you describe how this would be advantageous over "openroad -python"?

proppy commented 2 years ago

Would you describe how this would be advantageous over "openroad -python"?

It does make it easier to use openroad from other python projects or from a jupyter notebook.

but have never tried to support all of openroad

I wonder if the remaining part could be bridged with https://docs.python.org/3/library/tkinter.html, it does seems to allow TCL from python using: https://docs.python.org/3/library/tkinter.html#tkinter.Tcl

maliberty commented 2 years ago

I don't think that will help as we need to have the tcl interpreter object in c++ in order to register our swig wrapper on it. We could make a command to forward to tcl though that would be the ugliest interface possible.

proppy commented 2 years ago

I don't think that will help as we need to have the tcl interpreter object in c++ in order to register our swig wrapper on it.

Ah yes, you're right those must be two different Tcl context.

that would be the ugliest interface possible.

Ack, out of the command listed under https://openroad.readthedocs.io/en/latest/main/README.html#run, are there ones that are heavily dependent on a non-trivial amount of TCL to be usable? and therefore not trivially expose-able to python thru the existing SWIG interface?

maliberty commented 2 years ago

Ack, out of the command listed under https://openroad.readthedocs.io/en/latest/main/README.html#run, are there ones that are heavily dependent on a non-trivial amount of TCL to be usable? and therefore not trivially expose-able to python thru the existing SWIG interface?

pdngen & icewall are totally in tcl so they are the worst. Most others do argument handling in tcl and then drop to c++. Reading sdc implicitly requires tcl (SDC is a set of commands in tcl per its definition) - I think we will have to have a behinds the scenes interpreter for read_sdc.

proppy commented 2 years ago

another thing to consider could be to have something similar to http://yowasp.org/ (OpenROAD compiled as a WASM bundle using https://github.com/WebAssembly/wasi-sdk) to enable cross platform distribute on PyPI.

maliberty commented 2 years ago

In the future UPF support will also require TCL for the same reason as SDC. I looked at the wasi docs and it seem rather immature with lots of caveats. I don't want to add extra headaches.

I'm still unclear if you are proposing an identical interface in python as tcl or a more programmatic one.

proppy commented 2 years ago

To go back to my original issue https://github.com/The-OpenROAD-Project/OpenROAD/issues/1424#issue-1079742993, I'm just suggesting that we support importing the current python binding outside of the python interpreter embedded in the openroad process (from openroad import odb from a regular python process), we can discuss the scope of those bindings in a separate issue #1659.

maliberty commented 2 years ago

@proppy would you help to define what needs to happen here in specific actions. I'm not a pip expert and I'm hoping you are.

mithro commented 2 years ago

FYI - You will need to build OpenROAD against the "manylinux" platforms - https://github.com/pypa/manylinux#docker-images

mithro commented 2 years ago

@umarcor has been helping us get GitHub Actions running on a bunch of other repositories.

mithro commented 2 years ago

I was working on trying to make this easy for SymbiFlow,

proppy commented 2 years ago

Looks like https://cibuildwheel.readthedocs.io/en/stable/ can also help achieve this.

proppy commented 2 years ago

And https://github.com/marketplace/actions/pypi-publish for publishing.

proppy commented 2 years ago

Both are being maintained under @pypa so should be future safe to depends on.

umarcor commented 2 years ago

@proppy pypi-publish is actually a shell wrapper calling twine: https://github.com/pypa/gh-action-pypi-publish/blob/master/twine-upload.sh. Nowadays, it's not worth using a Container Action for such task. A Reusable Workflow or a Composite Action is enough. See https://github.com/pyTooling/Actions/#context.

proppy commented 2 years ago

The @pypa action does seems to have additional knobs to target different PyPI repository (ex: test PyPI) and run twine check to verify the package metadata before uploading.

Maybe it would worth filling an issue on https://github.com/pypa/gh-action-pypi-publish to suggest that they use composite action over containerized one? And then @pyTooling could simply reuse/extend it.

umarcor commented 2 years ago

The @pypa action does seems to have additional knobs to target different PyPI repository (ex: test PyPI) and run twine check to verify the package metadata before uploading.

That's a feature of twine. See:

Maybe it would worth filling an issue on https://github.com/pypa/gh-action-pypi-publish to suggest that they use composite action over containerized one? And then @pyTooling could simply reuse/extend it.

Might be interesting. So far pyTooling has been focused on building, testing, and generating docs/reports, rather than covering the widest range of platforms. Since building is out of scope of pypi-publish, pyTooling might need to be enhanced with e.g. manylinux before using pypi-publish (or specific options of twine).

/cc @Paebbels

Paebbels commented 2 years ago

Adding a twine check call is a welcome addition to pyTooling. Is signing the package with PGP also required? @umarcor, I think we should support this too, right?

Is there also a need for twine verify-upload?

But for supporting the PyPI test repository, I don't see a use case. What additional knowledge would be gained from a pipeline step uploading to that repository?

proppy commented 2 years ago

Since building is out of scope of pypi-publish, pyTooling might need to be enhanced with e.g. manylinux before using pypi-publish (or specific options of twine).

It seems that @pypa maintains https://github.com/pypa/cibuildwheel for this, so maybe pyTooling could be refactored as a composite action that uses both?

proppy commented 2 years ago

Let's move the meta discussion to https://github.com/pyTooling/Actions/issues/47 and keep this issue about the specifics of the OpenROAD implementation of building wheels and publishing them to PyPI.

maliberty commented 2 years ago

Is this moving towards a conclusion?

umarcor commented 2 years ago

@maliberty the dialogue during last week drifted towards CI solutions. We moved the discussion to https://github.com/pyTooling/Actions/issues/47. That is orthogonal to making openroad a Python package. First, the setup configuration file should be created and tested on a single platform only. Even before doing so, I think that the expected interaction should be prototyped. Overall, pip is "just another" packaging solution which can distribute python, tcl and other assets.

maliberty commented 2 years ago

I have no experience with pip or its configuration files, etc. We don't package at all today. Can you give a more specific set of steps?

umarcor commented 2 years ago

Do not care about "pip" for now. Let's focus on "Python setup files". Some examples:

So, conceptually it's very similar to any other "recipe" (say deb, rpm, conda...). You declare the sources that belong to the "package", the "entrypoints" you want to generate, and other metadata (author, license, compatible versions, etc.).

I believe that the "shell wrappers" of f4pga can be inspiring to you:

Now, when users install f4pga through pip, they get "binaries" such as symbiflow_pack created "somewhere in the system available in the PATH". At the same time, the python sources and shell scripts are installed "somewhere in the system where python libs are installed". The symbiflow_pack "binary" calls the ?/wrappers/sh/__init__.py, which executes ?/wrappers/sh/xc7/pack.f4pga.sh.

In the context of openroad, you might do something similar. That would provide a not-ideal but working solution to install openroad through pip. From there on, the Python wrapper functions can be reworked to use tkinter or some other "more elegant" solution than blindly calling an "external" TCL script as a system tool.

proppy commented 2 years ago

A bit more background: The goal of https://github.com/The-OpenROAD-Project/OpenROAD/pull/1849, is to expose a subset of the OpenROAD C++ API as a regular python extension, and not really to replace the main OpenROAD executable with a Python entrypoint.

In addition to the example that @umarcor pointed to for the "Python setup files", https://packaging.python.org/en/latest/tutorials/packaging-projects/ is also a good resources for getting started with Python packaging.

So the steps are likely to be: a. write the required setup.py/cfg, pyproject.toml metadata b. bridge the python build process with cmake using something like https://github.com/diegoferigo/cmake-build-extension c. integrate with a cross-platform build GH action like https://github.com/pypa/cibuildwheel and https://github.com/pyTooling/Actions to build the wheel for all supported platform on each commits d. integrate with a publishing GH action like https://github.com/pypa/gh-action-pypi-publish or https://github.com/pyTooling/Actions to publish to PyPI prod and/or test at the desired release cadence.

More than happy to contribute/help navigate those steps, maybe we could already start the work with the existing odb binding and update it include the newer binding when #1849 lands, wdyt?

maliberty commented 2 years ago

@proppy I'm happy to have you start with odb