Add Python API

[maliberty]

Multiple parties have requested a Python API to Openroad. Python is more popular than TCL in general and especially in the machine learning community. A proposal for such an API is here.

This issue is for community members to give feedback. The spec uses initialize_floorplan as an example. The goal is get feedback on the general structure rather than on initialize_floorplan in particular. Each tool as it is converted will require a more detailed API discussion.

A related effort will be to make Openroad available as a pip module. That will be tracked separately.

[grotival]

I've heard this API is planned to raise exceptions internally?

At Google exceptions are really hard to work with, would it be possible pipe them out somehow? Using absl::StatusOr would be great (but I think some parts of OpenROAD are using boost so that might not make sense) but really any other way than an exception would work.

[maliberty]

I've heard this API is planned to raise exceptions internally?

At Google exceptions are really hard to work with, would it be possible pipe them out somehow? Using absl::StatusOr would be great (but I think some parts of OpenROAD are using boost so that might not make sense) but really any other way than an exception would work.

What does 'pipe them out' mean? Given that they are a native part of both C++ and Python I'm not clear why we should be non-idiomatic. Can't you just catch them and convert them to whatever you wish?

[grotival]

There are more arguments pros & cons but I think the integration in larger systems is the most convincing to me:

For existing code, the introduction of exceptions has implications on all dependent code. If exceptions can be propagated beyond a new project, it also becomes problematic to integrate the new project into existing exception-free code. Because most existing C++ code at Google is not prepared to deal with exceptions, it is comparatively difficult to adopt new code that generates exceptions. Our advice against using exceptions is not predicated on philosophical or moral grounds, but practical ones. Because so much code has already been written without expecting exceptions, it would be difficult to integrate new code that generates exceptions without examining and fixing the negative implications throughout our large codebase. Things would probably be different if we had to do it all over again from scratch.

pipe them out would be: not raising the exception but rather populate a pointer through all the functions calls and let the caller decide what to do (raise the error or do something else).

If you decide it's not worth doing, we still have a way around that though:

what to do about a third-party library that throws exceptions as part of its interface: write a wrapper library that translates between the third-party library's style and Google style. Compile the wrapper library with -fexceptions, and have it catch all exceptions and translate them into a supported error-reporting mechanism. (e.g. Status.) Library clients invoke the wrapper instead of directly invoking the third-party library.

[maliberty]

This is a python API so are you objecting to python exceptions or c++ ones? The c++ exceptions would be converted to python ones by swig. Or are you planning to build directly on the c++ API? Do you not use boost or stl - they both throw exceptions.

[grotival]

oh sorry, just the C++.

I indeed do not boost or stl, I use absl

[maliberty]

oh sorry, just the C++.

I indeed do not boost or stl, I use absl

There is no chance that we will convert away from those. I imagine most packages you integrate will have the same issue as both libraries are quite common. Do you wrap everything you use?

I think the last line is the most telling "Things would probably be different if we had to do it all over again from scratch.".

[maliberty]

https://github.com/abseil/abseil-cpp/blob/1ae9b71c474628d60eb251a3f62967fe64151bb2/absl/base/internal/throw_delegate.cc#L109 is pretty icky - it just calls std::abort rather than throw which allows no handling at all (if ABSL_HAVE_EXCEPTIONS is off).

[QuantamHD]

Do you wrap everything you use?

Yes. Any exception generating third_party lib that Google pulls in has to be wrapped before it can be used outside of third_party code.

[maliberty]

Do you wrap everything you use?

Yes. Any exception generating third_party lib that Google pulls in has to be wrapped before it can be used outside of third_party code.

Then I think that is the path Google will have to take here in you are using the C++ api directly. Presumably Python users wouldn't need the wrapper.

[grotival]

I agree with you, in a perfect world, we would throw and catch exceptions. But given the complexity of the systems backward compatibility is important too. Most of Google (except 3rd party code) is like that because Google is large enough to do that.

No big deal if you don't think it's a good idea/tractable for OpenROAD C++ API, I'll do it on my side, it should not be too bad. Esp since that code should be stable enough since it's going to power some SWIG binding. I just thought about giving a shot at convince you (we talked about that with Ethan) because the most hands off the C++ API is (propagate errors for non exception codebases, Object oriented for multi threading/interactive mode etc...), the easier it is to use/the wider the audience.

[QuantamHD]

@maliberty Should we move forward with implementation now?

[maliberty]

Yes. I've started an implementation but keep getting side tracked. There is some complications with the app level callbacks that I think need to be moved into odb callbacks instead.

[proppy]

A related effort will be to make Openroad available as a pip module. That will be tracked separately.

Should we close #1659 and #1424 in favor of this issue?

[QuantamHD]

@maliberty Could you throw your WIP on a branch so I can take a look? I'd love to throw in a few cycles at it.

[maliberty]

@QuantamHD I'll clean it up later today and do that.

[maliberty]

@proppy I closed #1659 but I think #1424 should be kept for the pip work which is separate.

[maliberty]

@QuantamHD I'll clean it up later today and do that.

https://github.com/The-OpenROAD-Project/OpenROAD/pull/1849 which should be viewed as just where I am today.

[joamatab]

what's the plan for pip install openroad

we would like to use the router in gdsfactory

[maliberty]

It just wants for some one to finish the work. A start is in https://github.com/The-OpenROAD-Project/OpenROAD/pull/2311

[fxxhuang]

So, it's work? can I use it like python module as following?

python flow.py

Using the above method allows me to use other python modules (such as DREAMPlace) instead of just openroad. May be we can add some cmake option to build shared lib only for python-api users? In DREAMPlace, It first installs all C++&python interfaces and then uses python calls. I think python is much easier to use than tcl.

[maliberty]

@FuxingHuang This is an open request not completed work.