Open johnynek opened 7 months ago
@mfarrugi started this out in https://github.com/bazelbuild/rules_rust/pull/753 but it looks like it got a bit stalled. If someone's happy to pick it up and update it, we'll review it :)
The hardest part for python rust interop is passing correct flags when compiling pyo3, which is typically handled by a build script. The build script calls a python interpreter to gather correct flags. I believe some work needs to be done in rules_python
https://github.com/bazelbuild/rules_python/issues/824.
I have an example here https://github.com/liningpan/pyo3-bazel-example, which allows the pyo3 build script to call a hermetic python interpreter as a work around.
I stopped working on this because there was no way to address the One Definition Rule issue. The the example in: https://github.com/bazelbuild/rules_rust/pull/753#discussion_r711642854 shows two extensions that each use the helper
library, which is then ultimately loaded twice by a python interpreter. It's very possible this is not the true issue underlying the example, and it might just have been an under-baked pull request by me.
The naive implementation is probably workable 90%+ of the time, but otherwise requires careful usage or fully dynamic linking to be safe (including rust std?). For what it's worth, rules_python didn't address the ODR issue either, and I would be surprised if it does now; there are docs somewhere on how this is solved inside google/blaze.
(I'm a rules_python maintainer; I was pointed here by someone)
What information from the Python runtime/toolchain is needed? rules_python now has @rules_python//python/cc:current_py_*
targets to give the headers and runtime libraries. Does Rust need something else or more?
I stopped working on this because there was no way to address the One Definition Rule issue. The the example in: https://github.com/bazelbuild/rules_rust/pull/753#discussion_r711642854 shows two extensions that each use the helper library, which is then ultimately loaded twice by a python interpreter. It's very possible this is not the true issue underlying the example, and it might just have been an under-baked pull request by me.
I'd agree with you're assessment, you're probably right.
There's a couple solutions:
For (2), there's some unactivated code in py_binary that takes that dependency information and creates a shared library (grep for "native_deps" in the python code base) that Python should be able to load.
If embedded Python is being used, then statically linking everything is also an option.
HTH
There is now https://github.com/abrisco/rules_pyo3 in case it's helpful to anyone 😄
The PyO3 library provides a tool,
maturin
to help in connecting python and rust. Using that tool in a bazel repo opens a lot of new challenged in that it is suggested to be installed withpip
and ideally we want a hermetic and reproducible build. Even if that part is tractable, it is still not a solved problem how to link the libraries.It would be great to have some rules like:
It seems like putting a solution to this in these rules is the right place for it since I think it would need to interop pretty tightly with rules_rust and maybe less tightly with rules_python (which are still just a wrapper of native python support). On the python side, it seems PyInfo provider may be enough to make things work.