Thalyum
commented
1 month ago I created a draft PR where I suggest a change. As I do not know all the internals, I may have missed something.
I identified 3 ways to fix my issue:
- In the pyo3-ffi build script, to ignore the content of the
InterpreterConfig - in the function that parses the
sysconfigdata, to add a special case when cross-compiling, and ignoreLIBDIR - in the function that creates the
InterpreterConfigfrom the parsedsysconfigdatascript
I thought solution 3. was better, as solution 1. seems dirty. As for solution 2., I think the function that parses the script does its job perfectly. I believe it is up to the bit that uses the parsed output to handle the special case.
Let me know what you think. In the draft PR, I added a commit (to be removed later obviously) that points to the three parts in the code where I considered making a change.
Bug Description
I tried to use PyO3 crate, and cross-compile a simple tool for Aarch64. I was using buildroot to cross-compile for my target:
When I encountered an error during link. Here is the error I got with buildroot:
The error comes from a bad linker option:
-L /usr/lib. Buildroot uses a wrapper around the toolchain (including the linker) that checks if the build tries to use "unsafe" paths, especially in the includes path and library search path. As stated in buildroot documentation:We do not expect cross-compiled library in
/usr/lib, only host libraries or headers. I tried to reproduce with 'manual' cross-compilation. I reused the cross-compiled python interpreter built by Buildroot, as well as its cross-toolchain:setup the linker:
The custom rustflags is to add the search path for libgcc, as well as some verbosity.
Then compile:
No issue with manual cross-compilation. Since the linker recognizes the architecture of the libraries in
/usr/liband ignores them as they are for the host:But we still see the same kind of warning.
By using the
PYO3_PRINT_CONFIGvariable, I took a peek at the configuration generated bypyo3-build-config:I created a dummy configuration file with the content above, and I just changed the "lib_dir":
And used it to cross-compile:
And the compilation log shows that the 'faulty'
-L /usr/liboption has been replaced by-L /foo/bar:And there is no more warning for the linker.
This configuration file is generated from the
InterpreterConfigstruct, and based on the code comments, "lib_dir" should be the directory where the Python library is located. In the current case, "/usr/lib" is not a correct path.Extra question: since I used
PYO3_CROSS_LIB_DIRenv variable to cross-compile, should not "lib_dir" be filled with the content of this environment variable ?Steps to Reproduce
If you want to reproduce, here is how to use buildroot to get the cross-compiled python interpreter + cross-toolchain, as well as the simple application code.
simple app code
Here is the code for the tool, a very simple "hello" print:
=> Cargo.toml
=> main.rs
python interpreter cross-compilation with buildroot
Enable python3 for a basic configuration
Then compile. To be faster, and since only the cross-compilation of python3 is of interest here, we can compile directly the python3 package:
The build output will be available in
output/targetdirectory. We will find:output/target/usr/bin/python3.12output/target/usr/lib/libpython3.soBuildroot also builds the host version of the same interpreter:
output/host/bin/python3.12Backtrace
No response
Your operating system and version
Linux 6.6.34-1-MANJARO
Your Python version (
python --version)Python 3.12
Your Rust version (
rustc --version)rustc 1.79.0 (129f3b996 2024-06-10)
Your PyO3 version
0.22.0
How did you install python? Did you use a virtualenv?
Host + cross-compiled interpreter using Buildroot, no virtualenv.
Additional Info
No response