How to properly configure standalone build

[jochenhz]

Hey,

building python standalone works just fine on macOS. I can't find a proper way to configure the distribution, however. So when starting the interpreter sysconfig.get_config_var('LIBDIR') prints /install/lib - which breaks with lots of tools that rely on a proper sysconfig (e.g. pybind11). The distribution contains the makesetup but not the makefile template, for example.

Clearly, I must be missing something :-)

Thanks a lot for your help

[indygreg]

No, you aren't missing something: many values in the sysconfig config file are nonsensical.

The distributions work for many run-time configurations. However, some power user features (such as building/installing extension modules) will fail due to bad values in sysconfig.

One workaround for this is to rewrite the sysconfig config files when extracting the distribution archive so they reference paths appropriate for that machine. Ideally we'd provide a script inside the archives to facilitate doing this. Alternatively, we could ship modifications to sysconfig that dynamically resolve appropriate values at interpreter run-time. Neither of these approaches have been explored that much because most current users of these interpreters want a way to run Python applications and don't necessarily care about Python developer use cases.

If you spell out exactly what you are trying to accomplish with the interpreters that doesn't work today, I can take that as a feature request. At the very least we may want to improve this project's documentation so shortcomings are better documented.

[jochenhz]

Hey, thanks a lot for your detailed response. I am trying to embed the python interpreter in an application and would like to provide a standalone python as provided by this project.

This does not work out of the box due to wrong / incomplete sysconfig files. To quickly reproduce my setup you can do:

git clone --recursive https://github.com/pybind/pybind11.git
cd pybind11
mkdir build && cd build
cmake .. -DPYTHON_EXECUTABLE=path/to/standalone/install/bin/python3.9 -DDOWNLOAD_CATCH=ON
cmake --build .

When linking, this will fail with

ld: library not found for -lpython3.9
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[alecthomas]

For anyone wanting a solution to this, we wrote a small script for Hermit. It's pretty naive, but seems to work fine and others might find it useful.

[damian0815]

thanks @alecthomas that was super helpful

[LilyFoote]

Hey! I wanted to add my use-case here. I'm using these builds in https://github.com/LilyFoote/lilyenv. This is working very well for the most part, but I have a hybrid Python/Rust project where I'm running into the linking error mentioned above. This is specifically when compiling the Rust tests, which need to link against the interpreter to initialise it.

Fixing up the LIBDIR path (and other bits of sysconfig) is enough to partially fix this and I've added that to lilyenv, but I then see a different error I haven't worked out how to fix yet:

cargo test --manifest-path=rust/Cargo.toml -vv
       Fresh autocfg v1.1.0
<snip>
       Fresh pyo3 v0.20.0
       Fresh _kolo v2.16.0 (/home/lily/work/kloppindustries/kolo/python/rust)
warning: unused import: `_kolo::*`
 --> src/lib.rs:2:9
  |
2 | pub use _kolo::*;
  |         ^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: `_kolo` (lib test) generated 1 warning (run `cargo fix --lib -p _kolo --tests` to apply 1 suggestion)
    Finished test [unoptimized + debuginfo] target(s) in 0.01s
     Running `CARGO=/home/lily/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo CARGO_MANIFEST_DIR=/home/lily/work/kloppindustries/kolo/python/rust CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=_kolo CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=2.16.0 CARGO_PKG_VERSION_MAJOR=2 CARGO_PKG_VERSION_MINOR=16 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/lily/work/kloppindustries/kolo/python/rust/target/debug/deps:/home/lily/work/kloppindustries/kolo/python/rust/target/debug:/home/lily/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/lily/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' OUT_DIR=/home/lily/work/kloppindustries/kolo/python/rust/target/debug/build/_kolo-63393ec205212302/out /home/lily/work/kloppindustries/kolo/python/rust/target/debug/deps/_kolo-7de5c4c4609ef177`
/home/lily/work/kloppindustries/kolo/python/rust/target/debug/deps/_kolo-7de5c4c4609ef177: error while loading shared libraries: libpython3.12.so.1.0: cannot open shared object file: No such file or directory
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `CARGO=/home/lily/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo CARGO_MANIFEST_DIR=/home/lily/work/kloppindustries/kolo/python/rust CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=_kolo CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=2.16.0 CARGO_PKG_VERSION_MAJOR=2 CARGO_PKG_VERSION_MINOR=16 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/lily/work/kloppindustries/kolo/python/rust/target/debug/deps:/home/lily/work/kloppindustries/kolo/python/rust/target/debug:/home/lily/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/lily/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' OUT_DIR=/home/lily/work/kloppindustries/kolo/python/rust/target/debug/build/_kolo-63393ec205212302/out /home/lily/work/kloppindustries/kolo/python/rust/target/debug/deps/_kolo-7de5c4c4609ef177` (exit status: 127)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

I am able to work around this by setting LD_LIBRARY_PATH=/home/lily/.local/share/lilyenv/pythons/3.12/python/lib, but I didn't need this workaround in my previous workflow using pyenv and virtualenvwrapper, so I'm still looking for a better fix.

If I get this sorted too, it would be nice to use a standard script that is provided in the downloaded python archive instead of maintaining a patch in lilyenv.