search

astral-sh / rye

a Hassle-Free Python Experience
https://rye.astral.sh
MIT License
13.82k stars 467 forks source link

Ability to build Python with `--enable-shared` #1050

Open toiletsandpaper opened 6 months ago

toiletsandpaper commented 6 months ago

Problem

Rye has a one major disadvantage right now - rye is installing python versions without libpython.so which leads to many problems with using python from other languages.

For example, this https://github.com/astral-sh/rye/issues/920 issue is mostly because of libpython is not available for rye's python.

How it should work

For apt or dnf/yum we have python3-dev and python3-devel to download shared object For pyenv we have https://github.com/pyenv/pyenv/blob/master/plugins/python-build/README.md#building-with---enable-shared

IMO, this feature is very important and is a blocker for most of python developers

bluss commented 6 months ago

Rye's pythons are built with --enable-shared, and installs libpython and the library is there, but I think this is the same issue as #646 - the configuration paths to where those libraries are located are not correct and need to be set after installing the toolchain in its final location.

Trying to find it, we see it is there:

> find .rye/py -iname "*libpython*.so"
.rye/py/cpython@3.8.18/lib/libpython3.so
.rye/py/cpython@3.8.18/lib/libpython3.8.so
.rye/py/cpython@3.11.8/install/lib/libpython3.so
.rye/py/cpython@3.11.8/install/lib/libpython3.11.so
.rye/py/cpython@3.12.1/lib/libpython3.so
.rye/py/cpython@3.12.1/lib/libpython3.12.so
.rye/py/cpython@3.12.2/install/lib/libpython3.so
.rye/py/cpython@3.12.2/install/lib/libpython3.12.so

Unfortunately it doesn't tell the truth about where it is

> python +3.12 -m sysconfig | grep LIBDIR
    LIBDIR = "/install/lib"

This LIBDIR value is what it ships with and needs to be updated.

With the sysconfigpatcher script - which is WIP and only works on linux I think, then it does the right thing

> python +3.11 -m sysconfig | grep LIBDIR
    LIBDIR = "/home/user/.rye/py/cpython@3.11.8/install/lib"
    PLATLIBDIR = "lib"
dewoller commented 5 months ago

Hi @bluss

Rye is great, a breath of fresh air in this messy python library space. Unfortunately, I also need shared library support, and, on mac, the following line comes up empty

find ~/.rye/py -iname "libpython.so*"

There are no libpython* files under the .rye directory, so it appears that the libpython does not get installed on the mac.

Thanks Dennis

bluss commented 5 months ago

I haven't done any further experiments on macos, but the sysconfigpatcher script worked for me on macos, with it I can build something that seems to require the python lib, and libpython3.12.dylib is present - it's not an .so.

intercepted16 commented 3 months ago

@bluss Your script doesn't seem to work for me. The environment variables successfully change, but the actual LIBDIR echoed doesn't.

bluss commented 3 months ago

@intercepted16 which exact toolchain have you downloaded? Could you get its name inside ~/.rye/py for me? (And platform + architecture?)

intercepted16 commented 3 months ago

@bluss

ll -a ~/.rye/py Permissions Size User Date Modified Name drwxr-xr-x@ - ab 6 Aug 20:42 cpython@3.12.4

I'm on x64 and using Fedora (Linux) KDE 40

bluss commented 3 months ago

@intercepted16 unfortunately it seems to work for me with that version, so more information would be needed. If you can, open an issue to sysconfigpatcher with steps to reproduce?