Can shared python be made relocatable?

[shakfu]

Hi,

First of all thanks very much for sharing your project. I was very happy to find it considering how challenging this topic is to me. In my own limited experience trying to package and embed python into c 'externals' for Max/MSP for my project (https://github.com/shakfu/py-js) I've spent months trying to package them in a 'relocatable' way.

So far, I have managed to do the following to achieve this:

1. Repackage and repurpose Homebrew python (using the method outlined in this blog post which has been surprisingly successful.

2. Build statically self-contained externals using a statically compiled python which tries to compile everything into the static library outlined in this beeware project

There remain two other methods: using shared-python and framework-python builds.

Previously I paused trying to build using the Framework method because of my lack of experience and knowledge about code signing, notarization and what constitutes a well-formed framework, although I'm much better informed now than before. In any case, having found your project, I will be trying to apply it.

But my question relates to shared-python. In fact, I'm finding it ridiculously difficult to make a shared-python build relocatable. I've been hitting my head against the wall so recurrently on this that I wanted to write to ask if this problem is actually solvable.

Here's my experiment:

Let's say I build python 3.9.4 from source on a Mac (Catalina) using:

    ./configure \
        MACOSX_DEPLOYMENT_TARGET=$MAC_DEP_TARGET \
        --prefix=$PREFIX \
        --enable-shared \
        --with-openssl=$BUILD_LIB_SSL \
                --without-doc-strings \
                --enable-ipv6 \
                --without-ensurepip \
        --with-lto \
        --enable-optimizations
    make altinstall

It's successfully built and I got through all the dylib dependencies of the resulting build in $PREFIX and use otool -L ... and install_name_tool to make everything relative and to copy dependencies into the build itself.

• I copy the build from $PREFIX to let's say $PREFIX2

• cd into $PREFIX2 and scan recursively with grep -r $PREFIX for all references to $PREFIX in the built binaries and the python code itself, and discover many in both. I replace all of these with $PREFIX2

• I then compile my extensions or (externals in Max parlance) using $PREFIX2 which are successfully compiled without error

• I move the built externals to another location and load them into Max and they run successfully, however a quick test of import os; print(os.__file__) gives the result that the external still references the original $PREFIX. So the code is working but the external remains hard-coded to $PREFIX.

So I have come to the conclusion that $PREFIX is hardcoded into shared-python builds at compilation in a way that is opaque and certainly beyond me.

Does this mean that one should give up on share python and use either static or Framework builds to make python or python-derivative code relocatable?

S

[gregneagle]

It sounds like you have more experience with and know far more about building various flavors of Python on macOS than I do. I'm I don't have answers or any additional info for you.

[shakfu]

Hi Greg,

Thanks for your response. I just edited my original post slightly to remove some typos, but you are too modest (-: Actually, following my original post, I successfully applied your build method to my project and found to it be perfectly 'relocatable'. So it works great! Thanks again.

I think this quite challenging topic is probably best raised on the python bug tracker or at the python packaging sig which seems to be quite active these day. I saw that you have already done so with bug report 42514 and it has also been directly addressed to some extent in 18309.

In scanning through the packaging sig, I found this post -- Packaging interpreter embedding frameworks -- describing issues most like what I'm experiencing in my project.

In any case, at this stage, it's not critical since I am able to package either with the static method or via your 'relocatable framework' method. 👍

S

[gregneagle]

Closing this as there's not work to be done.