Open mdavidsaver opened 4 years ago
@mdavidsaver, thanks for starting this. The inplace build trees might be indeed unrelocatable, but a package can be changed from being inplace installed to being regularly installed. Imagine what would happen if package B links to DSO from package A; package A was first inplace installed and later changed to be regularly installed. If in B's exts and dsos we skip adding $ORIGIN to rpath, what was built for B will stop working.
An ideal solution would mirror python import search path into dynamic linker search path, so that e.g. a link to a.b.c
DSO would search via import a.c
-> find directory of imported package and ask dynamic linker to load c
from that directory. But offhand I don't know how to reach this.
Maybe a practical solution woult be to have both $ORIGIN
and absolute reference to inplace directory...
Or, something better: leave $ORIGIN
/ @loader_path
thing as is, and tweak inplace install to also create neccessary symbolic links to create addition links in venv, besides .egg-link
installed into venv, so that loaded dsos could be resolved from there following $ORIGIN
-> pkg-in-venv -> worktree directory with inplace build.
This should work on Linux/macOS, but not sure about Windows.
For the reference https://github.com/mdavidsaver/setuptools_dso/pull/15#issuecomment-789557804 describes a real-world issue I'm hitting on Linux when exising scheme that leverages $ORIGIN
in RPATH is not enough:
---- 8< ----
let me share that I actually already hit it in practice with my wendelin.core + libgolang work: wendelin.core (project A) builds wendelin.wcfs.client.libwcfs
DSO (https://lab.nexedi.com/kirr/wendelin.core/blob/b5fc98bb/setup.py#L312-317) which is linked to golang.runtime.libgolang
(= DSO-B https://lab.nexedi.com/nexedi/pygolang/blob/4f28dddf/golang/pyx/build.py#L122-155). And when wendelin.wcfs.client.libwcfs
is loaded by dynamic linker, without special preparations, the linker cannot load it because it cannot find libgolang.so that is coming from golang.runtime.libgolang
.
For now I've worked it around by explicitly preloading golang.runtime.libgolang
(https://lab.nexedi.com/kirr/wendelin.core/blob/b5fc98bb/bigfile/__init__.py#L24-27), but of course a more proper fix is needed.
---- 8< ----
More specifically:
libwcfs.so has the RPATH entry that setuptools_dso thinks would point to libgolang place ($ORIGIN/../../../golang/runtime
):
(neo) (z4-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core$ objdump -p wcfs/client/liblibwcfs.so |grep -i path
RUNPATH $ORIGIN/../../bigfile:\$ORIGIN/../../../golang/runtime:\$ORIGIN/../../bigfile:$ORIGIN/../../../golang/runtime
but this assumes that both packages would be non-develop installed. When wendelin.core is develop-installed, or pygolang is develop-installed, or both are develop-installed, dynamic linker cannot find where libgolang.so is:
(neo) (z4-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core$ ldd wcfs/client/liblibwcfs.so
linux-vdso.so.1 (0x00007ffd3d9f4000)
liblibgolang.so.0.1 => not found
liblibvirtmem.so => /home/kirr/src/wendelin/wendelin.core/wcfs/client/../../bigfile/liblibvirtmem.so (0x00007ff6d8333000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff6d8166000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff6d814c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff6d812a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff6d7f63000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff6d8400000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff6d7e1f000)
As explained above I've worked it around with import golang
before loading extension that links to libwcfs.
But the problem is in environment where built packages are verified to have properly resolved dynamic library dependencies, this is still a problem, because the checker complains that libgolang.so cannot be found and build fails.
Unfortunately my production environment is combination of all of the above: a) develop-install (non-pip - by buildout) + b) verifier that checks that all libraries are properly linked.
I'd not come across buildout before. Glancing through the getting started I only see mentions of .egg
, and not .whl
, so I can guess at why this is causing problems for you.
I still don't see a clear solution for develop builds. The (increasingly crazy?) ideas I've had so far:
dsocmd.py
aware of -e
so RPATH can be computed relative to os.getcwd()
and/or some explicit install location(s). More complex, and maybe not much better than absolute paths.setup.py develop
on Linux/Mac and install actual symlinks for .so/.dylib files. Where I get stuck is wanting to avoid also replacing the .egg-link
with individual symlinks for .py files as well.LD_LIBRARY_PATH
based on foreknowledge of which DSOs will be needed. A specialized solution which might be sufficient in CI environments, but not much else.( @mdavidsaver thanks for your feedback; writing just to let you know that I'm still here and need to address this issue. I will try to come back in some time )
With #10 I spent a little time investigating if editable installs could be made to work more generally. My common use case has
DSO
andExtension
in two different packages. So it might be the case that either or both are installed with-e
.Since
setup.py develop
in most cases setsinplace=1
, the easiest course may be to skip the$ORIGIN
or@loader_path
tricks for in-place builds and just use absolute paths. It don't think there would be any expectation that in-place built trees would be relocatable?