Generalize editable install (`-e`)

[mdavidsaver]

With #10 I spent a little time investigating if editable installs could be made to work more generally. My common use case has DSO and Extension in two different packages. So it might be the case that either or both are installed with -e.

Since setup.py develop in most cases sets inplace=1, the easiest course may be to skip the $ORIGIN or @loader_path tricks for in-place builds and just use absolute paths. It don't think there would be any expectation that in-place built trees would be relocatable?

[navytux]

@mdavidsaver, thanks for starting this. The inplace build trees might be indeed unrelocatable, but a package can be changed from being inplace installed to being regularly installed. Imagine what would happen if package B links to DSO from package A; package A was first inplace installed and later changed to be regularly installed. If in B's exts and dsos we skip adding $ORIGIN to rpath, what was built for B will stop working.

An ideal solution would mirror python import search path into dynamic linker search path, so that e.g. a link to a.b.c DSO would search via import a.c -> find directory of imported package and ask dynamic linker to load c from that directory. But offhand I don't know how to reach this.

Maybe a practical solution woult be to have both $ORIGIN and absolute reference to inplace directory...

---

Or, something better: leave $ORIGIN / @loader_path thing as is, and tweak inplace install to also create neccessary symbolic links to create addition links in venv, besides .egg-link installed into venv, so that loaded dsos could be resolved from there following $ORIGIN -> pkg-in-venv -> worktree directory with inplace build.

This should work on Linux/macOS, but not sure about Windows.

[navytux]

For the reference https://github.com/mdavidsaver/setuptools_dso/pull/15#issuecomment-789557804 describes a real-world issue I'm hitting on Linux when exising scheme that leverages $ORIGIN in RPATH is not enough:

---- 8< ---- let me share that I actually already hit it in practice with my wendelin.core + libgolang work: wendelin.core (project A) builds wendelin.wcfs.client.libwcfs DSO (https://lab.nexedi.com/kirr/wendelin.core/blob/b5fc98bb/setup.py#L312-317) which is linked to golang.runtime.libgolang (= DSO-B https://lab.nexedi.com/nexedi/pygolang/blob/4f28dddf/golang/pyx/build.py#L122-155). And when wendelin.wcfs.client.libwcfs is loaded by dynamic linker, without special preparations, the linker cannot load it because it cannot find libgolang.so that is coming from golang.runtime.libgolang.

For now I've worked it around by explicitly preloading golang.runtime.libgolang (https://lab.nexedi.com/kirr/wendelin.core/blob/b5fc98bb/bigfile/__init__.py#L24-27), but of course a more proper fix is needed. ---- 8< ----

More specifically:

libwcfs.so has the RPATH entry that setuptools_dso thinks would point to libgolang place ($ORIGIN/../../../golang/runtime):

(neo) (z4-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core$ objdump -p wcfs/client/liblibwcfs.so  |grep -i path
  RUNPATH              $ORIGIN/../../bigfile:\$ORIGIN/../../../golang/runtime:\$ORIGIN/../../bigfile:$ORIGIN/../../../golang/runtime

but this assumes that both packages would be non-develop installed. When wendelin.core is develop-installed, or pygolang is develop-installed, or both are develop-installed, dynamic linker cannot find where libgolang.so is:

(neo) (z4-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/nexedi/wendelin.core$ ldd wcfs/client/liblibwcfs.so 
        linux-vdso.so.1 (0x00007ffd3d9f4000)
        liblibgolang.so.0.1 => not found
        liblibvirtmem.so => /home/kirr/src/wendelin/wendelin.core/wcfs/client/../../bigfile/liblibvirtmem.so (0x00007ff6d8333000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff6d8166000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff6d814c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff6d812a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff6d7f63000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff6d8400000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff6d7e1f000)

As explained above I've worked it around with import golang before loading extension that links to libwcfs.

But the problem is in environment where built packages are verified to have properly resolved dynamic library dependencies, this is still a problem, because the checker complains that libgolang.so cannot be found and build fails.

Unfortunately my production environment is combination of all of the above: a) develop-install (non-pip - by buildout) + b) verifier that checks that all libraries are properly linked.

[mdavidsaver]

I'd not come across buildout before. Glancing through the getting started I only see mentions of .egg, and not .whl, so I can guess at why this is causing problems for you.

I still don't see a clear solution for develop builds. The (increasingly crazy?) ideas I've had so far:

1. Allow absolute RPATH. Easy, but likely not a complete solution.
2. Make the commands in dsocmd.py aware of -e so RPATH can be computed relative to os.getcwd() and/or some explicit install location(s). More complex, and maybe not much better than absolute paths.
3. Explicit preloading w/ ctypes. Effectively "formalizing" what you are already doing.
4. Replace setup.py develop on Linux/Mac and install actual symlinks for .so/.dylib files. Where I get stuck is wanting to avoid also replacing the .egg-link with individual symlinks for .py files as well.
5. Providing some helper/launcher CLI for setting LD_LIBRARY_PATH based on foreknowledge of which DSOs will be needed. A specialized solution which might be sufficient in CI environments, but not much else.

[navytux]

( @mdavidsaver thanks for your feedback; writing just to let you know that I'm still here and need to address this issue. I will try to come back in some time )