shlibs functionnality should be arch dependant

[bapt]

shlibs_provides vs shlibs_requires should take in account the ABI to avoid issue between lib32 and non lib32

Also when backing up libs, we should have compat & compat32 taken in account.

[ifreund]

I've just opened #2333 which is a first step towards making pkg's shlib features work in the presence of lib32.

I believe the next step is to append the architecture of the .so file to shlib_provides/requires names. This would allow differentiating between the "native" 64 bit libfoo.so.3-amd64 and the lib32 version libfoo.so.3-i386. For backwards compatibility, we would consider shlib names missing an architecture (e.g. libfoo.so.3) as if the architecture matches that of pkg config ABI.

I think this approach is preferable over modifying the pkgbase bits of the src/ build system to append additional metadata to lib32 packages or otherwise try to solve https://bugs.freebsd.org/bugzilla//show_bug.cgi?id=265061 outside of pkg.

[kevemueller]

This is very straightforward. Please also consider future-proofing any approach taken by considering kernel modules similar to shared libraries. In the end, kernel provides a set of modules, and every kernel module has also a set it provides and depends on. Might or might not be enough to do this with a "-kmod" suffix.

[bapt]

hum I think kernel modules should use the regular provides/requires (which didn't exist at the time we the the shlibs) but yes amended with the architecture, which makes me think we should probably make the "provides/requires" analyzer kind of pluggable (lua script ?) so the ports tree or the base build system provide their own rather than hardcoding everything inside pkg.

[ifreund]

Which makes me think we should probably make the "provides/requires" analyzer kind of pluggable (lua script ?) so the ports tree or the base build system provide their own rather than hardcoding everything inside pkg.

Doesn't pkg already allow specifying provides/requires in the manifest? I don't think anything more than that would be necessary to make it possible for the ports tree/base build system to generate their own provides/requires based on the package contents.

[bapt]

yes this is how it should be done, imho, one could argue we should do the same for shlibs* aka deprecate shlibs* and use the regular provides/requires, the only reason why not to do it, that the backup library functionality depends on it. (aka we only backup libraries which are marked as "provided" there are also rooms of thinking here on how this should be done)

[bapt]

note: https://github.com/freebsd/pkg/issues/1048 this should be taken in consideration is someone is working on improving the shlibs provides/requires

[bapt]

I don't know if anyone is working on this right now, if not, I will start working on this. We have roughly 2 cases on FreeBSD where we need to amend the libs: 32bit compatibility and linux compatibility.

What I propose to do is the following, keep as is anything which matches the ABI of the host aka what we currently have. Amend with "(32)" anything which is a 32bit binary Amend with "Linux" anything which is a linux binary same wordsize as the host Amend with "Linux32" anything which is a linux binary 32bit (aiming at being installed on a 64bit host.

@emaste @evadot @ifreund @kevemueller am I missing anything ?

For example nvidia drivers whill provide:

• libvdpau_nvidia.so.1
• libnvidia-tls.so.1
• libnvidia-ml.so.1
• libnvidia-glvkspirv.so.1
• libnvidia-glsi.so.1
• libnvidia-glcore.so.1
• libnvidia-eglcore.so.1
• libnvidia-cfg.so.1
• libGLX_nvidia.so.0
• libGLESv2_nvidia.so.2
• libGLESv1_CM_nvidia.so.1
• libEGL_nvidia.so.0
• libvdpau_nvidia.so.1(32)
• libnvidia-tls.so.1(32)
• libnvidia-ml.so.1(32)
• libnvidia-glvkspirv.so.1(32)
• libnvidia-glsi.so.1(32)
• libnvidia-glcore.so.1(32)
• libnvidia-eglcore.so.1(32)
• libnvidia-cfg.so.1(32)
• libGLX_nvidia.so.0(32)
• libGLESv2_nvidia.so.2(32)
• libGLESv1_CM_nvidia.so.1(32)
• libEGL_nvidia.so.0(32)

[bapt]

linux_base-rl9 on amd64 will provide:

• libc.so.6(Linux)
• libc.so.6(Linux32)
• libgeocode-glib.so.0.0.0(Linux)
• libgeocode-glib.so.0.0.0(Linux32) ...

[evadot]

Maybe it will be easier to split the tag, something like (Linux)(32) or (Linux,32) etc ...

[ifreund]

I don't know if anyone is working on this right now, if not, I will start working on this.

I started working on this but paused work to do some higher level design thinking. I see two alternative paths we could take with this feature that I wanted to explore and discuss before starting down one of them. I'll finish up the writeup I intended to post to this issue shortly. (I should have finished it last week, but I got sick)

[bapt]

I'll wait for your proposal

[ifreund]

I see two alternative options pkg could choose for its shlib handling:

Option 1: Try to duplicate ld-elf.so search order

This is the goal pkg currently seems to be working towards (https://github.com/freebsd/pkg/issues/1048)

From rtld(1) the search order is:

1. DT_RPATH of the referencing object unless that object also contains a DT_RUNPATH tag
2. DT_RPATH of the program unless the referencing object contains a DT_RUNPATH tag
3. Path indicated by LD_LIBRARY_PATH environment variable
4. DT_RUNPATH of the referencing object
5. Hints file produced by the ldconfig(8) utility
6. The /lib and /usr/lib directories, unless the referencing object was linked using the “-z nodefaultlib” option

In order to get the shlibs_provided/shlibs_required feature closer to the behavior of ld-elf.so I think the following changes would be needed:

1. Store the full path of shared objects provided in shlibs_provided (e.g. /usr/lib/libfoo.so.1 or /usr/lib32/libfoo.so.1).

2. Store the following for each entry in shlibs_required:

   • Shared library name
   • DT_RPATH or DT_RUNPATH
   • Whether ld-elf.so or ld-elf32.so search order is used. This can be determined by comparing the architecture in the elf header to pkg->arch. For example, on an amd64 system pkg->arch will be amd64 even for lib32 packages but lib32 packages will, of course, include i386 elf objects.

3. Duplicate the ld-elf.so or ld-elf32.so search order in the solver to determine if a shlib requirement is satisfied by a given shared object path from shlibs_provided.

   • LD_LIBRARY_PATH should be ignored, so this won't be a perfect reproduction of ld-elf.so logic
   • It doesn't seem feasible to me to check DT_RPATH of the program in addition to the referencing object, so the behavior here will be subtly different compared to ld-elf.so.

If BACKUP_LIBRARIES is enabled, copy any file with a path in shlibs_provided to the compat package. I don't see a need for e.g. a separate compat32 package though I could be missing something.

Option 2: only track shared objects in "standard" search paths

Get rid of shlibs_provided/shlibs_required as a pkg feature. Instead, use the more general provides/requires mechanism.

This approach intentionally ignores DT_RPATH and DT_RUNPATH as they cannot be tracked without a special shlibs_required/shlibs_provided feature.

Define "standard" search paths based on the target system.

On FreeBSD define "standard" search paths as the output of ldconfig -r, ldconfig -32 -r, or ldconfig-linux -r depending on the elf file in question.

Note that on FreeBSD packages can add their own "standard" paths by installing a file containing a list of paths to /usr/local/libdata/ldconfig/foo.

On pkg create provides/requires entries are generated to express shlib dependencies:

1. Add a provides entry with the shared library name plus a possible libcompat suffix for any shared libraries in the "standard" search paths. For example, libfoo.so.1 for a non-libcompat shared library or libfoo.so.1-lib32 for a lib32 shared library.

2. Add a requires entry with the same format for all required libraries.

If BACKUP_LIBRARIES is enabled, copy any libraries in the "standard" search paths to the compat package. I don't see a need for e.g. a separate compat32 package though I could be missing something.

Advantages/Disadvantages

Overall the first option is more complex but gets closer to the actual runtime linker's behavior.

I think the second option likely has better UX since it uses the more general provides/requires mechanism and doesn't require the user to know the runtime linker search order to determine whether a shlibs_required is fulfilled by a given shlibs_provided.

The main downside of the second option is of course that DT_RPATH and DT_RUNPATH would need to be handled manually by the porter (e.g. by adding a explicit dependency) rather than automatically by pkg. I personally think this is a reasonable tradeoff to make. Many linux distros (e.g. Debian) discourage usage of DT_RPATH and DT_RUNPATH in their packaging. I think it would be reasonable for FreeBSD to do the same in the ports tree, especially since there is a way for packages to add to the "standard" search paths with a /usr/local/libdata/ldconfig/foo file.

Other notes:

• CheriBSD has other libcompats aside from lib32 with ld-elf64c.so and ld-elf64cb.so for example, we should avoid special casing lib32 as the only libcompat.

• The job scheduler probably should consider provides/requires and shlibs_provided/shlibs_required for job ordering if these features see widespread use. This is another minor point in favor of dropping shlibs_provided/shlibs_required and using the more general provides/requires as it would reduce complexity there.

[bapt]

I think the second case is also the less complex to undertsand/debug so I am fine with the second case, about the RUNPATH, having a special case with hard dependencies for such case is perfectly fine. right now what we have is closer to the option 2 we read the elf hints to get the known ldconfig path, and lookup there we lookup for RUNPATH as well, and this is imho fine (we could arguably drop entirely the RUNPATH but that is another storry). As for the -lib32, I think my proposal complement yours, if we can tag the shlibs with: Linux, lib32 (maybe just 32) downstreams should just be able to add their own tags.

Regarding the solver provides/requires and shlibs_provides/shlibs_required are already working the same way, the only difference is they are in 2 separated fields, which in terms of UX is kind of convenient, I don't think eliminating one would simplify anything, but if it does, then so be it.

[ifreund]

Yeah, "Option 2" is pretty much identical to what you proposed. It just spells out more of the details of how it should actually be implemented.

Regarding the solver provides/requires and shlibs_provides/shlibs_required are already working the same way, the only difference is they are in 2 separated fields, which in terms of UX is kind of convenient, I don't think eliminating one would simplify anything, but if it does, then so be it.

Indeed, I think this is mostly a UX question in the end. What would you think about using a namespace for shared library provides/requires (e.g. so, full name so:libfoo.so.1(32)) in addition to dropping shlibs_provided/shlibs_required?

This could be potentially extended in the future for pkg-config provides/requires files (pc:foo) or other things.

[bapt]

i like namespaces

[kevemueller]

When looking at pkg_elf I noticed and very much disliked ld.so.hints parsing. This is an absolutely non-portable way of handling things, but I am the 1‰ here. In my use-case the (FreeBSD) packages are created and used completely outside FreeBSD, no way I can have an ld.so.hints lying around.

Namespaces also solve my previous request for kmod dependencies. If we want to go all-in, we should also add a dyld or ld or loader namespace for the required loader.

Happy to write the Mach-O update after the work was done for ELF (I would like to keep the code close in its logic).

[kevemueller]

Btw. standard search paths are a happy route with Mach-O, as I could just handle@rpath/@loader_path variables from the otherwise absolute path library references by dropping them.