Fix broken dependency to libudev

[kunaltyagi]

• [x] I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Details about system ( conda info ):

``` $ conda info libusb libusb 1.0.24 h18f079d_4 conda-forge ```

How libusb-1.0.pc looks like

Name: libusb-1.0
Description: C API for USB device access from Linux, Mac OS X, Windows, OpenBSD/NetBSD and Solaris userspace
Version: 1.0.24
Libs: -L${libdir} -lusb-1.0
Libs.private: -ludev -lrt -lpthread 
Cflags: -I${includedir}/libusb-1.0

How it should instead look like:

Name: libusb-1.0
Description: C API for USB device access from Linux, Mac OS X, Windows, OpenBSD/NetBSD and Solaris userspace
Version: 1.0.24
Libs: -L${libdir} -lusb-1.0 -ludev
Libs.private: -lrt -lpthread 
Cflags: -I${includedir}/libusb-1.0

cc: @Tobias-Fischer

[ryanvolz]

I'm happy to accept a patch. Has this issue been reported upstream?

[kunaltyagi]

I don't know if this is an upstream issue because I face it only on conda.

Either ubuntu (18.04 and later) have a patch for this and has not been upstreamed, or it's a conda issue.

[kunaltyagi]

But it does look like the issue is in upstream: https://github.com/libusb/libusb/blob/master/libusb-1.0.pc.in

[Tobias-Fischer]

Debians pc file looks the same:

Libs: -L${libdir} -lusb-1.0
Libs.private: -ludev -lpthread 

But yeah, as @kunaltyagi we are facing the same issue in RoboStack (cc @wolfv) where we have patched several packages to also link to libudev as otherwise we got compile errors.

[ryanvolz]

We're not doing anything tricky in the recipe, so I'm inclined to believe that this exists outside of conda too but nobody has been bothered enough to do anything about it. Anyway, I'm happy to fix it here.

[ryanvolz]

Hmm, do you have a link for what the linking error is in your downstream packages? From looking at the libusb source, it's not clear to me that consumer libraries should need to link against libudev.

I know that with building conda packages downstream from libusb, the fact that it depends on libudev as a CDT package means that you have to manually add the CDT as a dependency for anything that wants to use libusb. See, e.g. https://github.com/conda-forge/rtl-sdr-feedstock/blob/master/recipe/meta.yaml. You'll also likely need a yum_requirements.txt to provide libudev for the testing phase. But all that doesn't require a change to the pkgconfig file. Maybe that is what you're seeing? (Side note, I'm hoping to make a proper conda package for libudev so that these CDT issues and others will go away).

[Tobias-Fischer]

We get these kinds of linking issues:

/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `memcpy@GLIBC_2.14'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `clock_gettime@GLIBC_2.17'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `fcntl64@GLIBC_2.28'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `getrandom@GLIBC_2.25'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `getauxval@GLIBC_2.16'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `reallocarray@GLIBC_2.26'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `gettid@GLIBC_2.30'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `name_to_handle_at@GLIBC_2.14'
/opt/miniconda3/envs/ros1/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/miniconda3/envs/ros1/lib/./libudev.so.0: undefined reference to `__explicit_bzero_chk@GLIBC_2.25'
collect2: error: ld returned 1 exit status

We pull in the libudev cdt package, so that's not the issue :(.

[ryanvolz]

So it's expecting to find some relatively new glibc symbols (gettid@GLIBC_2.30) that the CDT libudev.so.0 shouldn't need (being from Centos 6 which has glibc 2.12), thus I expect something is going wrong with https://github.com/conda-forge/libusb-feedstock/blob/master/recipe/post-link.sh and that's not actually libudev.so.0 but a link to the system's libudev.so.1. Adding an explicit link to libudev probably causes it to find the correct libudev.so.0 which doesn't have the symbol problem. Argh, this is why I hate these CDTs and the ugly hacks that are needed to make it work.

If that's right, then the options are:

0. Continue to work around this issue in the manner that you're currently using, while waiting for a proper fix
1. Diagnose what's going wrong with the post-link script and fix it
2. Remove the need for the post-link script by building separate libusb packages that link against either libudev.so.0 or libudev.so.1 and somehow install the right one on a user's system depending on what they have
3. Make a proper libudev conda package and switch to that, avoiding all these headaches

As I said, I want to do (3) soon anyway, so maybe that's the best course of action. It might not be quick though, and I don't know how big of a hassle the workaround is for you in the near term. Options (1) or (2) might be quicker.

[ryanvolz]

There's now a conda-forge libudev package that can be used with at least glibc 2.17 (cos7 builds). This means any downstream cos7 builds can just depend on libusb and not worry, but cos6 builds will still need to install the libudev CDT when compiling against libusb as has been true. However, the bad linking with those cos6 builds that probably happened because of the post-link script should be fixed, because the post-link script is gone! Let me know if you still see any issues.

[Tobias-Fischer]

Cool, thanks @ryanvolz! Out of interest, where is the libudev package being built? I cannot seem to find the feedstock ..

[ryanvolz]

It's part of systemd, so it's one output of the systemd feedstock!

[kunaltyagi]

Thanks a lot @ryanvolz

Which package should we install in order to test this?

[ryanvolz]

Thanks a lot @ryanvolz

Which package should we install in order to test this?

I'm not sure what you mean. If you already have a package built against libusb, you should be able to install against the new libusb without issue, and it will run against the new libudev1 package when glibc>=2.17 is available. If you're building against the new libusb package, you'll have to use different strategies depending on the build system image. If that is cos7, then just use libusb as a host dependency and don't worry about libudev. If the system image is cos6 (i.e. building for linux-64 right now), then use libusb as a host dependency and put {{ cdt('libudev-devel') }} as a build dependency (as has been the case for a while now). If that was giving you linking issues before, it should no longer do that.

In case it isn't clear: there are now two versions of libusb for linux-64. One built with the cos6 image and linked against the system libudev.so.0, and one built with the cos7 image and linked against the conda-forge libudev.so.1. You have to build against the first one with the default cos6 image, but users will probably get the second version of libusb when they install because they will most likely have glibc>=2.17.

[Tobias-Fischer]

Hi @ryanvolz - unfortunately I just ran into this issue again:

: && $BUILD_PREFIX/bin/x86_64-conda-linux-gnu-c++ -fPIC -fdiagnostics-color=always -DBOOST_ERROR_CODE_HEADER_ONLY -D__STDC_FORMAT_MACROS=1  -O3 -DNDEBUG  -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/tobias/mambaforge/lib -Wl,-rpath-link,/home/tobias/mambaforge/lib -L/home/tobias/mambaforge/lib   -Wl,--as-needed -Wl,--no-undefined -Wl,--no-allow-shlib-undefined -shared -Wl,-soname,libmrpt-comms.so.2.1 -o lib/libmrpt-comms.so.2.1.3 libs/comms/CMakeFiles/comms.dir/src/CClientTCPSocket.cpp.o libs/comms/CMakeFiles/comms.dir/src/CInterfaceFTDI_LIN.cpp.o libs/comms/CMakeFiles/comms.dir/src/CInterfaceFTDI_common.cpp.o libs/comms/CMakeFiles/comms.dir/src/CSerialPort.cpp.o libs/comms/CMakeFiles/comms.dir/src/CServerTCPSocket.cpp.o libs/comms/CMakeFiles/comms.dir/src/CServerTCPSocket_common.cpp.o libs/comms/CMakeFiles/comms.dir/src/comms-precomp.cpp.o libs/comms/CMakeFiles/comms.dir/src/net_utils.cpp.o libs/comms/CMakeFiles/comms.dir/src/nodelets.cpp.o -L$PREFIX/lib -Wl,-rpath,$PREFIX/lib:$SRC_DIR/build/lib:  lib/libmrpt-io.so.2.1.3  -ludev  -lusb-1.0  lib/libmrpt-system.so.2.1.3  lib/libmrpt-containers.so.2.1.3  lib/libmrpt-core.so.2.1.3  -pthread  -lftdi1  $PREFIX/lib/libusb-1.0.so && :
$BUILD_PREFIX/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: $PREFIX/lib/libudev.so: undefined reference to `name_to_handle_at@GLIBC_2.14'
$BUILD_PREFIX/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: $PREFIX/lib/libusb-1.0.so: undefined reference to `memcpy@GLIBC_2.14'
$BUILD_PREFIX/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: $PREFIX/lib/libudev.so: undefined reference to `secure_getenv@GLIBC_2.17'
$BUILD_PREFIX/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: $PREFIX/lib/libudev.so: undefined reference to `getauxval@GLIBC_2.16'
$BUILD_PREFIX/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: $PREFIX/lib/libusb-1.0.so: undefined reference to `clock_gettime@GLIBC_2.17'

Although I use the new packages:

  + libudev                             249  h7f98852_1           conda-forge/linux-64                           Cached
  + libudev1                            249  h7f98852_1           conda-forge/linux-64                           Cached
  + libusb                           1.0.24  h4c0bc5a_105         conda-forge/linux-64                           Cached

ldd libusb-1.0.so
    linux-vdso.so.1 (0x00007ffe7738e000)
    libudev.so.1 => /home/tobias/mambaforge/conda-bld/ros-noetic-mrpt2_1637643133280/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/././libudev.so.1 (0x00007fd03c9a3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd03c956000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd03c764000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd03c9f3000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd03c759000)

Do you have any idea why that would be?

[ryanvolz]

Yeah, it's pulling in the libusb that was built with the cos7 base but are not depending on the corresponding sysroot (sysroot_linux-64 2.17), so you get the missing symbols. I'm not sure why it's getting that version of libusb, but you have two options (see my previous comment as well):

1. Add that sysroot version as a build dependency for linux64 (and do whatever else is necessary to build with the cos7 image, I don't remember if that's all).
2. Somehow force the solver to use the libusb with build number less than 100 and build with the cos6 image.

I think you probably want the first option?

[Tobias-Fischer]

Ah, I see. Thanks for that, that helps!

@wolfv: Any hint how I can either use cos7 or force the build number to be smaller than 100 using the new style boa recipes in RoboStack?

[wolfv]

we could try to pin using libusb * *_0 for example, that should give build number 0.

I think unfortunately more fancy stuff doesn't work yet. There is a fancy syntax that conda half-supports which looks like libusb[build_number="<100"] but it's not supported in conda-build and mamba/libsolv wouldn't support it either. We could make that part of our roadmap in the future though.

Maybe the libusb package could also have two outputs libusb-cos6 and libusb-cos7 to help with this selection? Just a thought.

[Tobias-Fischer]

Thanks @ryanvolz and @wolfv! Pinning to libusb * *_5 seems like a bit of a hack, but works for now. Supporting the fancy syntax in conda-build/boa/mamba would be amazing!

[ryanvolz]

Making those outputs wouldn't be hard, so I could do that if it would be useful going forward. I don't know what the latest conda-forge plans are for switching away from cos6, but maybe doing nothing until that happens is better.

[ryanvolz]

@Tobias-Fischer There's now a libusb-cos6 metapackage that you can use (see #17). Since conda-forge moved to using the cos7 docker images, it has become necessary to have something like this. I'm guessing RoboStack was already doing something similar.

Now there are two options for consumers of libusb:

1. Use the default cos6 sysroot and add libusb-cos6 # [linux and cdt_name=='cos6'] to the host dependencies in addition to libusb.
2. Use the cos7 sysroot following the standard procedure and depend on only libusb.

[Tobias-Fischer]

Awesome, thanks @ryanvolz!

[Tobias-Fischer]

@wolfv - I'm not sure how we can move RoboStack to cos7; I think it would be nice to do that soon.