[package] xorg/system transitively brings a libuuid from the system, with different symbols than Conan Center's libuuid

jcar87 commented 1 year ago

Description

Originally posted by @samuel-emrys in https://github.com/conan-io/conan-center-index/issues/17427#issuecomment-1545476420

To capture the issue with the current conan 2 build (summarised from this slack conversation):

This is an error where automake fails to build a test executable because it wants to use all of the deps to do so. The deps are available, however it fails with the following from config.log:

/usr/bin/ld: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/12.2.1/../../../../lib64/libSM.so: undefined reference to `uuid_unparse_lower@UUID_1.0'
/usr/bin/ld: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/12.2.1/../../../../lib64/libSM.so: undefined reference to `uuid_generate@UUID_1.0'

This is occurring because libSM is a dependency of xorg/system, which uses the system package manager to install libuuid and links libSM with the system version of libuuid. Because libuuid is also in the dependency tree of tk (from fontconfig), this is injected into the build path to enable the test executable to find the conan dependencies. The conan dependency of libuuid overrides the system dependency, but unfortunately these seem to be different libraries providing the same interface which means that the system version of libuuid contains the uuid_unparse_lower@UUID_1.0 symbol, but the conan version only contains uuid_unparse_lower - i.e. an unversioned representation of the same symbol, and so it fails to resolve.

In terms of libuuid libraries available in open source, there seem to be the following three:

(1) is the original implementation, and is currently in maintenance mode seeing no new features. (2) is a fork of (1) and is actively developed. This appears to be the variant used by at least the arch linux pacman and debian apt package maintainers. (3) appears to be an unmaintained fork of (2), and hasn't been updated in a couple of years. This is the implementation that is currently packaged with conan.

The conflict appears to be because (2) and (3) have similar interfaces but are fundamentally different. The following solutions seem like the way forward:

Create a package for the util-linux variant - util-linux-libuuid? This could set provides = "libuuid" to indicate that it provides the same interface
Re-direct the libuuid recipe to point at the sources for util-linux instead of the current link. This would be the best migratory solution given that there are a large number of packages that depend on libuuid, but perhaps not the best solution for reproducibility. Having said that, the change will be captured in a rrev so maybe this isn't that big a deal.

@prince-chrismc if you or anyone else on the conan team has a good view of a way forward here some advice would be appreciated.

Originally posted by @samuel-emrys in https://github.com/conan-io/conan-center-index/issues/17427#issuecomment-1545476420

Package and Environment Details

Operating System+version: Linux

Conan profile

TBD

Steps to reproduce

TBD

Logs

TBD

jcar87 commented 1 year ago

This is a problem that happens when both xorg/system and libuuid/* are in the same dependency graph. On Ubuntu systems at least, it has been observed that xorg/system brings the system version of libuuid (libuuid.so).

We are going to observe different side effects depending on the shared option across the dependency graph:

With *:shared=True, as recommended and done by the Conan 2.0 CI pipeline in Conan Center, will cause the Conan version to be loaded at runtime, which does not provide the symbols tagged with the version, causing the errors in this issue description.
With *:shared=False (both Conan 1.0 and Conan 2.0), libuuid.so from the system will still be loaded, but the executable is likely to also include libuuid.a from Conan. Currently, because Conan's has unversioned symbols and the system has versioned symbols, this should not cause conflicts.
In the Conan 1.0 pipeline for shared libraries, only "thisreference" is shared, so with [thisref]:shared=True will result in a copy of libuuid.a inside the shared library, and libuuid.so from the system being loaded transitively via xorg libraries - same as the second case, no symbol's being defined twice because some are version tagged and some are not. This is why the Conan 1.0 pipeline checks are not failing when we use util-linux-libuuid instead of libuuid.

Building the entire graph with *:shared=True, which we have only started doing with Conan 2.0, is what has ultimately help uncover the fact that Conan Center's libuuid is an outdated fork. And I agree we should use https://github.com/util-linux/util-linux to align ourselves with what the Linux distros are doing.

Recipes to update

Moving recipes to use util-linux-libuuid to provide libuuid introduces a second problem - there will be conflicts for any dependency graph where both util-linux-libuuid and libuuid are present. The solution should be to update ALL at once - or in quick succession. We have these open:

aaf: https://github.com/conan-io/conan-center-index/pull/18556
azure-storage-cpp: https://github.com/conan-io/conan-center-index/pull/18557
cppcommon: https://github.com/conan-io/conan-center-index/pull/18553
cpython: https://github.com/conan-io/conan-center-index/pull/18558
czmq: https://github.com/conan-io/conan-center-index/pull/18552
drogon: https://github.com/conan-io/conan-center-index/pull/18555
fontconfig: https://github.com/conan-io/conan-center-index/pull/18550
gdcm: https://github.com/conan-io/conan-center-index/pull/18211
stduuid (*and any recipe that depends on it): https://github.com/conan-io/conan-center-index/pull/18549
xapian-core: https://github.com/conan-io/conan-center-index/pull/18551

These should be migrated all at once or in very quick succession, to minimize the possibility of users experiencing a conflict due to both util-linux-libuuid and libuuid being present in the dependency graph resolved by Conan (this will be an error due to the provides field).

Additional validation required

When xorg/system is in the graph, this will solve the problem for *:shared in the sense that libuuid.so loaded from the Conan cache will now contain the versioned symbols expected by xorg and any of its dependencies as provided by the system package manager.

However before proceeding I think we need to consider that this may cause issues in the future:

A future Linux distro version may use a newer version of util-linux-libuuid than the one used by Conan Center, we may see errors like this undefined reference to 'uuid_xxx_yyy@UUID_2.xx' in the future - in this case meaning that xorg or its dependencies use a sytem version of libuuid.so that is of a newer version that contains newer symbols than the one provided by Conan.
For *:shared=False (the default for most users!) would now cause the executables to contain libuuid.a from Conan (now with the version symbols), and possibly load libuuid.so from the system via xorg. Now they will contain overlapping symbols (since both contain the same versioned symbols). At runtime this may show no errors at all, but it's not guaranteed where the symbols are loaded from. If errors do happen, they can be difficult to troubleshoot.

Ahajha commented 7 months ago

@jcar87 This should be able to be closed now, the CPython 2.0 migration was merged, which included the switch to util-linux-uuid.

Ahajha commented 1 month ago

@jcar87 Bump, I think you can close this :)

valgur commented 1 month ago

There's also a handful of recipes that likely have an unlisted dependency on libuuid, but have it accidentally provided by xorg/system, either directly or transitively. I'm not sure if it's worth fixing them, though.

Here's a list of all source archives that have an include against uuid/uuid.h, but don't list libuuid as a dependency:

aeron-1.41.4.tar.gz:
- aeron-1.41.4/aeron-driver/src/main/c/aeron_driver_context.c
apr-1.7.4.tar.bz2:
- apr-1.7.4/configure.in
- apr-1.7.4/include/arch/unix/apr_private.h.in
- apr-1.7.4/misc/unix/rand.c
boost_1_85_0.tar.gz:
- boost_1_85_0/boost/uuid/detail/md5.hpp
- boost_1_85_0/boost/uuid/detail/sha1.hpp
- boost_1_85_0/boost/uuid/basic_name_generator.hpp
- boost_1_85_0/boost/uuid/nil_generator.hpp
- boost_1_85_0/boost/uuid/random_generator.hpp
- boost_1_85_0/boost/uuid/string_generator.hpp
- boost_1_85_0/boost/uuid/uuid_hash.hpp
- boost_1_85_0/boost/uuid/uuid_io.hpp
- boost_1_85_0/boost/uuid/uuid_serialize.hpp
- boost_1_85_0/libs/serialization/performance/xml/harness.hpp
client-cpp-1.5.6.tar.gz:
- client-cpp-1.5.6/src/lib/src/policy_object.cpp
crashpad-e9937cb36cd12f24d73a07c4b91168cf1885b5db.tar.gz:
- crashpad-e9937cb36cd12f24d73a07c4b91168cf1885b5db/client/crash_report_database_mac.mm
- crashpad-e9937cb36cd12f24d73a07c4b91168cf1885b5db/snapshot/mac/process_types.cc
- crashpad-e9937cb36cd12f24d73a07c4b91168cf1885b5db/util/misc/uuid.cc
fast_io-316afccde333721b059a761b25217084e84a9ca0.tar.gz:
- fast_io-316afccde333721b059a761b25217084e84a9ca0/include/fast_io_driver/boost/uuid.h
fltk-release-1.3.9.tar.gz:
- fltk-release-1.3.9/src/Fl_Preferences.cxx
gcc-13.2.0.tar.xz:
- gcc-13.2.0/libphobos/src/std/uuid.d
hazelcast-cpp-client-5.3.0.tar.gz:
- hazelcast-cpp-client-5.3.0/hazelcast/generated-sources/src/hazelcast/client/protocol/codec/codecs.cpp
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/impl/hazelcast_client_instance_impl.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/impl/vector_clock.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/internal/nearcache/NearCacheRecord.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/internal/nearcache/impl/record/AbstractNearCacheRecord.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/member.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/membership_listener.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/protocol/ClientMessage.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/serialization/pimpl/data_output.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/serialization/serialization.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/spi/ClientContext.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/spi/impl/ClientInvocation.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/spi/impl/ListenerMessageCodec.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/spi/lifecycle_service.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/sql/hazelcast_sql_exception.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/sql/impl/query_id.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/sql/impl/sql_error.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/client/txn/TransactionProxy.h
- hazelcast-cpp-client-5.3.0/hazelcast/include/hazelcast/util/Preconditions.h
libfabric-1.18.1.tar.bz2:
- libfabric-1.18.1/prov/opx/include/rdma/opx/fi_opx_hfi1.h
- libfabric-1.18.1/prov/opx/include/rdma/opx/fi_opx_domain.h
- libfabric-1.18.1/prov/psm3/psm3/psm2.h
- libfabric-1.18.1/config.h.in
libmemcached-1.0.18.tar.gz:
- libmemcached-1.0.18/mem_config.in
llvm-13.0.0.src.tar.xz:
- llvm-13.0.0.src/lib/Support/LockFileManager.cpp
- llvm-13.0.0.src/tools/dsymutil/SymbolMap.cpp
metall-0.28.tar.gz:
- metall-0.28/include/metall/detail/uuid.hpp
miniupnp-miniupnpc_2_2_5.tar.gz:
- miniupnp-miniupnpc_2_2_5/miniupnpd/upnpevents.c
mqtt_cpp-13.2.1.tar.gz:
- mqtt_cpp-13.2.1/include/mqtt/broker/uuid.hpp
ntv2-16.1.tar.gz:
- ntv2-16.1/ajalibraries/ajabase/common/guid.h
ohNet-ohNet_1.36.5344.tar.gz:
- ohNet-ohNet_1.36.5344/thirdparty/mDNSResponder-1310.80.1/mDNSCore/mDNSEmbeddedAPI.h
- ohNet-ohNet_1.36.5344/thirdparty/mDNSResponder-1310.80.1/mDNSMacOSX/mdns_objects/mdns_dns_service.h
- ohNet-ohNet_1.36.5344/thirdparty/mDNSResponder-1310.80.1/mDNSShared/dnssd_clientstub.c
openE57-1.6.5.tar.gz:
- openE57-1.6.5/src/tools/las2e57.cpp
openldap-2.6.7.tgz:
- openldap-2.6.7/include/portable.hin
- openldap-2.6.7/libraries/liblutil/uuid.c
osrm-backend-5.27.1.tar.gz:
- osrm-backend-5.27.1/include/util/fingerprint.hpp
ozo-0.1.0.tar.gz:
- ozo-0.1.0/include/ozo/ext/boost/uuid.h
paho.mqtt.c-1.3.13.tar.gz:
- paho.mqtt.c-1.3.13/src/WebSocket.c
pcl-pcl-1.14.1.tar.gz:
- pcl-pcl-1.14.1/surface/include/pcl/surface/3rdparty/opennurbs/opennurbs_system.h
- pcl-pcl-1.14.1/surface/src/3rdparty/opennurbs/opennurbs_uuid.cpp
postgresql-15.5.tar.bz2:
- postgresql-15.5/contrib/uuid-ossp/uuid-ossp.c
- postgresql-15.5/src/include/pg_config.h.in
qt-everywhere-src-6.7.1.tar.xz:
- qt-everywhere-src-6.7.1/qtwebengine/src/3rdparty/chromium/base/profiler/module_cache_apple.cc
- qt-everywhere-src-6.7.1/qtwebengine/src/3rdparty/chromium/chrome/common/safe_browsing/mach_o_image_reader_mac_unittest.cc
- qt-everywhere-src-6.7.1/qtwebengine/src/3rdparty/chromium/third_party/crashpad/crashpad/client/crash_report_database_mac.mm
- qt-everywhere-src-6.7.1/qtwebengine/src/3rdparty/chromium/third_party/crashpad/crashpad/snapshot/mac/process_types.cc
- qt-everywhere-src-6.7.1/qtwebengine/src/3rdparty/chromium/third_party/crashpad/crashpad/util/misc/uuid.cc
- qt-everywhere-src-6.7.1/qtwebengine/src/3rdparty/chromium/third_party/dawn/third_party/dxc/lib/Support/LockFileManager.cpp
root-6-22-06.zip:
- root-6-22-06/cmake/modules/Findlibuuid.cmake
- root-6-22-06/interpreter/cling/include/cling/boost.modulemap
- root-6-22-06/interpreter/llvm/src/lib/Support/LockFileManager.cpp
sentry-native.zip:
- external/crashpad/util/misc/uuid.cc
- external/crashpad/snapshot/mac/process_types.cc
- external/crashpad/client/crash_report_database_mac.mm
zyre-2.0.1.tar.gz:
- zyre-2.0.1/Finduuid.cmake

valgur commented 1 month ago

Another possible solution for the libuuid issue is to use a non-system X11 altogether or at least substitute parts of it.

I'm not seriously suggesting replacing xorg/system, but I experimented with replacing it in full with Conan packages here: https://github.com/valgur/conan-center-index/compare/dev...valgur:conan-center-index:new/xorg And the necessary changes to swap out xorg/system on the rest of CCI: https://github.com/valgur/conan-center-index/compare/new/xorg...xorg-migrate

As you can see, swapping out just the libsm dependency that relies on libuuid is very doable with just two additional recipes required:

The recipes could be forced to package_type=shared to ensure that it provides the necessary .so files for the other system libs.

As I said, it's probably not worth the risk, though. It's not clear if FindX11.cmake would be able to find these relocated libraries correctly (I replaced it with a custom version for Conan packages) and many other build systems might hard-code these to system paths as well. On the other hand, there are currently no Conan packages that link against libsm and libice directly and there are probably barely any in general as well, so it should be doable.

My original motivation was to create a personal fork without system packages to test cross-compilation against, since xorg and other system libs cannot be installed side-by-side with the native architecture packages.

/cc @jwillikers Maybe that experiment is of interest to you since you have been working on packaging Wayland and Mesa.

conan-io / conan-center-index