search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.07k stars 14.13k forks source link

Switch most users from abandoned `libelf` to `elfutils` #271473

Open trofi opened 11 months ago

trofi commented 11 months ago

Describe the bug

libelf did not see an update since 2014: https://github.com/Distrotech/libelf. elfutils is a popular API compatible (?) ELF parser that most users expect.

I suggest switching most users (or maybe even all?) from libelf to elfutils. A few examples which should very likely use elfutils:

$ git grep -F ', libelf'
...
pkgs/development/libraries/mesa/default.nix:, libelf, libvdpau
pkgs/os-specific/linux/iproute/default.nix:, db, iptables, libelf, libmnl
pkgs/os-specific/linux/kernel/manual-config.nix:, libelf, cpio, elfutils, zstd, python3Minimal, zlib, pahole, kmod, ubootTools

Priorities

Add a :+1: reaction to issues you find important.

philiptaron commented 11 months ago

Here's a cheap-and-cheerful list of items that might need attention. There are about 40.

$ rg libelf --sort=path -g '*.nix' -m1 -w
pkgs/applications/audio/lv2lint/default.nix
1:{ stdenv, lib, fetchurl, pkg-config, meson, ninja, lv2, lilv, curl, libelf }:

pkgs/applications/emulators/dynamips/default.nix
5:, libelf

pkgs/applications/emulators/mgba/default.nix
8:, libelf

pkgs/applications/misc/elf-dissector/default.nix
9:, libelf

pkgs/applications/misc/opencpn/default.nix
18:, libelf

pkgs/applications/networking/ids/suricata/default.nix
13:, libelf

pkgs/applications/science/electronics/nvc/default.nix
10:, libelf

pkgs/applications/virtualization/libnvidia-container/default.nix
6:, libelf

pkgs/development/compilers/clasp/default.nix
60:    libelf

pkgs/development/compilers/wcc/default.nix
1:{ lib, stdenv, fetchFromGitHub, capstone, libbfd, libelf, libiberty, readline }:

pkgs/development/embedded/avrdude/default.nix
1:{ lib, stdenv, fetchFromGitHub, cmake, bison, flex, libusb-compat-0_1, libelf

pkgs/development/libraries/glib/default.nix
11:, libiconv, zlib, libffi, pcre2, libelf, gnome, libselinux, bash, gnum4, gtk-doc, docbook_xsl, docbook_xml_dtd_45, libxslt

pkgs/development/libraries/libdwarf/20210528.nix
1:{ callPackage, zlib, libelf }:

pkgs/development/libraries/libdwarf/common.nix
1:{ lib, stdenv, fetchurl, buildInputs, hash, version, libelf, url, knownVulnerabilities }:

pkgs/development/libraries/libelf/default.nix
11:  pname = "libelf";

pkgs/development/libraries/mesa/default.nix
6:, libelf, libvdpau

pkgs/development/libraries/spdk/default.nix
16:, libelf

pkgs/development/rocm-modules/5/rocm-runtime/default.nix
10:, libelf

pkgs/development/tools/analysis/hotspot/default.nix
14:, libelf

pkgs/development/tools/misc/prelink/default.nix
5:, libelf

pkgs/development/tools/misc/vtable-dumper/default.nix
1:{ lib, stdenv, fetchFromGitHub, libelf }:

pkgs/development/tools/profiling/EZTrace/default.nix
5:  libelf,

pkgs/development/tools/profiling/malt/default.nix
3:, cmake, nodejs, libelf, libunwind

pkgs/development/tools/simavr/default.nix
1:{ lib, stdenv, makeSetupHook, fetchFromGitHub, libelf, which, pkg-config, freeglut

pkgs/games/steam/fhsenv.nix
103:    libelf

pkgs/os-specific/bsd/freebsd/default.nix
314:      rm ''${!outputDev}/0-include/libelf.h

pkgs/os-specific/linux/dpdk/default.nix
5:, libbsd, numactl, libbpf, zlib, libelf, jansson, openssl, libpcap, rdma-core

pkgs/os-specific/linux/iproute/default.nix
3:, db, iptables, libelf, libmnl

pkgs/os-specific/linux/kernel/manual-config.nix
2:, libelf, cpio, elfutils, zstd, python3Minimal, zlib, pahole, kmod, ubootTools

pkgs/os-specific/linux/ndiswrapper/default.nix
1:{ lib, stdenv, fetchurl, kernel, perl, kmod, libelf }:

pkgs/os-specific/linux/odp-dpdk/default.nix
14:, libelf

pkgs/servers/frr/clippy-helper.nix
11:, libelf

pkgs/servers/frr/default.nix
18:, libelf

pkgs/tools/misc/rmlint/default.nix
8:, libelf

pkgs/tools/security/chipsec/default.nix
5:, libelf
Artturin commented 11 months ago

Arch has the libelf package as an output of the elfutils package

https://archlinux.org/packages/core/x86_64/libelf/

https://gitlab.archlinux.org/archlinux/packaging/packages/elfutils/-/blob/main/PKGBUILD?ref_type=heads#L142

Atemu commented 11 months ago

If it's API-compatible, I'd be in favour of removing the old, putting an alias in its place and replacing the remaining usages of libelf with elfutils.

philiptaron commented 11 months ago

I'm taking a crack at this.

philiptaron commented 11 months ago

I messed up the PR and made it unable to continue. I'll open a new one and get it right, hopefully.

philiptaron commented 11 months ago

If it's API-compatible, I'd be in favour of removing the old, putting an alias in its place and replacing the remaining usages of libelf with elfutils.

It actually requires zstd, so it's not drop-in replacement, as I see it.

Atemu commented 11 months ago

WDYM by "require"? As a library at build time? As an executable at build time?

trofi commented 11 months ago

ius on "Exotic Nix Targets" pointed out that elfutils is marked broken on pkgsStatic (elfutils loads backends dynamically).

That might limit the scope of transition to packages that already assume dynamic linking, like mesa. Or there is a chance to make the dependency optional just for dynamic case.

philiptaron commented 11 months ago

WDYM by "require"? As a library at build time? As an executable at build time?

I'm not totally sure. It's in the buildInputs for pkgs/development/tools/misc/elfutils/default.nix, and when I try to make the switch to elfutils in some packages, like mgba, the feature detection doesn't fire until zstd is added to the consuming package.

The configure.ac from the elfutils library is pretty involved. Here's the summary from the end of the file:

AC_MSG_NOTICE([
=====================================================================
        elfutils: ${PACKAGE_VERSION} (eu_version: ${eu_version})
=====================================================================

    Prefix                             : ${prefix}
    Program prefix ("eu-" recommended) : ${program_prefix}
    Source code location               : ${srcdir}
    Maintainer mode                    : ${enable_maintainer_mode}
    build arch                         : ${ac_cv_build}

    CFLAGS=${CFLAGS}
    CXXFLAGS=${CXXFLAGS}

  RECOMMENDED FEATURES (should all be yes)
    gzip support                       : ${with_zlib}
    bzip2 support                      : ${with_bzlib}
    lzma/xz support                    : ${with_lzma}
    zstd support                       : ${with_zstd}
    zstd compression support           : ${with_zstd_compress}
    libstdc++ demangle support         : ${enable_demangler}
    File textrel check                 : ${enable_textrelcheck}
    Symbol versioning                  : ${enable_symbol_versioning}

  NOT RECOMMENDED FEATURES (should all be no)
    Experimental thread safety         : ${use_locks}
    install elf.h                      : ${install_elfh}

  OTHER FEATURES
    Deterministic archives by default  : ${default_ar_deterministic}
    Native language support            : ${USE_NLS}
    Extra Valgrind annotations         : ${use_vg_annotations}
    libdebuginfod client support       : ${enable_libdebuginfod}
    Debuginfod server support          : ${enable_debuginfod}
    Default DEBUGINFOD_URLS            : ${default_debuginfod_urls}

  EXTRA TEST FEATURES (used with make check)
    have bunzip2 installed (required)  : ${HAVE_BUNZIP2}
    have zstd installed                : ${HAVE_ZSTD}
    C++11                              : ${HAVE_CXX11}
    debug branch prediction            : ${use_debugpred}
    gprof support                      : ${use_gprof}
    gcov support                       : ${use_gcov}
    run all tests under valgrind       : ${use_valgrind}
    gcc undefined behaviour sanitizer  : ${use_undefined}
    gcc address sanitizer              : ${use_address}
    clang memory sanitizer             : ${use_msan}
    use rpath in tests                 : ${tests_use_rpath}
    test biarch                        : ${utrace_cv_cc_biarch}
])

ius on "Exotic Nix Targets" pointed out that elfutils is marked broken on pkgsStatic (elfutils loads backends dynamically).

Here's the change that marked it as broken, from 2022. It references a thread from almost a decade ago. I wonder if that's still the case, and whether the suggested change is possible.

I note that configure.ac, linked above, appears to support BUILD_STATIC.

Atemu commented 11 months ago

In that case we might need to add zstd to elfutils' propagatedNativeBuildInputs.

philiptaron commented 11 months ago

propagatedNativeBuildInputs

I would think if it's also a runtime dependency (as buildInputs suggests that it is?) that it ought to go into propagatedBuildInputs?

I use this gist to understand this business. But I definitely don't have it on lockdown.

Atemu commented 11 months ago

It depends on what zstd is being used here; as a library, executable or both.

Artturin commented 11 months ago

zstd is only a build dependency on arch(and others) and it's not in the pc file therefore it's not necessary to propagate it

Artturin commented 11 months ago

Nevermind I was looking at libelf

the elfutils libelf.pc has zlib and libzstd in requires.private so it still shouldn't be necessary to propagate them?

prefix=/nix/store/3f2sdbc2lsapj5i0qvdd31d0chycc2ya-elfutils-0.190
exec_prefix=${prefix}
libdir=/nix/store/3f2sdbc2lsapj5i0qvdd31d0chycc2ya-elfutils-0.190/lib
includedir=/nix/store/k1hyq2bx37083ay5q0ryf7lw90g9z1a7-elfutils-0.190-dev/include

Name: libelf
Description: elfutils libelf library to read and write ELF files
Version: 0.190
URL: http://elfutils.org/

Libs: -L${libdir} -lelf
Cflags: -I${includedir}

Requires.private: zlib libzstd
philiptaron commented 11 months ago

Here's an example of a patch that failed to build with the trivial change to just use elfutils: odp-dpdk-elfutils.patch

diff --git a/pkgs/os-specific/linux/odp-dpdk/default.nix b/pkgs/os-specific/linux/odp-dpdk/default.nix
index 7ac560824db8..f7b6276eade7 100644
--- a/pkgs/os-specific/linux/odp-dpdk/default.nix
+++ b/pkgs/os-specific/linux/odp-dpdk/default.nix
@@ -11,7 +11,7 @@
 , openssl
 , zlib
 , libbsd
-, libelf
+, elfutils
 , jansson
 , libnl
 }:
@@ -38,7 +38,7 @@ stdenv.mkDerivation rec {
     openssl
     zlib
     libbsd
-    libelf
+    elfutils
     jansson
     libbpf
     libnl

It fails with this:

checking for libdpdk... no
configure: error: in `/build/odp-dpdk-1.42.0.0_DPDK_22.11':
configure: error: Could not locate system DPDK library directory

Which comes from odp_dpdk.m4:

# _ODP_DPDK_LEGACY_SYSTEM(ACTION-IF-FOUND, ACTION-IF-NOT-FOUND)
# ------------------------------------------------------------------------
# Locate DPDK installation
AC_DEFUN([_ODP_DPDK_LEGACY_SYSTEM], [dnl
    DPDK_CFLAGS="-isystem /usr/include/dpdk"
    DPDK_LDFLAGS=""
    DPDK_LIB_PATH="`$CC $AM_CFLAGS $CFLAGS $AM_LDFLAGS $LDFLAGS --print-file-name=libdpdk.so`"
    if test "$DPDK_LIB_PATH" = "libdpdk.so" ; then
        DPDK_LIB_PATH="`$CC $AM_CFLAGS $CFLAGS $AM_LDFLAGS $LDFLAGS --print-file-name=libdpdk.a`"
        AS_IF([test "$DPDK_LIB_PATH" = "libdpdk.a"],
           [AC_MSG_FAILURE([Could not locate system DPDK library directory])])
    else
        DPDK_SHARED=yes
    fi
    DPDK_LIB_PATH=`AS_DIRNAME(["$DPDK_LIB_PATH"])`
    DPDK_PMD_PATH="$DPDK_LIB_PATH"
    AS_IF([test "x$DPDK_SHARED" = "xyes"],
            [AC_MSG_NOTICE([Using shared DPDK library found at $DPDK_LIB_PATH])],
            [AC_MSG_NOTICE([Using static DPDK library found at $DPDK_LIB_PATH])])
    _ODP_DPDK_CHECK([$DPDK_CFLAGS], [$DPDK_LDFLAGS], [$1], [$2])
])

I don't understand why this is.

As soon as zstd is added to the buildInputs, it's able to find dpdk, and everything proceeds apace.

philiptaron commented 8 months ago

For those subscribed to this issue, I'd appreciate reviews and merging of #290034.

nixos-discourse commented 8 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/3523

lolbinarycat commented 7 months ago

ius on "Exotic Nix Targets" pointed out that elfutils is marked broken on pkgsStatic (elfutils loads backends dynamically).

That might limit the scope of transition to packages that already assume dynamic linking, like mesa. Or there is a chance to make the dependency optional just for dynamic case.

technically, shouldn't it be possible for a statically linked library to call dlopen?

trofi commented 7 months ago

On glibc it might work today, but it will change eventually: https://sourceware.org/glibc/wiki/FAQ "In glibc 2.27, released in 2018, support for statically linked programs that call dlopen was deprecated. The intent was to simplify support for statically linked programs and avoid the problematic use cases that users created by mixing the two linkage models. Users should either link statically or dynamically, picking the model that best meets their needs while avoiding a hybrid mix of the two."

Static musl (which is what pkgsStatic on nixpkgs uses) would always fail at runtime on dlopen() calls: https://git.musl-libc.org/cgit/musl/tree/src/ldso/dlopen.c