[liburing] build failure at link stage

0xg0nz0 commented 2 months ago

Operating system

Ubuntu 24.04

Compiler

clang 18.1.3

Steps to reproduce the behavior

Add a dependency on liburing, and rather than compiling with gcc, compile with clang:

CC=/usr/bin/clang CXX=/usr/bin/clang++ vcpkg install

Note I use manifest mode. My vcpkg.json:

{
    "dependencies": [
        "ada-url",
        "catch2",
        "fmt",
        "gperftools",
        "liburing",
        "nanobench",
        "nlohmann-json",
        "spdlog"
    ]
}

and vcpkg-configuration.json:

{
    "default-registry": {
        "kind": "git",
        "baseline": "4e08971f3ddc13018ca858a692efe92d3b6b9fce",
        "repository": "https://github.com/microsoft/vcpkg"
    },
    "registries": [
        {
            "kind": "artifact",
            "location": "https://github.com/microsoft/vcpkg-ce-catalog/archive/refs/heads/main.zip",
            "name": "microsoft"
        }
    ]
}

With gcc it's fine; with clang there's a linker error:

/usr/bin/ld: register.os: in function `io_uring_unregister_buf_ring':
/usr/local/vcpkg/buildtrees/liburing/arm64-linux-dbg/src/register.c:324:(.text+0xa8c): undefined reference to `memset'
/usr/bin/ld: register.os: in function `io_uring_buf_ring_head':
/usr/local/vcpkg/buildtrees/liburing/arm64-linux-dbg/src/register.c:331:(.text+0xae0): undefined reference to `memset'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:82: liburing.so.2.7] Error 1
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: register.os: in function `io_uring_unregister_buf_ring':
/usr/local/vcpkg/buildtrees/liburing/arm64-linux-dbg/src/register.c:324:(.text+0xa8c): undefined reference to `memset'
/usr/bin/ld: register.os: in function `io_uring_buf_ring_head':
/usr/local/vcpkg/buildtrees/liburing/arm64-linux-dbg/src/register.c:331:(.text+0xae0): undefined reference to `memset'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:85: liburing-ffi.so.2.7] Error 1
make: *** [Makefile:10: all] Error 2

Note I was able to build and link all other dependencies using clang without any issues, and from what I can tell from the liburing source repository on GitHub it's supposed to support both gcc and clang.

Failure logs

See error log above.

Additional context

More context including how I am currently working around this issue in below discussion answer:

https://github.com/microsoft/vcpkg/discussions/38030

WangWeiLin-MV commented 2 months ago

Add optimization option (-O3) could fix this error, but I'm not quite sure how the difference with GCC occurred. I would be grateful if you could help find a reproducible case and report it upstream (Clang).

0xg0nz0 commented 2 months ago

I will dig into it a bit more and come back to you, @WangWeiLin-MV. What is odd to me is that liburing is supposed to support both, which makes me wonder if it's specifically vcpkg + clang build. I'll take a closer look at the generated build files and see if I can figure out what's going on. Unfortunately it's a private repo and a fairly complex build, so isolating a reproducible case might be tough; I'll see if I can diagnose locally.

dg0yt commented 2 months ago

Unfortunately, the issue lacks the command link which invokes the linker.

0xg0nz0 commented 2 months ago

Here is the full set of build log files:

build-arm64-linux-dbg-out.log build-arm64-linux-dbg-err.log

The link is using mold (alternative linker) but equally the problem happens with lld:

/usr/bin/clang -fPIC -g -Wno-unused-parameter -DLIBURING_INTERNAL  -nostdlib -nodefaultlibs -ffreestanding -fno-builtin -fno-stack-protector -shared -Wl,--version-script=liburing-ffi.map -Wl,-soname=liburing-ffi.so.2 -o liburing-ffi.so.2.7 setup.os queue.os register.os syscall.os version.os nolibc.os ffi.os -Wl,-z,defs -L/workspaces/diabolica/build/vcpkg_installed/arm64-linux/debug/lib -L/workspaces/diabolica/build/vcpkg_installed/arm64-linux/debug/lib/manual-link -fuse-ld=mold -nostdlib -nodefaultlibs /usr/bin/../lib/gcc/aarch64-linux-gnu/13/libgcc.a

What's even stranger is this, my triplet definition:

set(VCPKG_CMAKE_SYSTEM_NAME Linux)
set(VCPKG_TARGET_ARCHITECTURE arm64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

Why is it creating dynamic libs at all if library linkage is static?

So I looked at the configuration logs:

config-arm64-linux-rel-out.log config-arm64-linux-dbg-out.log

which have multiple errors:

ERROR: unknown option --disable-silent-rules
ERROR: unknown option --verbose
ERROR: unknown option --disable-shared
ERROR: unknown option --bindir=${prefix}/../tools/liburing/debug/bin
ERROR: unknown option --sbindir=${prefix}/../tools/liburing/debug/sbin

So this may be something wrong in vcpkg's portfile for liburing. Probably what's happening is gcc is able to build dynamic libraries without errors while clang cannot, but the bottom line is it should not be building dynamic in the first place. Taking a look at the port, this looks suspicious:

https://github.com/microsoft/vcpkg/blob/master/ports/liburing/fix-configure.patch

It's patching in --enabled-shared and --enable-static but the error implies that rather than --enable-static the name of the flag should be --disable-shared?

+  --enable-shared) ENABLE_SHARED=1
+  ;;
+  --enable-static) ENABLE_SHARED=0
+  ;;

0xg0nz0 commented 2 months ago

Until above fixed, the following workaround (as @WangWeiLin-MV suggested) fixes the issue:

# set compiler to clang
set(CMAKE_C_COMPILER /usr/bin/clang CACHE FILEPATH "C compiler" FORCE)
set(CMAKE_CXX_COMPILER /usr/bin/clang++ CACHE FILEPATH "C++ compiler" FORCE)

# use -O3 to work around problems with liburing build
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O3")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")

# set the linker to mold
set(CMAKE_LINKER /usr/local/bin/mold CACHE FILEPATH "Linker" FORCE)
set(COMMON_LINKER_FLAGS "-fuse-ld=mold")
set(CMAKE_EXE_LINKER_FLAGS "${COMMON_LINKER_FLAGS}" CACHE STRING "Executable linker flags" FORCE)
set(CMAKE_MODULE_LINKER_FLAGS "${COMMON_LINKER_FLAGS}" CACHE STRING "Module linker flags" FORCE)
set(CMAKE_SHARED_LINKER_FLAGS "${COMMON_LINKER_FLAGS}" CACHE STRING "Shared linker flags" FORCE)

I also found an explanation for the clang / gcc difference:

https://discourse.llvm.org/t/disable-memset-synthesization/28559

clang follows gcc in requiring you to supply your own memset and other built-ins if you use -ffreestanding which is exactly what liburing does, since it's very low-level code. When it linked the shared library, those synthesized memset calls described above are not found. There are gcc-specific workarounds for this scenario, but the flags are different. Other posts I found referred to the -O3 workaround, which likely as a side-effect of optimization removes the synthesized memset calls and fixes the linking issue. However, no shared library should be linked given this triplet, so ideally that should be fixed in the port to avoid unnecessary forcing of C code optimization.

dg0yt commented 2 months ago

-nostdlib -nodefaultlibs -ffreestanding -fno-builtin

AFAIU these flags cause the problem because they take away implementations of some symbols. The flags don't come from vcpkg.

-O3 might work only via a side effect. This is not reliable. And the flags harms the purpose of the debug build type.
mold adds a another variable to the mix. Using the same compiler settings, the symbols should still be needed at the linking step. IDK how mold resolves them. Does it ignore (some of) the flags?

If liburing can't use the "gcc-specific workarounds" for clang, it should perhaps not apply some of these flags.

dg0yt commented 2 months ago

It's patching in --enabled-shared and --enable-static but the error implies that rather than --enable-static the name of the flag should be --disable-shared?

vcpkg_configure_make use all combinations of enabled/disabled with static/shared, as demanded by the triplet.

microsoft / vcpkg