musl build fails, ld.lld: undefined symbol: sendfile64

[wisonye]

Hi guys, I try to integrate zap into my project and I want to build the musl version that runs inside the Alpine docker container, but I found that it fails on the x86_64-linux-musl target build.

Here is my custom Cross-compilation build:

//
// Create musl release build
//
const musl_target: CrossTarget = CrossTarget.parse(.{
    .arch_os_abi = "x86_64-linux-musl",
}) catch unreachable;

const musl_bin = b.addExecutable(.{
    .name = SERVICE_NAME,
    .root_source_file = .{ .path = "src/main.zig" },

    //
    // Cross-compilation
    //
    .target = musl_target,

    //
    // Enable release build (best performance)
    //
    // .optimize = std.builtin.OptimizeMode.ReleaseFast,

    //
    // Enable release build (best binary size)
    //
    .optimize = std.builtin.OptimizeMode.ReleaseSmall,
});

//
// Integrate with `zap`
//
const zap = b.dependency("zap", .{
    .target = musl_target,
    .optimize = std.builtin.OptimizeMode.ReleaseSmall,
});
musl_bin.addModule("zap", zap.module("zap"));
musl_bin.linkLibrary(zap.artifact("facil.io"));

Here is the error:

zig build-exe zig-demo-service ReleaseSmall x86_64-linux-musl: error: the following command failed with 1 compilation errors:

/home/wison/my-shell/zig-nightly/zig build-exe /home/wison/zig/zig-http-server/src/main.zig /home/wison/zig/zig-http-server/zig-cache/o/b13ae56265c6f3a04c951370499bf664/libfacil.io.a /home/wison/zig/zig-http-server/zig-cache/o/b8235fee8479f607b8ae0c422b0e73e9/libfacil.io.a -lc -OReleaseSmall --cache-dir /home/wison/zig/zig-http-server/zig-cache --global-cache-dir /home/wison/.cache/zig --name zig-demo-service -target x86_64-linux-musl -mcpu x86_64 --mod zap::/home/wison/.cache/zig/p/1220e30645c293c943dccc9d4394f7088679cf0969c63a7f4b23f72e375744123729/src/zap.zig --deps zap -I /home/wison/zig/zig-http-server/zig-cache/i/faa100beff85fdd06f2eb0d8196143c1/include --listen=-
Build Summary: 51/55 steps succeeded; 1 failed (disable with --summary none)
service-build transitive failure
└─ run ls transitive failure
   └─ install zig-demo-service transitive failure
      └─ zig build-exe zig-demo-service ReleaseSmall x86_64-linux-musl 1 errors

error: ld.lld: undefined symbol: sendfile64

    note: referenced by fio.c:2691 (/home/wison/.cache/zig/p/12205e49e9c2c6f0dcab9423fc92d8f0450a4dc2706b68a6d45a448a738ebff70103/lib/facil/fio.c:2691)
    note:               /home/wison/zig/zig-http-server/zig-cache/o/a9b7fc5770a0cd3eb9b5e9619bb49ea6/fio.o:(fio_sock_sendfile_from_fd) in archive /home/wison/zig/zig-http-server/zig-cache/o/b8235fee8479f607b8ae0c422b0e73e9/libfacil.io.a

And I confirm that the libfacil.io.a exists:

-rw-r--r-- 1 wison wison 1.7M Jul  5 16:48 /home/wison/zig/zig-http-server/zig-cache/o/b8235fee8479f607b8ae0c422b0e73e9/libfacil.io.a

OS: Linux my-arch 6.1.22-1-lts #1 SMP PREEMPT_DYNAMIC Thu, 30 Mar 2023 14:10:04 +0000 x86_64 GNU/Linux

Any idea?:)

[renerocksai]

Yes, an idea: musl apparently seems to not provide sendfile64. I assume a workaround could be to patch the facil.io C lib to not use sendfile64 on musl targets. Sorry, can't go into more details, am on the move.

[wisonye]

Yes, an idea: musl apparently seems to not provide sendfile64. I assume a workaround could be to patch the facil.io C lib to not use sendfile64 on musl targets. Sorry, can't go into more details, am on the move.

Actually, I just have a look at the source code of fio.c and I think I should add the Alpine distro to the following macros to disable the sendfile usage, right?:)

/* *****************************************************************************
Internal socket flushing related functions
***************************************************************************** */

#ifndef BUFFER_FILE_READ_SIZE
#define BUFFER_FILE_READ_SIZE 49152
#endif

#if !defined(USE_SENDFILE) && !defined(USE_SENDFILE_LINUX) &&                  \
    !defined(USE_SENDFILE_BSD) && !defined(USE_SENDFILE_APPLE)
#if defined(__linux__) /* linux sendfile works  */
#define USE_SENDFILE_LINUX 1
#elif defined(__FreeBSD__) /* FreeBSD sendfile should work - not tested */
#define USE_SENDFILE_BSD 1
#elif defined(__APPLE__) /* Is the apple sendfile still broken? */
#define USE_SENDFILE_APPLE 2
#else /* sendfile might not be available - always set to 0 */
#define USE_SENDFILE 0
#endif

#endif

[renerocksai]

I would take a less invasive approach and only modify build.zig of facil.io.

I just did a quick test, putting the following into build.zig passes on the required USE_SENFILE=0 and the resulting library contains no reference to sendfile64 anymore:

    const use_musl = b.option(bool, "musl", "build for musl") orelse false;
    if (use_musl) {
        lib.defineCMacro("USE_SENDFILE", "0");
    }

I'd put it in the 'Generate flags' section.

You can then zig build -Dmusl=true. zig build --help shows the musl flag that defaults to false, too.

You would need to put a similar flag into zap's build.zig, too, that passes on the flag to the facil.io dependency.

Unfortunately, I won't have time to look into getting all of this done right now, as I'm about to start my vacation travels.

[renerocksai]

Update: I don't have experience with passing options to a dependency. It's probably best to ask on the zig discord or the zig showtime discord for help.

[renerocksai]

Update: maybe this can be a start, for zap's build.zig:

    // added stuff
    const use_musl = b.option(bool, "musl", "use musl") orelse false;
    const shared_opts = b.addOptions();
    shared_opts.addOption(bool, "musl", use_musl);
    facil_lib.addOptions("facil.io", shared_opts);

    // already in build.zig
    facil_lib.linkLibrary(facil_dep.artifact("facil.io"));

Unfortunately I can't test if this really works, rn.

[wisonye]

Update: maybe this can be a start, for zap's build.zig:

    // added stuff
    const use_musl = b.option(bool, "musl", "use musl") orelse false;
    const shared_opts = b.addOptions();
    shared_opts.addOption(bool, "musl", use_musl);
    facil_lib.addOptions("facil.io", shared_opts);

    // already in build.zig
    facil_lib.linkLibrary(facil_dep.artifact("facil.io"));

Unfortunately I can't test if this really works, rn.

Ok, thanks for the quick reply, very very appreciated :+1:

So, here is your point:

• The following code needs to be added to https://github.com/zigzap/facil.io/build.zig

  const use_musl = b.option(bool, "musl", "build for musl") orelse false;
  if (use_musl) {
      lib.defineCMacro("USE_SENDFILE", "0");
  }

  Then it makes facil static library supports the -Dmusl=true option when using zig build -Duse=true to build the facil by disabling the USE_SENDFILE macro.

• The following code needs to be added to https://github.com/zigzap/zap/build.zig

  // added stuff
  const use_musl = b.option(bool, "musl", "use musl") orelse false;
  const shared_opts = b.addOptions();
  shared_opts.addOption(bool, "musl", use_musl);
  facil_lib.addOptions("facil.io", shared_opts);
  
  // Before the following line
  // facil_lib.linkLibrary(facil_dep.artifact("facil.io"));

  It makes zap is able to pass the -Dmusl=true to facil.io's build.zig to disable the USE_SENDFILE macro.

Am I on the right way? (I believe so)

If that's what you mean, then before you make the commit to the master branch, what's the best way I can do to make my project compiles and run??? As I'm not very good at the zig package manager mechanism (cache, hash ....), I just worry that I can't make it work before you make your master branch commit:)

Of course, you mentioned you're on holiday, and that's totally fine, I can wait for that, hope you enjoy your trip:)

[wisonye]

@renerocksai Btw, I just gave it a try on the real Alpine Linux (Not the docker container one), and..... it works without complaining about any sendfile linking issue.

I git clone the zap and build the hello_json and sendfile examples without applying any modification above, it works and run's fine.

ldd hello_json
#        /lib/ld-musl-x86_64.so.1 (0x7f7e71ef8000)
#        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f7e71ef8000)

objdump -T hello_json | rg sendfile
# sendfile:     file format elf64-x86-64
# 0000000000000000      DF *UND*  0000000000000000 sendfile

ldd sendfile
#        /lib/ld-musl-x86_64.so.1 (0x7fd7b08a2000)
#        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fd7b08a2000)

objdump -T sendfile | rg sendfile
# sendfile:     file format elf64-x86-64
# 0000000000000000      DF *UND*  0000000000000000 sendfile

OS:

uname -a Linux my-alpine-linux 6.1.38-0-lts #1-Alpine SMP PREEMPT_DYNAMIC Wed, 05 Jul 2023 18:57:35 +0000 x86_64 Linux

So, why the zig build fails on the x64_64-linux-musl???

[renerocksai]

@renerocksai Btw, I just gave it a try on the real Alpine Linux (Not the docker container one), and..... it works without complaining about any sendfile linking issue.

So, why the zig build fails on the x64_64-linux-musl???

You ask questions. As if I was an Alpine Linux expert. I am not.

What I think happens when you cross-compile: zig statically links to its own provided lib-musl. Apparently, when you build on Alpine, it uses the system musl and dynamically links to it, like it would with glibc on other Linux distros. Now, I don't know if Alpine uses a specially patched version of musl or if zig uses a minimal version of musl - but obviously, "standard zig musl" and (probably) "custom" Alpine musl seem to differ.

Please feel free to join the zig or zig showtime discord. You find them here. There are incredibly knowledgeable people there, it wouldn't surprise me if they could help answer your questions in detail.

[renerocksai]

Regarding your other message: I wasn't able to really test my suggestions. But I think they should be a good starting point even if they might not work straight away.

Regarding how to approach the whole package situation. Here, git submodules would be a tad simple to use, I guess. But, here's what I would do:

• check out zap's facil.io project and make sure youre on the 0.7.5-zapped branch
• edit its build.zig
• create a .tar.gz of the entire source tree
• start a webserver on port 8000. e.g.: python3 -m http.server 8000
• in zap's build.zig.zon replace the URL of facil.io by http://localhost:8000/filename.tar.gz
• leave the hash as it is
• try to build zap
• it will complain that the hash and provide the correct one after found:
• fix the hash
• rinse and repeat

I wrote about this here

Also, here people on the zap discord (if they're online), and zig, zig showtime discord will be able to help you out

[wisonye]

Regarding your other message: I wasn't able to really test my suggestions. But I think they should be a good starting point even if they might not work straight away.

Regarding how to approach the whole package situation. Here, git submodules would be a tad simple to use, I guess. But, here's what I would do:

• check out zap's facil.io project and make sure youre on the 0.7.5-zapped branch
• edit its build.zig
• create a .tar.gz of the entire source tree
• start a webserver on port 8000. e.g.: python3 -m http.server 8000
• in zap's build.zig.zon replace the URL of facil.io by http://localhost:8000/filename.tar.gz
• leave the hash as it is
• try to build zap
• it will complain that the hash and provide the correct one after found:
• fix the hash
• rinse and repeat

I wrote about this here

Also, here people on the zap discord (if they're online), and zig, zig showtime discord will be able to help you out

Thanks for all the valuable information and tips!:)

Actually, I think I figured out what's happening there and how to fix it, here is my first try and it works:

I take a look at the musl source shipped from zig (lib/libc/musl/include/sys/sendfile.h), its content:

#ifndef _SYS_SENDFILE_H
#define _SYS_SENDFILE_H

#ifdef __cplusplus
extern "C" {
#endif

#include <features.h>
#include <unistd.h>

ssize_t sendfile(int, int, off_t *, size_t);

#if defined(_LARGEFILE64_SOURCE)
#define sendfile64 sendfile
#define off64_t off_t
#endif

#ifdef __cplusplus
}
#endif

#endif

That _LARGEFILE64_SOURCE macro is what I need, as there is NO 64bit version sendfile implementation if you look at lib/libc/musl/src/linux/sendfile.c.

So, here are the step how I prove my thought:

• git clone https://github.com/zigzap/facil.io and 0.7.5-zapped is the default branch

  Add the follow test code into build.zig to force it to produce the x64_64-linux-musl just like what I did into my project:

  const musl_target: CrossTarget = CrossTarget.parse(.{
      .arch_os_abi = "x86_64-linux-musl",
  }) catch unreachable;
  
  const lib = b.addStaticLibrary(.{
      .name = "facil.io",
      // .root_source_file = .{ .path = "src/main.zig" },
      .target = musl_target,
      .optimize = std.builtin.OptimizeMode.ReleaseSmall,
  });

  Then just run zig build (or zig build -Dtarget=x86_64-linux-musl without the above modification) on my Arch host and it fails with call to undeclared function 'sendfile64' which that's what I excepted:

  zig build
  
  # zig build-lib facil.io ReleaseSmall x86_64-linux-musl: error: error(compilation): clang failed with stderr: /home/wison/zig/facil.io/lib/facil/fio.c:2691:7: error: call to undeclared function 'sendfile64'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]

  
  

• Then, I add the following code into build.zig:

  lib.defineCMacro("_LARGEFILE64_SOURCE", null);

  
  

  Re-run zig build, it works!:)

So, I will test it locally by the way you described above, if it works, then I will fire PRs to both repos:)

Again, thanks for your time and all your help, hope you enjoy your trip:)

[wisonye]

Regarding your other message: I wasn't able to really test my suggestions. But I think they should be a good starting point even if they might not work straight away. Regarding how to approach the whole package situation. Here, git submodules would be a tad simple to use, I guess. But, here's what I would do:

• check out zap's facil.io project and make sure youre on the 0.7.5-zapped branch
• edit its build.zig
• create a .tar.gz of the entire source tree
• start a webserver on port 8000. e.g.: python3 -m http.server 8000
• in zap's build.zig.zon replace the URL of facil.io by http://localhost:8000/filename.tar.gz
• leave the hash as it is
• try to build zap
• it will complain that the hash and provide the correct one after found:
• fix the hash
• rinse and repeat

I wrote about this here Also, here people on the zap discord (if they're online), and zig, zig showtime discord will be able to help you out

Thanks for all the valuable information and tips!:)

Actually, I think I figured out what's happening there and how to fix it, here is my first try and it works:

I take a look at the musl source shipped from zig (lib/libc/musl/include/sys/sendfile.h), its content:

#ifndef _SYS_SENDFILE_H
#define _SYS_SENDFILE_H

#ifdef __cplusplus
extern "C" {
#endif

#include <features.h>
#include <unistd.h>

ssize_t sendfile(int, int, off_t *, size_t);

#if defined(_LARGEFILE64_SOURCE)
#define sendfile64 sendfile
#define off64_t off_t
#endif

#ifdef __cplusplus
}
#endif

#endif

That _LARGEFILE64_SOURCE macro is what I need, as there is NO 64bit version sendfile implementation if you look at lib/libc/musl/src/linux/sendfile.c.

So, here are the step how I prove my thought:

• git clone https://github.com/zigzap/facil.io and 0.7.5-zapped is the default branch Add the follow test code into build.zig to force it to produce the x64_64-linux-musl just like what I did into my project:

  const musl_target: CrossTarget = CrossTarget.parse(.{
    .arch_os_abi = "x86_64-linux-musl",
  }) catch unreachable;
  
  const lib = b.addStaticLibrary(.{
    .name = "facil.io",
    // .root_source_file = .{ .path = "src/main.zig" },
    .target = musl_target,
    .optimize = std.builtin.OptimizeMode.ReleaseSmall,
  });

  Then just run zig build (or zig build -Dtarget=x86_64-linux-musl without the above modification) on my Arch host and it fails with call to undeclared function 'sendfile64' which that's what I excepted:

  zig build
  
  # zig build-lib facil.io ReleaseSmall x86_64-linux-musl: error: error(compilation): clang failed with stderr: /home/wison/zig/facil.io/lib/facil/fio.c:2691:7: error: call to undeclared function 'sendfile64'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]

• Then, I add the following code into build.zig:

  lib.defineCMacro("_LARGEFILE64_SOURCE", null);

  Re-run zig build, it works!:)

So, I will test it locally by the way you described above, if it works, then I will fire PRs to both repos:)

Again, thanks for your time and all your help, hope you enjoy your trip:)

Actually, it works right now!!!

zig build -Dmusl=true -Dtarget=x86_64-linux-musl run-hello_json

>>> Enabled MUSL build: true.
 Listening on 0.0.0.0:3000

 Check out:
 http://localhost:3000/user/1   # -- first user
 http://localhost:3000/user/2   # -- second user
 http://localhost:3000/user/3   # -- non-existing user

[wisonye]

@renerocksai Ok, I think I fixed it and tested locally, could you plz add the following changes to both repo's master plz (after your trip of course:)

• https://github.com/zigzap/facil.io/blob/0.7.5-zapped/build.zig:

  // const lib = b.addStaticLibrary(.{
  // ....
  // });
  
  //
  // For MUSL build
  //
  const use_musl = b.option(bool, "musl", "build for musl") orelse false;
  if (use_musl) {
      //
      //  This macro solves the `call to undeclared function 'sendfile64';`
      //  linking error.
      //
      lib.defineCMacro("_LARGEFILE64_SOURCE", null);
  }
  
  // // Generate flags
  // var flags = std.ArrayList([]const u8).init(std.heap.page_allocator);
  // ....

  
  

• https://github.com/zigzap/zap/blob/master/build.zig:

  //
  // Pass `-Dmusl` build option to `facil.io`
  //
  const use_musl = b.option(bool, "musl", "use musl") orelse false;
  const facil_dep = b.dependency("facil.io", .{
      .target = target,
      .optimize = optimize,
      .musl = use_musl,
  });
  
  // // create a module to be used internally.
  // var zap_module = b.createModule(.{
  //     .source_file = .{ .path = "src/zap.zig" },
  // });

  
  

• https://github.com/zigzap/zap/blob/master/README.md:

  README.md should updated the following code related to user's build.zig:

  const zap = b.dependency("zap", .{
      .target = target,
      .optimize = optimize,
  
      //
      // Pass `-Dmusl=true` build option to `zap -> facil.io` if your target
      // uses `MUSL` lib, e.g. `X86_64-linux-musl`
      //
      .musl = true,
  });
  exe.addModule("zap", zap.module("zap"));
  exe.linkLibrary(zap.artifact("facil.io"));

  
  

  Or you're able to add the -Dmusl build option for your project.

  For example, you can pass different configuration to b.dependency("zap") based on the -Dmusl build option:

  const use_musl = b.option(bool, "musl", "use musl") orelse false;
  std.debug.print("\n>>> [ build ] - use_musl: {}\n", .{use_musl});
  
  const zap = if (use_musl) b.dependency("zap", .{
      .target = CrossTarget.parse(.{
          .arch_os_abi = "x86_64-linux-musl",
      }) catch unreachable,
  
      //
      // Enable release build (best performance)
      //
      // .optimize = std.builtin.OptimizeMode.ReleaseFast,
  
      //
      // Enable release build (best binary size)
      //
      .optimize = std.builtin.OptimizeMode.ReleaseSmall,
  
      //
      // Pass `-Dmusl=true` build option to `zap -> facil.io`
      //
      .musl = true,
  }) else b.dependency("zap", .{
      .target = target,
      .optimize = optimize,
  
      //
      // Pass `-Dmusl=false` build option to `zap -> facil.io`
      //
      .musl = false,
  });
  
  exe.addModule("zap", zap.module("zap"));
  exe.linkLibrary(zap.artifact("facil.io"));

  
  

[renerocksai]

@mattnite's PR referenced above fixes musl builds with a one-liner in facil.io's build.zig. Closing this. Please reopen if you feel the fix didn't address your issue..