Explore MUSL to avoid GLIBC requirement

[sunchao]

What is the problem the feature request solves?

Currently we require dynamically linked glibc on the host machine which isn't desirable since it requires certain version of glibc, otherwise the native Comet lib won't be able to load. In particular, the glibc version used to compile and build Comet must be compatible (i.e., older) than the version used to execute Comet.

Another option is to explore x86_64-unknown-linux-musl compile target, which statically links against the MUSL libc and can effectively remove the above constraints. However, there are some reports that MUSL can cause performance degradation, see here and here. We may need to do some benchmark to evaluate.

Describe the potential solution

Evaluate MUSL and check if it can be a potential candidate to co-exist (or replace) with the current GLIBC approach.

Additional context

No response

[advancedxy]

In particular, the glibc version used to compile and build Comet must be compatible (i.e., older) than the version used to execute Comet.

In the native world, it usually implies to build and release the dynamic lib on a decent old enough Linux distro. In that way, the built lib could be executed on newer Linux hosts. That' pretty much the simplest way to archive compatibility. And maybe we can reuse rustc's docker image to target lower glibc: https://github.com/rust-lang/rust/blob/master/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile

[snmvaughan]

I tend to agree with using a lower target glibc for compatibility.

If we want to consider MUSL we should also plan on benchmarking different combinations of target (glibc vs musl) and alloc (jemalloc vs system alloc) using a benchmark like TPC-DS.

[sunchao]

Yes, this ticket belongs to the "good to have" category IMO. One disadvantage of having to stick to the lowest common denominator for GLIBC is that certain crates may not able to compile. For instance, we ran into issues with mimalloc when building on GLIBC 2.17 IIRC.

[andygrove]

This recent blog post is worth a read. It talks about some performance issues with musl's default allocator and how switching to jemalloc helps.

https://blog.sdf.com/p/fast-development-in-rust-part-one

[viirya]

I remember we used jemalloc at the beginning, but we encountered some libraries issues from it and switched it back.

[advancedxy]

hhhm, I recently know there's a project called https://github.com/rust-cross/cargo-zigbuild which is able to cross compile rust project to different target including various glibc versions by using zig as a linker. I think we can explore this option to cross compile to low version glibc.

Ref: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html see the glibc part.