Unknown linux gets a failed to load NIF library error due to GLIBC version

[samfrench]

I have been trying out rustler with a few libraries and am getting the NIF failing to load when I am using CentOS 7. Looking into this, the targets in the application example release.yml look to all be for Ubuntu. I am wondering if this is expected to work on other linux distributions, or if Ubuntu or Ubuntu compatible ones only. I also tried on Rocky 8 - which is compatible with CentOS - and this works fine.

For more information, on startup it fails to load the NIF library.

09:13:49.336 [debug] Copying NIF from cache and extracting to /code/rustler_precompilation_example/_build/dev/lib/rustler_precompilation_example/priv/native/libexample-v0.5.0-nif-2.16-x86_64-unknown-linux-gnu.so

09:13:49.366 [warning] The on_load function for module Elixir.RustlerPrecompilationExample.Native returned:
{:error,
 {:load_failed,
  'Failed to load NIF library: \'/lib64/libc.so.6: version `GLIBC_2.28\' not found (required by /code/rustler_precompilation_example/_build/dev/lib/rustler_precompilation_example/priv/native/libexample-v0.5.0-nif-2.16-x86_64-unknown-linux-gnu.so)\''}} 

Then when I am in an iex session I also see the same thing when running the example application.

RustlerPrecompilationExample.Native.add(2, 2)

09:43:35.754 [warning] The on_load function for module Elixir.RustlerPrecompilationExample.Native returned:
{:error,
 {:load_failed,
  'Failed to load NIF library: \'/lib64/libc.so.6: version `GLIBC_2.28\' not found (required by /code/rustler_precompilation_example/_build/dev/lib/rustler_precompilation_example/priv/native/libexample-v0.5.0-nif-2.16-x86_64-unknown-linux-gnu.so)\''}} 

09:43:35.824 [warning] The on_load function for module Elixir.RustlerPrecompilationExample.Native returned:
{:error,
 {:load_failed,
  'Failed to load NIF library: \'/lib64/libc.so.6: version `GLIBC_2.28\' not found (required by /code/rustler_precompilation_example/_build/dev/lib/rustler_precompilation_example/priv/native/libexample-v0.5.0-nif-2.16-x86_64-unknown-linux-gnu.so)\''}} 

** (UndefinedFunctionError) function RustlerPrecompilationExample.Native.add/2 is undefined (module RustlerPrecompilationExample.Native is not available)
    (rustler_precompilation_example 0.5.0) RustlerPrecompilationExample.Native.add(2, 2)

I have looked into the GLIBC message and there are suggestions that this may be a different version for different operating systems and it is not advisable to necessarily upgrade as these are core to the OS and expected to be different. Looking at the version in CentOS 7 is lower than Rocky. Rocky 8 is using "2.28" which is why it works there.

ldd --version
ldd (GNU libc) 2.17

I was wondering if the GLIBC library should either be dynamically linked or a version could be looked up or specified, if any version can be used. I know I can build from source and run it using that approach, but I was looking at the precompiled versions to simplify things where possible.

Any advice on if this can work on other linux distributions or any feedback about this is appreciated.

[philss]

Hey @samfrench :wave:

I am wondering if this is expected to work on other linux distributions, or if Ubuntu or Ubuntu compatible ones only. I also tried on Rocky 8 - which is compatible with CentOS - and this works fine.

It should work on most distributions - I'm using Fedora 38 on my computer and it works fine. It is being compiled with Ubuntu because it's the default runner for Linux on GitHub Actions. But perhaps we should try to use an older version of it, in order to be compatible with more distros. I don't know exactly what are the drawbacks of that.

I was wondering if the GLIBC library should either be dynamically linked or a version could be looked up or specified, if any version can be used. I know I can build from source and run it using that approach, but I was looking at the precompiled versions to simplify things where possible.

It is already dynamically linked for targets that are compiled with GCC, but I guess the problem is that the program was compiled with a newer version that includes some API that is used by the binary, and is not available for older versions of glibc.

Any advice on if this can work on other linux distributions or any feedback about this is appreciated.

For what I could understand, I think the only possible solution is to everybody agrees to compile using an older version of a Linux distribution. But I don't think that's entirely feasible, because people may depend on newer versions of other dependencies. I'm going to research more if there is another way to set a "version compatible" of glibc.

I would say that upgrading to a newer distribution is the easiest solution, if possible.

An alternative is to precompile the project you need, using the same OS and version you need, and force Rustler to load the NIF from a path - Rustler allows that by setting the :load_from in their options. WDYT?

[samfrench]

Thank you for the response @philss. It is detailed and addresses all of the points.

That explains it about Ubuntu. I was thinking it was the most compatible or popular, but it is the default on GitHub actions.

I am mainly looking at libraries others have created, so the options would be to either compile from source and install dependencies or fork the source and precompile to use this approach. I am looking to precompile as this reduces the need for the dependencies and should make it easier to install - if the library doesn't change too often.

I think the direction at the moment longer term, is to use a more recent linux distribution as it works as expected. In the short term I can use the option of "load_from" as you suggest. I hadn't considered that, but this should allow the libraries to be installed on other linux distributions.

I am not sure on how the older or different ones should be supported and would be interested in your findings. Even if an older one is supported the same glibc is unlikely to be compatible on all relevant older ones. Potentially this is a similar problem when Ubuntu gets upgraded as that could use a newer version and others still be relying on 2.28.

[philss]

Just to let you know that the Variants feature has landed in v0.7. This is probably not going to solve the issue, but it is a tool for maintainers to keep compatibility while offering modern versions by default.

I'm closing the issue now.

[samfrench]

Thank you for this. I'll take a look.