Precompile Linux binaries with older glibc to ensure compatibility with older distros

kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB

Other

484 stars 39 forks source link

Precompile Linux binaries with older glibc to ensure compatibility with older distros #87

Closed ascorbic closed 2 years ago

ascorbic commented 2 years ago

Hi. Thanks for maintaining this project. Currently prebuilt Linux binaries are built on a system with glibc 2.28, which means they won't work on Amazon Linux 2 as used on AWS Lambda which has glibc 2.26. This means that any code that is to be deployed to a Lambda needs to build the package from source on a system with glibc 2.26, which is not ideal. If the libraries were built on a system with glibc 2.26 (such as Ubuntu Xenial) then the binaries would be forwards-compatible and packages could be built on newer systems. Is this something that might be possible?

kriszyp commented 2 years ago

I'd be glad to do this, but I am not very familiar with this process, so certainly would appreciate any pointers. I am currently running Linux Mint on a computer to prebuild the linux binaries. If I understand correctly, I think I need to:

Find and download glibc or libc6 version 2.26. However, the closest I can find is 2.23 or 2.27 (not sure which would be better) in: http://archive.ubuntu.com/ubuntu/pool/main/g/glibc/
Once I have extracted this to /opt, point to it with a LIBRARY_PATH env var.
Recompile the lmdb-store binaries.

Does that sound right, or is there a specific distribution you would recommend I use?

ascorbic commented 2 years ago

Thanks! It's quite annoying trying to manually install it, as you need the whole toolchain. Better is if you can do it either in a container or ideally in CI. Does your travis.yml no longer run? Most CI providers seem to use Ubuntu, where Xenial would be the release you'd want. GitHub Actions would do the trick too.

ascorbic commented 2 years ago

This is awesome, @kriszyp! When do you expect to be able to release this?

kriszyp commented 2 years ago

After a number of failed attempts at getting travis working past, looks like I finally managed to get it up and running. Which is great, this will make building binaries way easier. Anyway, this is setup to use Xenial for the linux builds, so hopefully we should have the right glibc. I built and published v1.6.8, if you want try that and see if it is working on AWS?

ascorbic commented 2 years ago

Need a couple more tests, but it looks like it works! Once again, thanks so much for doing this!

kriszyp commented 2 years ago

BTW, I included linux arm64 in the build this time, but I assume that is rarely used, so might omit in the future. Probably more valuable would be builds for macos on M1, which I believe is arm64, but doesn't look like travis has that yet.

ascorbic commented 2 years ago

I think arm64 linux is becoming more widely-used. M1 macOS would be good. I know that sharp has M1 builds from CI. I wonder what Lovell uses for that.

ascorbic commented 2 years ago

It works!!

mischnic commented 2 years ago

I know that sharp has M1 builds from CI. I wonder what Lovell uses for that.

You don't need an M1 machine to compile binaries for that target, you can cross-compile from a "normal" macOS Intel system. You're using C++ but for example with Rust, it's just a different target triplet: https://github.com/parcel-bundler/parcel/blob/2013dfff74468ee8a3a1dbed6742db32dd24032a/.github/workflows/nightly-release.yml#L171-L177