Shipping libevent.a and libpcre.a with the linux binary packages

[lh3]

It seems that the crystal compiler requires libevent and pcre. If either library is absent, the linker will throw an error like

/usr/bin/ld: cannot find -levent (this usually means you need to install the development package for libevent)

Unfortunately, libevent and pcre are often missing from a system and installing/compiling them without root/sudo is non-trivial for inexperienced users.

A simple solution to this problem is to compile the two libraries as static libevent.a and libpcre.a on older Linux and put them in crystal-0.34-1/lib/crystal/lib along with libgc.a. This way, we can compile without system libevent and pcre. Furthermore, the resulting binary is not dynamically linked to libevent.so or libpcre.so – it is more portable.

PCRE and libevent don't have additional dependencies. They can be easily compiled as static libraries with ./configure --disable-shared. On my system, the compiled libraries can be found in .libs/lib{event,pcre}.a. I have tried this approach. It is working well for me.

[RX14]

I think two packages is the best solution, as long as they're documented and have the necessary warnings

[HertzDevil]

Just hit this recently because I was trying to get Crystal to work on Compiler Explorer.

PCRE reached end-of-life about a week ago, so if we distribute Crystal with PCRE 8.45 then it is always "up-to-date". Obviously it's a different story if we migrate to PCRE2 though.

If we build PCRE ourselves we also always get JIT support (#3852) regardless of the system library version.

[RX14]

development doesn't mean no security bugfixes i think. it is a positive though.

[orangeSi]

The libevent and pcre had packaged into crystal-xxx-linux-x86_64-bundled.tar.gz since crystal-1.2.0-1 in the https://github.com/crystal-lang/crystal/releases

$ls crystal-1.4.1-1/lib/crystal/lib
libevent.a  libevent_pthreads.a  libpcre.a

[noraj]

As the documentations says

Building fully statically linked executables is currently only supported on Alpine Linux.

For that purpose, the documentation recommends using musl-libc.

Official Docker Images based on Alpine Linux are available on Docker Hub at crystallang/crystal. The latest release is tagged as crystallang/crystal:latest-alpine.

As an example replace the local build:

$ crystal build src/miniss.cr --release --static -o bin/miniss
$ shards build --release --static

with

$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    crystal build src/miniss.cr --release --static -o bin/miniss
$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    shards build --release --static

Since you are cross-compiling from a docker container, it would be wiser to ensure to provide the correct cross compilation flags.

Find your platform with llvm-config --host-target on your host.

So change from

$ crystal build src/miniss.cr --release --static -o bin/miniss --cross-compile --target x86_64-pc-linux-gnu
$ cc bin/miniss.o -o bin/miniss  -rdynamic -static -L/home/noraj/.asdf/installs/crystal/1.7.2/bin/../lib/crystal -lpcre -lm -lgc -lpthread -levent  -lrt -lpthread -ldl

to

$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    crystal build src/miniss.cr --release --static -o bin/miniss --cross-compile --target x86_64-pc-linux-gnu
$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    cc bin/miniss.o -o bin/miniss  -rdynamic -static -L/usr/bin/../lib/crystal -lpcre -lm -lgc -lpthread -levent -lrt -lpthread -ldl

Finally, check your binary is fully static:

$ ldd bin/miniss
        not a dynamic executable

[Blacksmoke16]

@noraj Pretty sure this issue was related to using Crystal itself and not statically building some library/application. I.e. the case where you're installing Crystal via https://crystal-lang.org/install/from_targz/ and are unable to use a package manager or the ability to build the necessary libs from source.

[Sija]

@Blacksmoke16 doesn't seem like it... what makes you think that?

[Blacksmoke16]

@Sija Because the OP mentions Unfortunately, libevent and pcre are often missing from a system and installing/compiling them without root/sudo is non-trivial for inexperienced users. and the name of the issue is Shipping libevent.a and libpcre.a with the linux binary packages and it was solved via https://github.com/crystal-lang/distribution-scripts/pull/102 which added the static files to the *-bundled.tar.gz release assets.

[Sija]

@Blacksmoke16 Ah, it seems I misread the last comments. Somehow I thought they're referring to the documentation and not OP's issue, thank you for correcting me 🙏

[noraj]

@Blacksmoke16 Hum 🤔 because I encountered the same issue as the author only when statically building (libevent and libpcre) while I had no issue when dynamically building and at runtime (with crystal both from archlinux system and asdf). Those libraries are also missing from archlinux and even when you are root and have pcre and pcre2 installed you still don't have libpcre.a (there is no package providing it). Unlike the author I had no issue for dynamic building and runtime even if my code is using some regexp. And since internet leads me here when looking for those errors I guess that it will help others.

[Blacksmoke16]

@noraj Right, this issue is NOT related to building your application statically, but instead using Crystal itself from the pre-built Crystal binaries provided via the *.tar.gz release assets on a system that doesn't have pcre installed. I.e. when your only way to use Crystal is to download/extract the archive on a user local level.

In your case, you probably installed Crystal via your package manager which installed all the runtime deps, such as pcre, along with Crystal itself. This explains why you are able to run your applications just fine. However, your comment is correct when you want to go and build a static binary for something you wrote when Crystal is already installed.

[noraj]

In your case, you probably installed Crystal via your package manager which installed all the runtime deps, such as pcre, along with Crystal itself. This explains why you are able to run your applications just fine. However, your comment is correct when you want to go and build a static binary for something you wrote when Crystal is already installed.

I said that I tried with ASDF as well (https://crystal-lang.org/install/from_asdf/) which which is on a user local level too. This is because asdf-crystal doesn't use the bundled tarball.

[Blacksmoke16]

Okay, but is the issue that when you do like crystal build --static app.cr it fails, or when you try to do crystal run app.cr it fails? This issue relates to the latter in the case Crystal was installed via the *.tar.gz archive.

I'm not so sure ASDF can use the bundled tarball since the pre-built static libs included in it would prob only work on linux?

EDIT: If you're running into an issue, the forums or one of that chat rooms might be a good place to get help.

[noraj]

Okay, but is the issue that when you do like crystal build --static app.cr it fails, or when you try to do crystal run app.cr it fails? This issue relates to the latter in the case Crystal was installed via the *.tar.gz archive.

In my case crystal build --release app.cr and crystal run app.cr doesn't fails but crystal build --release --static app.cr fails both with Archlinux Crystal and ASDF Crystal (and also with the bundled tarball Crystall because the static libraries are not in the proper path or something).

I'm not so sure ASDF can use the bundled tarball since the pre-built static libs included in it would prob only work on linux?

It's possible to make an if only for linux.

EDIT: If you're running into an issue, the forums or one of that chat rooms might be a good place to get help.

I don't need help, I found the solution with the Alpine docker container, since Crystal supports static build only on Alpine. But as it's not trivial to find the answer in the official document on the exact way to do it I thought it would be nice from me to share the solution.

Update: But yeah I understand the original issue is a bit different and it's maybe not the perfect place to share this solution but it's not off topic either as it has the same cause and the docker solution could be helpful for user local level too.

[Blacksmoke16]

In my case crystal build --release app.cr and crystal run app.cr doesn't fails but crystal build --release --static app.cr fails both with Archlinux Crystal and ASDF Crystal (and also with the bundled tarball Crystall because the static libraries are not in the proper path or something).

Okay yea, that's a separate issue than what this issue is about and is somewhat expected as, as you already pointed out, Alpine docker container is the easiest way to create a static binary of an application.

EDIT: Worth pointing out, it's not only supported on Alpine linux. It just so happens to be the easiest. My understanding is you can do it on any system where you have access to that static `.alibs and pointing theCRYSTAL_LIBRARY_PATH` env var at them.

*At least for Linux static builds as I know Mac is a bit diff/not supported at all.

But as it's not trivial to find the answer in the official document on the exact way to do it

https://crystal-lang.org/reference/1.7/guides/static_linking.html spells this all out quite well. If there's something there that you found missing or confusing, opening an issue on the book's repo https://github.com/crystal-lang/crystal-book would be more helpful.