Could not compile dependency :erlexec

[Skoda091]

I'm running into an error when compiling erlexec with mix deps.compile erlexec.

Error Message

System Information

OS: macOS Monterey 12.1 (21C52) CPU: Apple M1 Pro Erlang: ~24.1.7~ 24.2 Elixir: 1.13.0-otp-24 Phoenix: 1.6.0

Erlang/Elixir and iTerm installed in ARM (M1) native versions.

Misc

Any combination of those commands doesnt help:

• rm -Rf deps
• rm -Rf _build
• mix deps.clean --all
• mix deps.get
• mix deps.compile

[saleyn]

I was able to reproduce this with OTP 24.2, but not with OTP 24.1 and prior versions. For some reason the linker is not finding any of the ei_* library functions in the libei.a. This is weird. Maybe an OTP build issue?

[Skoda091]

I was able to reproduce this with OTP 24.2, but not with OTP 24.1 and prior versions. For some reason the linker is not finding any of the ei_* library functions in the libei.a. This is weird. Maybe an OTP build issue?

You're right, in fact I run the app on erlang 24.2 not 24.1. Any tips how to overcome this on 24.2?

[lukebakken]

I can reproduce this in my up-to-date Arch Linux environment:

===> Compiling c_src/exec_impl.cpp
===>
===> Linking /home/lbakken/development/saleyn/erlexec/priv/x86_64-pc-linux-gnu/exec-port
===> sh(g++ c_src/ei++.o c_src/exec.o c_src/exec_impl.o  -lcap  -L"/usr/lib/erlang/lib/erl_interface-5.1/lib" -lei -o /home/lbakken/development/saleyn/erlexec/priv/x86_64-pc-linux-gnu/exec-port)
failed with return code 1 and the following output:
/usr/bin/ld: c_src/ei++.o: in function `ei::Serializer::print(std::ostream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
/home/lbakken/development/saleyn/erlexec/c_src/ei++.cpp:62: undefined reference to `ei_s_print_term'

An interesting side note is that I can't get nm to use the correct plugin to read the symbols from libei.a:

$ nm --plugin /usr/lib/bfd-plugins/liblto_plugin.so /usr/lib/erlang/lib/erl_interface-5.1/lib/libei.a

ei_connect.o:
nm: ei_connect.o: plugin needed to handle lto object
0000000000000001 C __gnu_lto_slim

When I need a distraction I'll return to this issue because it'll be an interesting learning experience.

[saleyn]

What's interesting is that when I build the OTP 24.2 from source, the libei.a is not compiled with LTO, and ld and nm correctly resolve all symbols in that library.

[lukebakken]

@saleyn I see the same thing 🤷

[saleyn]

@lukebakken, it looks like there's nothing special in terms of compilation options for Arch linux. @Skoda091: what linux distribution are you getting this issue in?

[lukebakken]

it looks like there's nothing special in terms of compilation options for Arch linux

Indeed, so you'd think our custom-compiled Erlangs would exhibit the same behavior. Maybe kerl adds a flag that isn't in the PKGBUILD

[saleyn]

Maybe there's some other workaround how to reference LTO compiled objects within an *.a archive? I haven't seen this issue before with other C++ projects.

[Skoda091]

@Skoda091: what linux distribution are you getting this issue in?

I'm getting this on macOS Monterey 12.1 (21C52)

[saleyn]

@Skoda091: what linux distribution are you getting this issue in?

I'm getting this on macOS Monterey 12.1 (21C52)

@Skoda091, when you build erlang 24.2 from sources on your macOS, do you get the same error?

[saleyn]

Please follow the discussion of erlang/otp#5609 to dig to the bottom of the issue. At least on Arch Linux this is related to the way erlang package was build with striping symbols from the libei.a archive, which produces unusable libei.a library when included object files use the LTO optimization.

[lukebakken]

Thank you for pointing out the discussion in https://github.com/erlang/otp/issues/5609.

is there a way to link with the stripped libei.a compiled with LTO that would resolve all symbols?

I would follow up with the Arch Linux maintainers and Allan in particular. Since you've gotten to the bottom of this issue the maintainers should help you out.

[saleyn]

This issue with Arch linux has been addressed by the maintainers of the erlang package for Arch Linux. So, @Skoda091, if you are still experiencing the issue with MacOS please see how to get the erlang package updated for MacOS to ensure that LTO compilation of the package includes CFLAGS+=' -ffat-lto-objects'.

I am going to close this issue, as it's not related to erlexec but to the way how the erlang/otp is built for your OS.

[lukebakken]

I'd like to confirm that adding the CFLAGS+=" -ffat-lto-objects" to build the erlang package works fine with stripping in my environment.

Well done @saleyn

[Skoda091]

Removing all x86 libraries installed via homebrew and installing all of them with arm64 homebrew solved that issue for me.