Failure when loaded in Alpine Linux with musl-dev

ronaldtse commented 2 years ago

In Alpine, the musl-dev library is used instead of glibc. Somehow, the platform gem "x86_64-linux" is installed for expressir.

$ docker run --platform=linux/amd64 -it ruby:3.1-alpine sh
/ # apk add gcc g++ cmake make libc-dev
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
(1/25) Installing libacl (2.2.53-r0)
(2/25) Installing libbz2 (1.0.8-r1)
(3/25) Installing expat (2.4.7-r0)
(4/25) Installing lz4-libs (1.9.3-r1)
(5/25) Installing xz-libs (5.2.5-r0)
(6/25) Installing zstd-libs (1.5.0-r0)
(7/25) Installing libarchive (3.6.0-r0)
(8/25) Installing brotli-libs (1.0.9-r5)
(9/25) Installing nghttp2-libs (1.46.0-r0)
(10/25) Installing libcurl (7.80.0-r0)
(11/25) Installing rhash-libs (1.4.2-r2)
(12/25) Installing libuv (1.42.0-r0)
(13/25) Installing cmake (3.21.3-r0)
(14/25) Installing binutils (2.37-r3)
(15/25) Installing libgomp (10.3.1_git20211027-r0)
(16/25) Installing libatomic (10.3.1_git20211027-r0)
(17/25) Installing libgphobos (10.3.1_git20211027-r0)
(18/25) Installing isl22 (0.22-r0)
(19/25) Installing mpfr4 (4.1.0-r0)
(20/25) Installing mpc1 (1.2.1-r0)
(21/25) Installing gcc (10.3.1_git20211027-r0)
(22/25) Installing musl-dev (1.2.2-r7)
(23/25) Installing libc-dev (0.7.2-r3)
(24/25) Installing g++ (10.3.1_git20211027-r0)
(25/25) Installing make (4.3-r0)
Executing busybox-1.34.1-r4.trigger
OK: 256 MiB in 60 packages
/ # gem install expressir
Fetching expressir-1.2.1-x86_64-linux.gem
Fetching thor-1.2.1.gem
Successfully installed thor-1.2.1
Successfully installed expressir-1.2.1-x86_64-linux
2 gems installed
/ # irb
irb(main):001:0> require 'expressir'
=> true
irb(main):002:0> require 'expressir/express/parser'
/usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:5:in `require_relative': cannot load such file -- /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/express_parser (LoadError)
        from /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:5:in `rescue in <top (required)>'
        from /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:1:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from (irb):3:in `<main>'                                            
        from /usr/local/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
        from /usr/local/bin/irb:25:in `load'                                
        from /usr/local/bin/irb:25:in `<main>'                              
/usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:3:in `require_relative': Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/3.1/express_parser.so) - /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/3.1/express_parser.so (LoadError)
    from /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:3:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from (irb):3:in `<main>'
    from /usr/local/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
    from /usr/local/bin/irb:25:in `load'
    from /usr/local/bin/irb:25:in `<main>'

When I install on aarch64, it correctly detects that I am on a different platform and therefore does not install the platform gem and compiles properly.

$ docker run --platform=linux/arm64/v8 -it ruby:3.1-alpine sh
/ # apk add gcc g++ cmake make libc-dev
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/aarch64/APKINDEX.tar.gz
(1/25) Installing libacl (2.2.53-r0)
(2/25) Installing libbz2 (1.0.8-r1)
(3/25) Installing expat (2.4.7-r0)
(4/25) Installing lz4-libs (1.9.3-r1)
(5/25) Installing xz-libs (5.2.5-r0)
(6/25) Installing zstd-libs (1.5.0-r0)
(7/25) Installing libarchive (3.6.0-r0)
(8/25) Installing brotli-libs (1.0.9-r5)
(9/25) Installing nghttp2-libs (1.46.0-r0)
(10/25) Installing libcurl (7.80.0-r0)
(11/25) Installing rhash-libs (1.4.2-r2)
(12/25) Installing libuv (1.42.0-r0)
(13/25) Installing cmake (3.21.3-r0)
(14/25) Installing binutils (2.37-r3)
(15/25) Installing libgomp (10.3.1_git20211027-r0)
(16/25) Installing libatomic (10.3.1_git20211027-r0)
(17/25) Installing libgphobos (10.3.1_git20211027-r0)
(18/25) Installing isl22 (0.22-r0)
(19/25) Installing mpfr4 (4.1.0-r0)
(20/25) Installing mpc1 (1.2.1-r0)
(21/25) Installing gcc (10.3.1_git20211027-r0)
(22/25) Installing musl-dev (1.2.2-r7)
(23/25) Installing libc-dev (0.7.2-r3)
(24/25) Installing g++ (10.3.1_git20211027-r0)
(25/25) Installing make (4.3-r0)
Executing busybox-1.34.1-r4.trigger
OK: 250 MiB in 60 packages
/ # gem install expressir
Fetching rice-4.0.3.gem
Fetching expressir-1.2.1.gem
Fetching thor-1.2.1.gem
Successfully installed thor-1.2.1
Successfully installed rice-4.0.3
Building native extensions. This could take a while...
Successfully installed expressir-1.2.1
3 gems installed
/ # irb
irb(main):001:0> require 'expressir'
=> true
irb(main):002:0> require 'expressir/express/parser'
=> true
irb(main):003:0>

ronaldtse commented 2 years ago

The problem is also related to this: https://github.com/metanorma/emf2svg-ruby/issues/17

alexeymorozov commented 2 years ago

$ gem environment | grep -A 2 "RUBYGEMS PLATFORMS"
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux-musl

Platform is detected as x86_64-linux-musl.

$ gem help platforms
RubyGems matches platforms as follows:

  * The CPU must match exactly unless one of the platforms has
    "universal" as the CPU or the local CPU starts with "arm" and the gem's
    CPU is exactly "arm" (for gems that support generic ARM architecture).
  * The OS must match exactly.
  * The versions must match exactly unless one of the versions is nil.

So it matches x86_64-linux because the version part is nil.

The next step is to create a build for x86_64-linux-musl, or rename x86_64-linux to a more specific build.

ronaldtse commented 2 years ago

The mismatch between glibc and musl platforms has been subject of work at rubygems:

And that it is a known issue on Alpine that Rubygems can mistakenly install glibc platform gems on Alpine, and therefore they maintain such a patch to only install platform=ruby gems:

I reviewed the code in https://github.com/rubygems/rubygems/pull/4082 and apparently there is this rather undocumented differentiation of platforms:

https://github.com/lloeki/rubygems/blob/2743af107e1d542a3cece421ead752e1b8141f93/test/rubygems/test_gem_platform.rb#L136-L139

      'x86_64-linux'           => ['x86_64',    'linux',     nil],
      'x86_64-linux-gnu'       => ['x86_64',    'linux',     nil],
      'x86_64-linux-musl'      => ['x86_64',    'linux',     'musl'],
      'x86_64-linux-uclibc'    => ['x86_64',    'linux',     'uclibc'],

According to this PR, technically, a platform gem that specifies x86_64-linux should not be installed on x86_64-linux-musl

However, in the latest master branch of Rubygems, this differentiation has been removed (this PR was reverted in https://github.com/rubygems/rubygems/pull/4434).

https://github.com/lloeki/rubygems/blob/8a712a0dd52e661ea973845b70681da09ce75edd/test/rubygems/test_gem_platform.rb#L136-L137

      'x86_64-linux'           => ['x86_64',    'linux',     nil],
      'x86_64-linux-musl'      => ['x86_64',    'linux',     'musl'],

And the behavior at rubygems is now just broken:

https://github.com/rubygems/rubygems/pull/4488

The way we have to solve this is either:

The x86_64-linux platform gem works natively on musl. I don't know if our source code can be fixed that way (i.e. avoiding GNU extensions)
We publish platform gems for x86_64-linux-musl so that on Alpine the x86_64-linux-musl is always used.

maxirmx commented 2 years ago

"/lib/ld-linux-x86-64.so.2" -- is a dynamic loader.
IMHO it is worth trying compatibility package: https://pkgs.alpinelinux.org/contents?file=ld-linux-x86-64*

expressir is using rake-compiler-dock that supports:

platform: x86-mingw32
platform: x64-mingw-ucrt
platform: x64-mingw32
platform: x86-linux
platform: x86_64-linux
platform: x86_64-darwin
platform: arm64-darwin
platform: arm-linux
platform: aarch64-linux
platform: jruby

Alpine will need to be added to this list

ronaldtse commented 2 years ago

The compatibility package gcompat worked:

/ # apk add gcc g++ cmake make libc-dev
...
/ # apk add gcompat
(1/3) Installing musl-obstack (1.2.2-r0)
(2/3) Installing libucontext (1.1-r0)
(3/3) Installing gcompat (1.0.0-r4)
OK: 256 MiB in 64 packages
/ # irb
irb(main):001:0> require 'expressir'
=> true
irb(main):002:0> require 'expressir/express/parser'
=> true

@maxirmx Maybe we should make a PR to rake-compiler-block?

maxirmx commented 2 years ago

@maxirmx Maybe we should make a PR to rake-compiler-block?

@ronaldtse, we can if ruby really support differentiation but I am afraid that it won't be straightforwards either.

maxirmx commented 2 years ago

This (or closely related) issue is a blocker for: https://github.com/metanorma/packed-mn/issues/158

Whe packaged with tebako in linux-musl environment expressir loads x86_64-linux extension which requires /lib/ld-linux-x86-64.so It is worth to note that this issue occurs only for expressir and libemf2svg (that inherits platform logic from expressir) All other gems with native extensions work

ronaldtse commented 2 years ago

It is worth to note that this issue occurs only for expressir and libemf2svg (that inherits platform logic from expressir) All other gems with native extensions work

Then if we update the platform logic, so it does not treat it as x86_64-linux, it should work?

maxirmx commented 2 years ago

Then if we update the platform logic, so it does not treat it as x86_64-linux, it should work?

I do not understand why other gems work. Probably it shall be possible to implement platform logic as others do it

maxirmx commented 2 years ago

Well, so far it looks like a mistery

nokogiri gem taht uses the same build system installs and runs on Alpine using linux prebuild native gem. The authour and maintainer of nokogiri is sure that gcompat is a prerequisite, but it is not true.

maxirmx commented 2 years ago

Expressir looks different for ldd. The reason is yet unknown

maxirmx commented 2 years ago

Summary of research re Alpine (musl) support

Since the effort to support x86_64-linux-musl platform looks stalled (https://github.com/rubygems/rubygems/pull/4488) I was looking at options to make the x86_64-linux platform gems (expressir and others) work natively on musl
Initial view was to identify gnu extensions in express_parser and either replace them with std equivalents or neutralize by linking glibc statically. However, I found that the show-stopper is related to functionality defined by standard. Any C/C++ static variable results in the usage of thread local storage (tls); tls implementation is provided by dynamic linker; dynamic linkers are not compatible between gnu and musl.
Rice implementation is heavily dependant on static variables that it creates. I do not see any way to change it.

The bottom line. For gems that use Rice there is no way to x86_64-linux platform gems to work on musl unless gcompat package is installed

maxirmx commented 2 years ago

What can be done

We make take over rubygems platform effort (https://github.com/rubygems/rubygems/pull/4488) and attempt to drive it to completion. This is the 'right' approach but it may last forever
For expressir and similar gems we can deploy musl binaries inside linux gem and implement a hack that will use these libaries if on Alpine. This is the risky approach, result cannot be guaranteed.
We can make gcompat or force_ruby_platform a requirement for Alpine installations. I can probably implement reasonable error message if this requirement is not met. This is 'fast' approach, though packed_mn may require additional effort.

ronaldtse commented 2 years ago

@maxirmx thank you for the extensive investigation!

We can make gcompat or force_ruby_platform a requirement for Alpine installations. I can probably implement reasonable error message if this requirement is not met. This is 'fast' approach, though packed_mn may require additional effort.

This is reasonable for packed-mn and our gems. We should take this approach.

We make take over rubygems platform effort (Support non gnu libc linux (part 2) rubygems/rubygems#4488) and attempt to drive it to completion. This is the 'right' approach but it may last forever

This is risky for us because we do not know when the Ruby team is interested enough in musl. This PR has implications in the Rubygems system because other there is a backwards compatibility concern too, i.e. all currently available platform gems are assigned to 'glibc' to differentiate from 'musl'.

For expressir and similar gems we can deploy musl binaries inside linux gem and implement a hack that will use these libaries if on Alpine. This is the risky approach, result cannot be guaranteed.

I think this is also a reasonable approach, and it should be doable, but it will bloat the gems, and there are not that many people who use Alpine anyway. Prefer using the gcompat requirement.

maxirmx commented 2 years ago

I will do more tests with gcompat and/or packed_mn to be sure that gcompat is solid approach

maxirmx commented 2 years ago

Update Alpine test workflow: https://github.com/lutaml/expressir/actions/workflows/alpine.yml

force_ruby_platform works, of course
gcompat does not work. It is good enough to let extension load; however, the tests do not pass with blind exception

RuntimeError:
        No error information
      # ./lib/expressir/express/parser.rb:35:in `initialize'
      # ./lib/expressir/express/parser.rb:35:in `new'
      # ./lib/expressir/express/parser.rb:35:in `from_file'
      # ./spec/expressir/model/model_element_spec.rb:174:in `block (3 levels) in <top (required)>'
      # ./spec/spec_helper.rb:19:in `block (2 levels) in <top (required)>'

I found that this exception is related to std::call_once, i.e.: thread local memory again.

force_ruby_platform helped to create packed_mn Alpine package but it will be painful for any user to take this approach. This setting cannot be applied to a single gem so we actually force to build everything in the bundle even if there are precompiled extensions out there.

gcompat issue can be debugged further but neither time nor effort can be predicted.

Attn @ronaldtse

ronaldtse commented 2 years ago

@maxirmx what is the best course ahead for now? Is Alpine a lost cause?

maxirmx commented 2 years ago

@ronaldtse Considering Expressir native extension gor musl I can imagine three options:

fix bundler (https://github.com/rubygems/rubygems/pull/4488)
hack pandler. For example, develop bundler plugin that will patch bundler on the go. Not sure if it is possible though
debug run-time exception with gcompat. In order to do it some minimal example is required. Expressir extension is really big and virtually impossible to use it for debugging. This effor may end up with something simple, like compilation flag, but can also lead to gcompat patch.

If you ask about packed-mn, the first version is ready as far as compilation and packaging is considered (https://github.com/metanorma/packed-mn/issues/158#issuecomment-1152660114) There is run-time issue with fontist gem that needs to be fixed

So the real question is if and where you are ready to invest

ronaldtse commented 2 years ago

Rubygems now supports differentiation of platform between glibc and musl: https://github.com/rubygems/rubygems/pull/5852 , which works from 3.1.0.pre.1 onwards
We wait for the merging of https://github.com/rubygems/rubygems/pull/4488 for bundler to work
Then we can re-do this task of publishing the gem on the musl platforms

ronaldtse commented 1 year ago

@maxirmx is this issue to be closed? Thanks!

maxirmx commented 1 year ago

@ronaldtse

Actually there are (there were) two related issues:

There was an issue with bundler which ignored specialization and installed _x8664-linux extension on Alpine even when _x8664-linux-musl was present. This issue was fixed in bundler.
There are some gems which provide gnu version of extension and name it _x8664-linux In this case it gets installed on Alpine and fails for some gems.

As a solution for both problems we do not use precompiled native extensions when Alpine version of tebako package is created. If we consider metanorma the issue was nokogiri. The maintainers claimed that its x64-linux extension worked for Alpine but in fact it did not. At least not for me.

So the answers is 1) We can publish x64-linux-musl native extension (now we do not have it published). It will close this issue. 2) We can revisit metanorma packaging on alpine to see if others also resolved there issues on Alpine. If they did it we can use prebuilt native extensions. will help to reduce Alpine package size and packaging time.

ronaldtse commented 1 year ago

@maxirmx in this case, could you please help schedule 1 and 2 after the recent tebako work? Thanks!

maxirmx commented 1 year ago

Packaging of expressir gem is implemented with rack-compiler-dock I suggest we wait until https://github.com/rake-compiler/rake-compiler-dock/issues/75 is completed

lutaml / expressir

Failure when loaded in Alpine Linux with musl-dev #106