Gem commonmarker 1.x fails to compile native extensions with rust

[rwstauner]

Using the latest commit and a jvm-ce build

truffleruby 24.1.0-dev-00307c24, like ruby 3.2.2, GraalVM CE JVM [x86_64-linux]

with the latest gem version (1.0.4) installation fails due to several items missing from RbConfig::CONFIG:

error: failed to run custom build command for `rb-sys v0.9.85`

Caused by:
  process didn't exit successfully: `/home/spin/.gem/truffleruby/3.2.2/gems/commonmarker-1.0.4/ext/commonmarker/target/release/build/rb-sys-4475e29d236d2359/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
  cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
  cargo:rerun-if-env-changed=RBCONFIG_MAJOR

  --- stderr
  thread 'main' panicked at /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-build-0.9.85/src/rb_config.rs:206:32:
  Key not found: MAJOR

If I add these values (and use something like export TRUFFLERUBYOPT=-r$PWD/config) I get past those issues.

RbConfig::CONFIG["MAJOR"] = "3"
RbConfig::CONFIG["MINOR"] = "2"
RbConfig::CONFIG["TEENY"] = "1"
RbConfig::CONFIG["PATCHLEVEL"] = "1"
RbConfig::CONFIG["archincludedir"] = "/tmp/archincludedir"

Then I get other errors:

error[E0432]: unresolved import `crate::rb_gc_adjust_memory_usage`
 --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/tracking_allocator.rs:3:13
  |
3 | use crate::{rb_gc_adjust_memory_usage, utils::is_ruby_vm_started};
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^ no `rb_gc_adjust_memory_usage` in the root

error[E0412]: cannot find type `RData` in this scope
-->
/home/spin/.gem/truffleruby/3.2.2/gems/commonmarker-1.0.4/ext/commonmarker/target/release/build/rb-sys-f73ca150f0763414/out/bindings-0.9.85-x86_64-linux-3.2.1.rs:3478:51
     |
3478 |         pub fn rb_tr_rdata(object: VALUE) -> *mut RData;
     |                                                   ^^^^^ not found in this scope

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:19:35
   |
19 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:34:35
   |
34 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:52:34
   |
52 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:56:38
   |
56 |             let mut f = rarray.basic.flags;
   |                                      ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:70:34
   |
70 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/spin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.85/src/stable_api/ruby_3_2.rs:94:35
   |
94 |         let ret: u32 = ((*rbasic).flags & crate::ruby_value_type::RUBY_T_MASK as VALUE) as _;
   |                                   ^^^^^ unknown field

Some errors have detailed explanations: E0412, E0432, E0609.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `rb-sys` (lib) due to 8 previous errors
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:377: target/release/libcommonmarker.so] Error 101

make failed, exit code 2

[jac241]

Gems that use the rb-sys for rust don't appear to work. I'm running into an identical issue with truffleruby 24.0.2 trying to install the polars-df gem (https://github.com/ankane/ruby-polars) which also uses the rb-sys crate.

[eregon]

I have heard @nirvdrum has been looking at this recently. In general TruffleRuby should be able to support Rust extensions easier now that extensions are run natively. However there seems to be some issues like the missing RbConfig entries above and rb-sys accessing too many internals that need to be addressed first.

[eregon]

With https://github.com/oracle/truffleruby/pull/3664 fixed, I tried gem i -V commonmarker and got:

error: failed to run custom build command for `rb-sys v0.9.99`
...
  thread 'main' panicked at /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.69.4/lib.rs:622:31:
  Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"

OK I guess I need to install libclang on Linux: https://github.com/oxidize-rb/rb-sys?tab=readme-ov-file#what-dependencies-do-i-need-to-build-a-ruby-extension-in-rust After sudo dnf install clang-libs:

  /usr/include/stdio.h:33:10: fatal error: 'stddef.h' file not found
  thread 'main' panicked at /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/build/main.rs:55:6:
  generate bindings: ClangDiagnostic("/usr/include/stdio.h:33:10: fatal error: 'stddef.h' file not found\n")

OK, so after sudo dnf install clang:

  error: 'rustfmt' is not installed for the toolchain 'stable-x86_64-unknown-linux-gnu'.
  To install, run `rustup component add rustfmt`
  Failed to run rustfmt: Internal rustfmt error (non-fatal, continuing)
  error: 'rustfmt' is not installed for the toolchain 'stable-x86_64-unknown-linux-gnu'.
  To install, run `rustup component add rustfmt`
  thread 'main' panicked at /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-build-0.9.99/src/rb_config.rs:209:32:
  Key not found: archincludedir
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

So we are still missing RbConfig::CONFIG['archincludedir'] it seems, we should define that like MRI does:

> RbConfig::MAKEFILE_CONFIG['archincludedir']
=> "$(includedir)/$(arch)"

^ @andrykonchin Could you make a PR adding that?

[eregon]

Doing a quick workaround for that, similar to what Randy did in the first post:

# config.rb
RbConfig::CONFIG["archincludedir"] = "#{RbConfig::CONFIG["includedir"]}/#{RbConfig::CONFIG["arch"]}"

I get:

error[E0432]: unresolved import `crate::rb_gc_adjust_memory_usage`
 --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/tracking_allocator.rs:3:13
  |
3 | use crate::{rb_gc_adjust_memory_usage, utils::is_ruby_vm_started};
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^ no `rb_gc_adjust_memory_usage` in the root

error[E0412]: cannot find type `RData` in this scope
 --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/target/release/build/rb-sys-b6e62330b9516279/out/bindings-0.9.99-x86_64-linux-3.2.4.rs:1:299289
  |
1 | ....2.4"] pub fn rb_tr_rdata (object : VALUE) -> * mut RData ; # [doc = "This is the primitive way to wrap an existing C struct into ::RD...
  |                                                        ^^^^^ not found in this scope

   Compiling emojis v0.6.3
error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:19:35
   |
19 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:34:35
   |
34 |         let flags = rstring.basic.flags;
   |                                   ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:52:34
   |
52 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:56:38
   |
56 |             let mut f = rarray.basic.flags;
   |                                      ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:70:34
   |
70 |         let flags = rarray.basic.flags;
   |                                  ^^^^^ unknown field

error[E0609]: no field `flags` on type `uncategorized::RBasic`
  --> /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/commonmarker-1.1.5/ext/commonmarker/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.99/src/stable_api/ruby_3_2.rs:94:35
   |
94 |         let ret: u32 = ((*rbasic).flags & crate::ruby_value_type::RUBY_T_MASK as VALUE) as _;
   |                                   ^^^^^ unknown field

Some errors have detailed explanations: E0412, E0432, E0609.
For more information about an error, try `rustc --explain E0412`.
error: could not compile `rb-sys` (lib) due to 8 previous errors

Re rb_gc_adjust_memory_usage, TruffleRuby does define that as static inline: https://github.com/oracle/truffleruby/blob/e834bb5fe3d0877f5a664901adb655fd86b6b63c/lib/cext/include/ruby/internal/intern/gc.h#L388-L395 I guess Rust doesn't find it because it doesn't end up in the libruby then, we could fix that by declaring it in src/main/c/cext/gc.c instead.

Regarding RData, no clue.

Regarding RBasic.flags that's something that TruffleRuby on purpose does not and cannot expose, so that's something that will need to be fixed in rb-sys so it doesn't assume struct RBasic to have those fields.

[eregon]

@andrykonchin added archincludedir and moved rb_gc_adjust_memory_usage in https://github.com/oracle/truffleruby/pull/3671. So now the remaining issues are in the rb-sys crate.

[eregon]

Looking at the first flags error, it comes from https://github.com/oxidize-rb/rb-sys/blob/991df6de21d244631e1b889070a8e4136f4220fe/crates/rb-sys/src/stable_api/ruby_3_2.rs#L19 rb-sys is hardcoding CRuby internals, but it should use the RSTRING_LEN macro instead.

[eregon]

I found some more info in https://github.com/oxidize-rb/rb-sys/pull/229. So probably TruffleRuby should be treated like ruby-head then, because ruby_3_2.rs hardcoded macros are incompatible and CRuby-specific.

[eregon]

I submitted an issue to rb-sys: https://github.com/oxidize-rb/rb-sys/issues/424