Provide 3.2 build support

[gjtorikian]

Should close https://github.com/gjtorikian/commonmarker/issues/211

[ianks]

https://github.com/oxidize-rb/rb-sys/pull/128

[ianks]

Looks like the build passed, but the Cargo.lock at the root of the gem isn't included in spec.files. I think adding it to spec.files should resolve the issue.

[kivikakk]

Ta! Just giving that a go now.

[kivikakk]

Nice! I'll leave this for @gjtorikian to consider, since there might be aspects of this he has in mind that I'm not aware of, but hopefully this gets us there faster.

[kivikakk]

Hm! I've noticed that trying to build this branch on Ruby 3.2.0 with something like this:

source "https://rubygems.org"

gem "commonmarker", github: "gjtorikian/commonmarker", branch: "support-32"

succeeds on (e.g.) Linux x86_64, but fails on Darwin arm64:

Fetching https://github.com/gjtorikian/commonmarker.git
Resolving dependencies...Fetching gem metadata from https://rubygems.org/.
...
Using bundler 2.3.25
Using rb_sys 0.9.54
Using commonmarker 1.0.0.pre5 (was 1.0.0.pre4) from https://github.com/gjtorikian/commonmarker.git (at support-32@a18b3e1)
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784/ext/commonmarker
cargo rustc --target-dir /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784 --manifest-path
/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784/ext/commonmarker/Cargo.toml --lib --profile release --locked -- -C linker\=clang
-L native\=/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib -L native\=/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib -L native\=/opt/homebrew/opt/gmp/lib -C
link_arg\=-Wl,-multiply_defined,suppress -l pthread -C debuginfo\=1
   Compiling commonmarker v1.0.0 (/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784/ext/commonmarker)
error: linking with `clang` failed: exit status: 1
  |
= note: "clang" "-Wl,-exported_symbols_list,/var/folders/l1/j6x78td56fq_ctkt8qx5hjdw0000gn/T/rustcZEUPiH/list" "-arch" "arm64"
"/var/folders/l1/j6x78td56fq_ctkt8qx5hjdw0000gn/T/rustcZEUPiH/symbols.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.0.rcgu.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.1.rcgu.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.10.rcgu.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.11.rcgu.o"
[…]
[…]
[…]
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/liballoc-945b7af0c9f21aac.rlib"
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-0251f0b5857602a2.rlib"
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcore-9382e8c089006a25.rlib"
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-6b5d600bff28faab.rlib" "-lSystem" "-lresolv" "-lc" "-lm"
"-liconv" "-L" "/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "-o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/libcommonmarker.dylib"
"-Wl,-dead_strip" "-dynamiclib" "-Wl,-dylib" "-nodefaultlibs" "-Wl,-multiply_defined,suppress"
  = note: Undefined symbols for architecture arm64:
            "_rb_any_to_s", referenced from:
                magnus::value::Value::to_s_infallible::hc9ca6d51fb4404b8 in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.2.rcgu.o)
                magnus::value::Value::inspect::hd02724b944f40252 in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.2.rcgu.o)
            "_rb_ary_to_ary", referenced from:
                magnus::error::protect::call::h928de4cd683d6f7b (.llvm.14460302740317915909) in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.0.rcgu.o)
            "_rb_ascii8bit_encindex", referenced from:
[…]
[…]
[…]
                magnus::error::protect::call::h8ee51cf65c05a7c5 (.llvm.14460302740317915909) in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.0.rcgu.o)
                magnus::error::raise::hb1f04e67eecc0f1e in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.0.rcgu.o)
magnus::r_string::_$LT$impl$u20$core..convert..From$LT$alloc..string..String$GT$$u20$for$u20$magnus..value..Value$GT$::from::h2a2a140ad0df8154 in
libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.2.rcgu.o)
          ld: symbol(s) not found for architecture arm64
          clang-15: error: linker command failed with exit code 1 (use -v to see invocation)

error: could not compile `commonmarker` due to previous error

cargo failed, exit code 101

Gem files will remain installed in /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784 for inspection.
Results logged to /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/gem_make.out

  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:102:in `run'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/cargo_builder.rb:32:in `call'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/cargo_builder.rb:32:in `build_crate'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/cargo_builder.rb:22:in `build'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:171:in `build_extension'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:205:in `block in build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:202:in `each'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:202:in `build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:843:in `build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/rubygems_gem_installer.rb:72:in `build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/source/path/installer.rb:34:in `post_install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/source/path.rb:244:in `generate_bin'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/source/git.rb:194:in `install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/gem_installer.rb:54:in `install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/parallel_installer.rb:186:in `do_install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:62:in `apply_func'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:57:in `block in process_queue'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:54:in `loop'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:54:in `process_queue'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:91:in `block (2 levels) in create_threads'

An error occurred while installing commonmarker (1.0.0.pre5), and Bundler cannot continue.

In Gemfile:
  commonmarker

Looks like possibly magnus's preview 3.2.0 support.

[kivikakk]

Looks like it might be related to https://github.com/oxidize-rb/rb-sys/issues/88 maybe not triggering as it should? If I repeat the cargo rustc invocation from the failing build log but append -C link_arg=-Wl,-undefined,dynamic_lookup, it succeeds.

… Actually, It looks like this is to do with RubyGems' own Rust building capabilities running this command for us, which, on further investigation, does actually add this flag these days.

Why does my own Ruby 3.2.0's RubyGems not have this, which is part of the v3_2_0 tag? I have no idea whatsoever — I lack some knowledge about how RubyGems is packaged these days, I guess. gem update --system was all I needed.

[ianks]

Looks like it might be related to oxidize-rb/rb-sys#88 maybe not triggering as it should? If I repeat the cargo rustc invocation from the failing build log but append -C link_arg=-Wl,-undefined,dynamic_lookup, it succeeds.

… Actually, It looks like this is to do with RubyGems' own Rust building capabilities running this command for us, which, on further investigation, does actually add this flag these days.

Why does my own Ruby 3.2.0's RubyGems not have this, which is part of the v3_2_0 tag? I have no idea whatsoever — I lack some knowledge about how RubyGems is packaged these days, I guess. gem update --system was all I needed.

Yes it should work now. On another note, this thread is worth reading regarding builtin ruby gems cargo builder. Ling story short, using rb_sys as a dependency will yield more predictable results until some issues with Rubygems are addressed with external builders.

[gjtorikian]

Appreciate you both <3

[gjtorikian]

Nice! I'll leave this for @gjtorikian to consider, since there might be aspects of this he has in mind that I'm not aware of, but hopefully this gets us there faster.

Thanks for the help y'all, this looks great.