YJIT support in upcoming MRI 3.2

[casperisfine]

As you may already know, in Ruby 3.2 YJIT has been rewritten in Rust, as such to have YJIT available a semi-recent rustc (1.58.1+) is required.

I know ruby-install generally don't want to test for tool availability, but would you be open to do so for YJIT? To give you an idea, here's how its done in ruby-build: https://github.com/rbenv/ruby-build/blob/55575fdb5ea862fda0545578520eec8f91264fc5/bin/ruby-build#L1287-L1306

If not, are you open to add to the documentation how to enable YJIT in Ruby 3.2?

[postmodern]

ruby-install currently does not have logic to check if certain configuration flags were enabled, and add optional dependencies. This would require creating multiple dependencies files for the other ./configure options, which add additional options, then load and append those dependencies to the packages in the install_deps function. I'm not against adding this behavior, but it would be difficult to support it in a generalized/extendable way.

Generally, ruby-install should install the minimum required dependencies in order to compile/install Ruby with the default configuration options. One could argue that if you specify additional configuration options to enable additional features, it's up to the user to install additional dependencies? Although, I could definitely see how automatically detecting enabled features and installing the additional dependencies would be convenient.

[casperisfine]

Apologies, my issue might not have been clear. I'm not suggesting to install the dependencies, only to pass --enable-yjit if the required dependencies (rustc) are already present, and maybe to print a warning of some sort if they aren't.

[postmodern]

Oh. I am not a fan of enabling additional configuration options by default, as that might surprise users or annoy users who do not want YJIT enabled by default. Users can explicitly pass in --enable-yjit to opt-in to the new JIT. It's ultimately up to ruby-core to decide when --enable-yjit should be enabled by default; their ./configure script could check for the existence of rustc.

[casperisfine]

I'll see about getting Ruby's configure script to automatically enable YJIT if a sufficiently recent rustc is available.

[maximecb]

@postmodern just to clarify, if you do compile YJIT by passing --enable-yjit, then YJIT will still not be enabled at run-time unless you run ruby --yjit.

Last year with Ruby 3.1, YJIT was compiled/built by default, and that was easy because it was written in C, but as we rewrote it in Rust, and Rust compilers are not yet ubiquitous, we had no choice but to guard compilation behind --enable-yjit.

I agree with Jean that adding detection to the configure script is probably the best option. It would still be helpful for us if ruby-install could, for example, print a warning to users that Ruby is being compiled without YJIT and that they should install rustc if they want to build Ruby with YJIT. We're trying to avoid a situation where people build Ruby without YJIT and they don't know why YJIT was never compiled/available. We're hoping to get to a point where anyone with a Ruby installation would be able to run ruby --yjit if they want to.

[postmodern]

@maximecb if ruby wasn't compiled with --enable-yjit, will ruby --yjit print an option error, or will it print a message explaining that ruby wasn't compiled with --enable-yjit? Still defining a --yjit option that prints a helpful error message even when YJIT wasn't enabled would help inform the user on how to enable YJIT.

Also, if we want to incentivize users to compile ruby with --enable-yjit, might I suggest having ./configure print out some cool ASCII Art if --enable-yjit is enabled. This would be similar to nmap's ASCII dragon that's printed by their ./configure script.

We can also add --enable-yjit to the README examples, the man-page examples, and --help examples.

[maximecb]

I like the idea of ASCII art, and I agree it would be useful to print a message on how to build with YJIT if --yjit is not available.

You also make a good point that we should explain how to build with YJIT here: https://github.com/ruby/ruby/blob/master/doc/contributing/building_ruby.md

[havenwood]

I was really glad to see YJIT built by default when an adequate Rust is available in https://github.com/ruby/ruby/pull/6662. It seems great to handle that logic upstream so our only downstream consideration is whether to install Rust as a dependency.

[todd-a-jacobs]

TL;DR

I think it's likely trivial to add support for YJIT into ruby-installer at this point, but RC1 has other compilation problems that put it lower on the priority queue than other things IMHO.

Trying to Build Ruby 3.2.0-RC1 and YJIT on an M2

As of today, it seems that YJIT is built when it sees Rust, but may or may not be linked into the binary. The configure script still explicitly notes the existence of an --enable-yjit flag for configure, so until I can build a working Ruby on an M2 I can't verify whether or not it actually gets linked in even though the code for YJIT itself gets compiled successfully if Rust, OpenSSL, and Bison >= 3.0 are detected.

The main issue I'm seeing right now with ruby-install and YJIT is that ruby-install doesn't currently include Rust in its brew install dependencies, and the macOS SDK version of libffi seems to take precedence over a brewed version when passed with --with-opt-dir. Since the native libffi doesn't have FFI_GO_CLOSURES defined, the compilation issues a warning but completes anyway, despite the fact that some things remain deeply broken after the build is finished "successfully."

The configure script will detect Rust support for 3.2.0-RC1 if the rustc binary is present in PATH and the necessary library and header paths are passed, but it may not link unless the user passes the --enable-yjit to the configure script. It probably wouldn't break anything to simply include it in the ruby-install dependencies and the CFLAGS and LDFLAGS for it, but then again the feature is marked "experimental" so it's unclear how much support there should be in ruby-install for the feature.

On the other hand, ruby-build is already requiring openssl@3, gmp, and rust for its 3.2 builds. Whether or not that's a compelling argument I can't say.

I don't see why the feature can't be supported, but I think there are higher priority build issues around RC1. On the other hand, my only real objection is that if openssl@3 is in fact a hard requirement for 3.2+, this could potentially cause issues with Rust which as of today is being built against openssl@1 in Homebrew. That's something that would have to be addressed with the Rust team and the Homebrew formula if there is a conflict, although it seems possible that it's a non-issue.

In the meantime, Ruby 3.2.0-RC1 won't build properly under macOS Ventura with either ruby-install or ruby-build because libffi, bison, and openssl all have issues on M2 with the Apple SDK. I'm bisecting the problem, but at heart the problem is that Ruby will build, but can't build Psych, so any gem command will fail. Once those types of issues are resolved, I think supporting YJIT is potentially simpler than it currently seems.

[maximecb]

the feature is marked "experimental" so it's unclear how much support there should be in ruby-install for the feature.

YJIT is getting pretty mature at this point. It's definitely quite stable. The reason it's not enabled by default is that it does incur some memory overhead (any JIT compiler has to). I'll ask if we should get rid of the experimental label.

[todd-a-jacobs]

I'm not sure how much work a pull request to change the formula would be, but as of today I'm able to get YJIT as the default on macOS with the following:

# if you already have chruby installed...
chruby system

# not currently installed by default
brew install rust

ruby-install -U
ruby-install 3.2.0 -- --enable-yjit

# chruby won't pick up the new interpreter until the
# shell is restarted
exec "$SHELL" --login

# not all gems are currently compiling with native extensions,
# so irb will throw errors resolved by making the following
# gems pristine
gem pristine bigdecimal --version 3.1.3
gem pristine cgi --version 0.3.6
gem pristine date --version 3.3.0
gem pristine debug --version 1.7.0
gem pristine debug --version 1.7.1
gem pristine digest --version 3.1.1
gem pristine erb --version 4.0.2
gem pristine etc --version 1.4.1
gem pristine fcntl --version 1.0.2
gem pristine io-nonblock --version 0.2.0
gem pristine json --version 2.6.3
gem pristine openssl --version 3.0.1
gem pristine pathname --version 0.2.1
gem pristine psych --version 5.0.0
gem pristine racc --version 1.6.1
gem pristine rbs --version 2.8.1
gem pristine rbs --version 2.8.2
gem pristine strscan --version 3.0.4
gem pristine zlib --version 3.0.0

# add the following or similar to your login shell in .bash_profile or .zprofile;
# note that RUBYOPT won't stick unless chruby is already activated before
# exporting the variable (which is probably a bug)
source /opt/homebrew/opt/chruby/share/chruby/chruby.sh
source /opt/homebrew/opt/chruby/share/chruby/auto.sh
chruby "ruby-3.2.0"

# Use YJIT by default but don't clobber the existing +RUBYOPT+
# if one is already set.
if [[ -z "$RUBYOPT" ]]; then
    export RUBYOPT="--yjit"
else
    export RUBYOPT="$RUBYOPT --yjit"
fi

The if-then statement is pretty portable, but if you use ZSH or the Fish shell then there are probably more efficient ways to test-and-export RUBYOPT. However, this works well on anything Bash-like, and can easily be ported to straight Bourne with only minor tweaks.

Fish is a different story; I work around that by exec-ing Fish from ZSH to avoid having to solve for "Am I running Fish?" for a number of things, but it's easy enough to migrate most of these settings to Fish universal variables or set -gx in .config/fish/config.fish.

Once these things are done, the following works as expected from Bash, ZSH, and Fish:

$ ruby -e 'puts ENV["RUBYOPT"]; puts RUBY_DESCRIPTION'
--yjit
ruby 3.2.0 (2022-12-25 revision a528908271) +YJIT [arm64-darwin22]

While not automated yet, "it works for me." I hope it helps others out, too.

[todd-a-jacobs]

@postmodern Is the material my comment above something you'd like to see as a pull request, or is it something you'd prefer to analyze or refactor in some way first?

[postmodern]

@todd-a-jacobs I haven't had to run gem pristin after installing ruby-3.2.0. chruby already supports passing additional arguments to RUBYOPT.

$ chruby ruby-3.2.0 --yjit
$ echo $RUBYOPT
--yjit

[bfad]

I installed ruby 3.2.0 (ruby-install ruby 3.2.0) with and without rustc being installed. With out:

$> ruby --yjit -e 'puts ENV["RUBYOPT"]; puts RUBY_DESCRIPTION'
ruby: warning: Ruby was built without YJIT support. You may need to install rustc to build Ruby with YJIT.

ruby 3.2.0 (2022-12-25 revision a528908271) [arm64-darwin22]

With:

$> ruby --yjit -e 'puts ENV["RUBYOPT"]; puts RUBY_DESCRIPTION'

ruby 3.2.0 (2022-12-25 revision a528908271) +YJIT [arm64-darwin22]

I think it's safe to say that this feature is supported since it seems to be supported by the Ruby configuration.

[postmodern]

Closing this for now. Upstream Ruby currently provides YJIT as an optional feature, so ruby-install will treat it as such and defer to upstream ruby's ./configure's logic. I'm still open to discussion about adding special options to ruby-install to enable certain features (ex: yjit, jemalloc) and automatically install additional dependencies, but that should be a separate issue.

[todd-a-jacobs]

@todd-a-jacobs I haven't had to run gem pristine after installing ruby-3.2.0. chruby already supports passing additional arguments to RUBYOPT.

The pristine stuff could certainl be system-specific. When this happened, I was running an earlier version of Ventura on an M2; the problems with native extensions may have been fixed in an OS update, an Xcode CLI update, or in the upstream gems since then.

I posted what I had to do at the time, but I haven't re-tested, so the need to do that may no longer exist.

I'd still be happy to at least do a PR for the Wiki if you think it would help others who want to use YJIT. If not, that's okay too.