search

Shopify / yjit-bench

Set of benchmarks for the YJIT CRuby JIT compiler and other Ruby implementations.
MIT License
87 stars 22 forks source link

Enable protoboeuf-encode benchmark on TruffleRuby #332

Open andrykonchin opened 1 month ago

andrykonchin commented 1 month ago

The benchmark was disabled in #325. Now the issue is fixed (in https://github.com/oracle/truffleruby/pull/3674).

k0kubun commented 1 month ago

Is truffleruby-head not updated yet?

Failed benchmarks:
  ruby: protoboeuf-encode
k0kubun commented 1 month ago

TBH we want to test yjit-bench here instead of the TruffleRuby project. Given how unstable this job has been so far and the fact that it has been mostly due to TruffleRuby's bugs, I think this job should live in oracle/truffleruby and not in this repository.

If our interest is not to test the edge TruffleRuby but make sure yjit-bench works with TruffleRuby, we should run a released stable version of TruffleRuby instead of the HEAD TruffleRuby. yjit-bench's CI shouldn't randomly fail when nothing has been changed in yjit-bench.

andrykonchin commented 1 month ago

Yeah, looks like a build for the fix isn't ready yet (the fix was merged today). Will retry jobs tomorrow.

eregon commented 1 month ago

Yeah, looks like a build for the fix isn't ready yet (the fix was merged today). Will retry jobs tomorrow.

FWIW that's easily visible at https://github.com/ruby/truffleruby-dev-builder which shows which commit the latest build used.

eregon commented 1 month ago

TBH we want to test yjit-bench here instead of the TruffleRuby project. Given how unstable this job has been so far and the fact that it has been mostly due to TruffleRuby's bugs,

In this case at least it was not a bug, protoboeuf is using a method from Ruby 3.3, so that was not available yet on other Ruby implementations. For most gems out there they would support older Ruby versions and so it's quite a special case.

Given how unstable this job has been so far

The only case I recall is https://github.com/Shopify/yjit-bench/pull/262, were there many others?

I also think it is important to test another Ruby than just CRuby, otherwise the harness is likely to become accidentally CRuby-specific.

If our interest is not to test the edge TruffleRuby but make sure yjit-bench works with TruffleRuby, we should run a released stable version of TruffleRuby instead of the HEAD TruffleRuby. yjit-bench's CI shouldn't randomly fail when nothing has been changed in yjit-bench.

That's fair, we should try switching to a release of TruffleRuby. I think only protoboeuf-encode* would fail, which seems OK as it's a very CRuby/YJIT-specific gem, considering the generated code it emits (huge methods).

maximecb commented 1 month ago

I think only protoboeuf would fail, which seems OK as it's a very CRuby/YJIT-specific gem, considering the generated code it emits (huge methods).

You should have no problem handling that generated code. It's not CRuby-specific.

eregon commented 1 month ago

It executes correctly, it's just that https://github.com/Shopify/yjit-bench/blob/d3ce8dace054c0ac1a3339b69e1d5b686fc5868c/benchmarks/protoboeuf/benchmark_pb.rb#L888 is too big to be compiled by a method JIT (1400 lines long). Anyway, I shouldn't have mentioned that here, it's pretty much orthogonal.

k0kubun commented 1 month ago

The only case I recall is https://github.com/Shopify/yjit-bench/pull/262, were there many others?

It had three benchmarks. yjit-bench used to stop at a single benchmark failure, so it was three round-trips, which were already too many for me. We have two others failing right now that happened separately, so five cases in total.

That's fair, we should try switching to a release of TruffleRuby.

:+1:

maximecb commented 1 month ago

too big to be compiled by a method JIT (1400 lines long).

I think you mean too big for your method JIT. Google's V8 handles this sort of thing just fine 😉

eregon commented 1 month ago

Actually it compiles fine, my bad, probably because it's a lot of very simple arithmetic, and the benchmark likely uses only a small portion of the many branches: [engine] opt done engine=1 id=2126 ProtoBoeuf::TrunkItem#decode_from |Tier 2|Time 236( 110+126 )ms|AST 11185|Inlined 47Y 0N|IR 4401/ 7516|CodeSize 29163|Addr 0x7f20e00da000|UTC 2024-09-27T10:10:02.328|Src benchmark_pb.rb:888 0x2acaf309

In the source code it certainly looks like a lot of duplication and something the JIT should be able to decide whether to inline or not (which it could if it's a separate method).

I'll make a PR to switch to a release of TruffleRuby in CI: https://github.com/Shopify/yjit-bench/pull/334