Truffle is slower than MRI

[phortx]

Hi folks,

I have a simple ruby written CLI tool, for which performance is really important. So I wanted to give truffleruby a try today.

Before that I added a simple benchmark to be able to compare the performance. But from all I've read about truffleruby and all the benchmark charts I've seen, I expected truffleruby to be much faster.

With MRI 2.6.3 and --jit flag, my cli tool needs 1.150s to run.

I'm on macos and installed truffle via asdf-ruby (which uses ruby-build): asdf install ruby truffleruby-1.0.0-rc15:

HEAD is now at 44b0c75 Merge pull request #1299 from koic/remotes/upstream/add_definition_for_ruby_2_6_3
ruby-build: use openssl from homebrew
Downloading truffleruby-1.0.0-rc15-macos-amd64.tar.gz...
-> https://dqw8nmjcqpjn7.cloudfront.net/8b664a836ec080ddee043ae78de4d2c362ae840706dc045ede6c40af8533a6f2
Installing truffleruby-1.0.0-rc15...
Using the precompiled OpenSSL C extension
TruffleRuby was successfully installed in /Users/benjamin.klein/.asdf/installs/ruby/truffleruby-1.0.0-rc15
Installed truffleruby-1.0.0-rc15 to /Users/benjamin.klein/.asdf/installs/ruby/truffleruby-1.0.0-rc15

Looks good so far:

> $ ruby --version
truffleruby 1.0.0-rc15, like ruby 2.6.2, GraalVM CE Native [x86_64-darwin]

I ran bundle and then executed my gem. I was confused because it took way longer: 11.478s.

So I've added some benchmark calls (via the ruby benchmark module) to see whether the code is slow at all or just parts (maybe I make use of a slow feature or whatever).

Results:

ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18] --jit
[Benchmark] requires:   0.115472   0.070992   0.384875 (  0.225389)
[Benchmark] banner:   0.000329   0.000124   0.000453 (  0.000452)
[Benchmark] gli setup:   0.000013   0.000007   0.000020 (  0.000019)
[Benchmark] setup builtin commands:   0.000315   0.000052   0.000367 (  0.000367)
[Benchmark] setup interface commands:   0.002239   0.001173   0.003412 (  0.003448)
[Benchmark] run:   0.002271   0.000712   0.002983 (  0.003021)
real    0m1.612s

truffleruby 1.0.0-rc15, like ruby 2.6.2, GraalVM CE Native [x86_64-darwin]
[Benchmark] requires:   0.804971   0.132401   0.937372 (  0.807652)
[Benchmark] banner:   0.080212   0.001214   0.081426 (  0.029650)
[Benchmark] gli setup:   0.000210   0.000004   0.000214 (  0.000160)
[Benchmark] setup builtin commands:   0.054000   0.000604   0.054604 (  0.022413)
[Benchmark] setup interface commands:   0.204533   0.006994   0.211527 (  0.091065)
[Benchmark] run:   0.230432   0.004089   0.234521 (  0.102056)
real    0m14.332s

So truffle is slower in all aspects of the script. Especially the startup time.

Now I'm not sure whether my installation is broken, truffle is very slow or not suitable for small programs.

Any ideas? :)

[chrisseaton]

Hi thanks for trying out TruffleRuby - we appreciate it.

Before that I added a simple benchmark to be able to compare the performance. But from all I've read about truffleruby and all the benchmark charts I've seen, I expected truffleruby to be much faster.

TruffleRuby is designed to be primarily faster for longer-running workloads, such as servers. The technology we use to optimise Ruby - just-in-time compilation - needs time to profile your application and then compile it at runtime. This warm up period is needed to be effective. This isn't unique to TruffleRuby - the same thing applies to JRuby, the V8 JavaScript interpreter in Node.js, and others. The MRI JIT doesn't need it because it's currently simpler. As it gets more advanced it'll need more warm-up period.

I'm on macos and installed truffle via asdf-ruby (which uses ruby-build): asdf install ruby truffleruby-1.0.0-rc15:

Could you try a more recent version of TruffleRuby? Please try 19.0.0. This probably won't make a big difference, however.

I ran bundle and then executed my gem. I was confused because it took way longer: 11.478s.

Yes, so with such a short-running program the just-in-time compiler doesn't really have long enough to run and make a difference.

Now - we are interested in short-running programs, and we want to make these fast as well. We have some unique technology here, where we're looking at pre-initialising programs so that we remove the overhead of loading and parsing Ruby code, and setting up libraries, but these are just proposals at the moment.

So I've added some benchmark calls (via the ruby benchmark module) to see whether the code is slow at all or just parts (maybe I make use of a slow feature or whatever).

Which benchmark tool are you using here? We'd normally recommend benchmark-ips, because it's designed to account for the warm-up period.

or not suitable for small programs.

It isn't focused on small programs at the moment, that's true. We're working on improving that.

Any ideas? :)

Are you able to share the code of your program and benchmarks? Either publicly or privately? So we could investgiate ourselves.

[phortx]

Thank you very much for the comprehensive answer! :)

I understand that truffle targets longer running processes. However I love ruby but I'm also bothered by rubys performance and thus I'm searching for alternatives.

I'll send you a mail with the repo of the project.

[eregon]

FWIW, there are cases where we're fast on short-running programs, see https://eregon.me/blog/2018/02/19/truffleruby-native-fast-short-scripts.html Those programs load very little code though, I think part of the bottleneck currently is loading code, as you noticed (requires). I think we can improve quite a bit there, but it's much easier to optimize with a given workload.