What is Helix doing that affects performance so much?

[bbugh]

Hi, I'm exploring Rust as an extension for Ruby for a lot of expensive calculations. I made a project that implements a simple financial algorithm in six ways (Ruby, C, Helix, ruru, FFI). They are all implemented with the minimum viable code to allow Ruby to be able to call a Rust function cash_flow.

The benchmark for Helix was surprising, and I'm curious what is unique about Helix that causes the function calls to return Ruby so slowly in comparison with the other methods.

As you can see from the numbers below, helix's iterations per second when called from Ruby are almost half of ruru, C, and Ruby for a simple function:

Warming up --------------------------------------
         ruby method    203479 i/100ms
 rust helix instance    120885 i/100ms
    rust helix class    121661 i/100ms
      rust ffi class    161558 i/100ms
     rust ruru class    199846 i/100ms
             c class    221703 i/100ms
Calculating -------------------------------------
                      iterations per second     total iterations    time
         ruby method  4966573.4 (±6.8%) i/s -   24824438          in 5.022462s
 rust helix instance  1875397.8 (±6.1%) i/s -    9429030          in 5.046921s
    rust helix class  1852779.7 (±5.9%) i/s -    9246236          in 5.008588s
      rust ffi class  3082134.8 (±8.1%) i/s -   15348010          in 5.019943s
     rust ruru class  4275527.6 (±6.2%) i/s -   21383522          in 5.021156s
             c class  5483016.5 (±6.0%) i/s -   27491172          in 5.032074s

However, when running a criterion benchmark for the function within the Rust repository, the performance is superb:

Benchmarking cash_flow
Benchmarking cash_flow: Warming up for 3.0000 s
Benchmarking cash_flow: Collecting 100 samples in estimated 5.0000 s (955,252,950 iterations)
Benchmarking cash_flow: Analyzing
cash_flow               time:   [5.2014 ns 5.2626 ns 5.3245 ns]
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild
slope  [5.2014 ns 5.3245 ns] R^2            [0.8278649 0.8276527]
mean   [5.2079 ns 5.3387 ns] std. dev.      [268.40 ps 395.58 ps]
median [5.1706 ns 5.3213 ns] med. abs. dev. [208.08 ps 356.25 ps]

This is a significant difference between the actual function and whatever Helix is doing to connect Ruby to Rust. Obviously with interop there's going to be some performance drop, but as you can see the other methods were approximately comparable.

I want to dig deeper into it because Helix was the best API and usability of all of the methods I tried, but I want to know exactly why the performance is inhibited before we implement critical code with it. Any ideas? Thank you!

[rafbgarcia]

@bbugh what did you end up doing? Did you figure out what was the deal with Helix?

[bbugh]

@rafbgarcia wow, I cant believe this was already more than a year ago!

After a deep dive with all of the options, we ended up writing C extensions, for the sake of long term maintainability and speed. We know that the C integration will continue to evolve alongside Ruby since it's the official API, but didn't feel that we could confidently say the same of any of the Rust libraries (and at the time, Rutie was the only one still actively being developed). The C extensions are also very portable since anywhere you deploy Ruby also has to be able to build C for things like nokigiri, but using a Rust gem would have required additional infrastructure. Our use case doesn't do any allocation and most of the appealing Rust features (like Option) wouldn't be applicable. Given all the cons and very few pros over C, the trade offs were not worth it. I'd love for the Ruby community to embrace using Rust for gems, but at the time, I don't think it was production-ready. I am not sure what the state of things are now.

[rafbgarcia]

That makes sense, thanks for responding!

[wagenet]

Unfortunately, there have been some problems in Helix that have been difficult for us to resolve, as much as we love the API! In our app, Skylight, we continue to use Rust with C-compatible method signatures and a small C glue layer to tie that in to Ruby.

[rafbgarcia]

@wagenet is it the case to deprecate the project then?

[rafbgarcia]

I say that because I don't know if I would have spent the time I did yesterday looking at the project if I knew that Skylight is not using it anymore because they found some problems.

[wagenet]

@rafbgarcia sorry about that, we didn't intend to mislead. We've actually never used it in production ourselves though we did hope to. At this point deprecation in the right thing to do and it's something that's on our radar. My apologies for your wasted effort 😞

[rafbgarcia]

@wagenet no problem, it was interesting to learn about it anyways :)

[wagenet]

@rafbgarcia it's something I would love to see revisited, it just turned out that we'd basically need to rework things significantly and we haven't had the bandwidth to do so when our current solution does the job for us.