Static Ruby

[ankane]

Hey @jasonroelofs, thanks for working on this library for 10+ years! 🎉

Do mind explaining a bit about the complications of making Rice work with static Ruby?

It looks like the check was added in 2013 (a594b04cae609f6a3db5a482ad2c37b4900f914f), disabled in 2015 (83fccdbfadbaddb00e090966bebdd0488b9f716f), and reenabled a month later (1e01905a0e26daf10bdd4c4029c0daa045515f05) with the comment:

1) Heroku runs it's own custom static-link Ruby which we can't get working with Rice.

2) Ruby 2.2.2 and on has some serious issues with static linking in general and there doesn't seem to be any pressure to fix this.

Do you know if more recent versions of Ruby and/or Heroku stacks have addressed this?

Edit: The gem tests fail against static Ruby 2.6.5, so that may be the answer to the 2nd question (unless more is required than commenting out the ENABLE_SHARED check). This latest release installs successfully on the Heroku-18 stack.

[jasonroelofs]

Hey, good question. Reading up on those commits and the associated PRs, I would expect that trying to build Rice against a static Ruby is still difficult, if not impossible depending on the system in question. However it has been a number of years since the last attempts and discussions so I would expect quite a bit has changed since then so it's hard to say without trying it.

Have you run into issues that require a static build? Or are trying to build against a static Ruby and are getting errors?

[ankane]

Hey @jasonroelofs, thanks for the response. I'm hoping to get Rice and the rbenv defaults to play nicely together (the Rice tests currently fail without --enable-shared).

Some background: The past few months, I've been working on improving the ML ecosystem for Ruby (creating bindings for XGBoost, LightGBM, TensorFlow, and other libraries that have a C API). Many ML libraries are written in C++ for performance. One thing I've found is that pybind11 is an extremely important part of the Python ecosystem, as it makes it easy to interface with C++ when a C API is not present, which is often.

From what I've seen, Rice is the best Ruby library for interfacing with C++. However, it's a jarring experience for users on rbenv (which is pretty popular), as it's the only gem I've seen that requires you to reinstall Ruby (granted, it's different than most gems).

I think there are two primary ways to address this:

1. Get Rice working with static Ruby
2. Get rbenv to build with --enable-shared by default

I imagine that 2 will be an easier change, so I've followed up w/ rbenv here: https://github.com/rbenv/ruby-build/issues/35.

[jasonroelofs]

Awesome! I do want to be sure you know the state of Rice. I don't actually use the library for anything and haven't for some time, but I am happy to look into things if they aren't working (e.g. normal "maintenance mode").

As far as I'm aware, Rice is the only library for Ruby of its kind. It's not an easy problem to solve and was definitely influenced by Boost.Python though designed in a way that's more for Ruby, of course. The successor you linked, pybind11 looks interesting. The implementation details of Rice are pretty gnarley and I'd be interested to see what better ways are possible with newer C++ features. So thanks for linking to that!

This is the first I've heard of people having problems with rbenv + Rice, though maybe they just rebuild with --enabled-shared and move on from there. I've got some free time this week and next, I'll make a note to look into static vs shared Ruby again and see if things have improved.

[ankane]

--enable-shared is now the default for rbenv (https://github.com/rbenv/ruby-build/pull/1368). I'm still curious as to why Rice doesn't work with static Ruby, but this should make the installation process much smoother for rbenv users.

[jasonroelofs]

Got curious and tried to get a static Ruby built on my machine to see what's up again, and ran into quite a few problems just getting Ruby itself compiled. Now, I could get Ruby core compiled as a static lib pretty easily but many parts of the standard library simply don't build (e.g. objspace). So I'm not really sure if this is much of an issue anymore.

I'm going to set up a test Heroku app to see what the state of things is over there as that was the source of the initial conflict.

[jasonroelofs]

I had no problem on Ruby 2.7 putting up an app on Heroku that used your fasttext gem. I think I'm going to close this out as a non-problem anymore. I would need to see a specific situation where someone needs and has a full static build of Ruby to be able to figure out why Rice doesn't work there. I'll go through docs and update accordingly.

[ankane]

Great, I think installation on Mac and Linux is in a good spot (with both rbenv and Heroku enabling shared).

I spent some time today testing installation on Windows. With AppVeyor, everything works great 🎉

I'm still working on GitHub Actions (Rice installs successfully, which is a great start).

[ankane]

And of course the next thing I try works for GitHub Actions: https://github.com/ankane/fastText/actions/runs/40962473 (looks like it still needs more work)

Anyways, will try to get the Rice test suite working.

[ankane]

A few more findings on GitHub Actions:

• Shared Ruby isn't support yet: https://github.com/actions/virtual-environments/issues/73
• MSYS2 (which AppVeyor uses) is probably needed for Windows: https://github.com/actions/virtual-environments/issues/229

Ref: https://github.com/ankane/fastText/runs/451864842?check_suite_focus=true