search

ankane / tokenizers-ruby

Fast state-of-the-art tokenizers for Ruby
Apache License 2.0
132 stars 6 forks source link

Use `rake-compiler` and `rb-sys` for build #1

Closed ianks closed 1 year ago

ianks commented 2 years ago

This PR changes the build system to use rb-sys and rake-compiler to build the gem. With this change, it will be possible to publish native binaries of the gem so that consumers will not have to maintain a rust toolchain for deployment.

The actual publishing of the gem is not handled by this PR, but the cross-build Github action will build a bunch of *.gem files which can pushed to Rubygems.

❤️

ankane commented 2 years ago

Hey @ianks, thanks for the PR, and all the work you're doing to make it easier to use Rust with Ruby. Will try this out when I have some time. Is it possible to make rake-compiler a development dependency (instead of runtime)?

kojix2 commented 2 years ago

I hope this wonderful gem will be easy to install.

ankane commented 1 year ago

Hey @ianks, sorry for the delay. Switched to rb_sys + rake-compiler and the directory structure from https://github.com/rubygems/rubygems/pull/5613 in the commit above (wanted to do this separately from precompiling). Will hopefully try out the cross-gem-action soon.

Thanks again for all the work on rb_sys and driving this!

ankane commented 1 year ago

Hey @ianks,

Added precompiling for a few platforms in the commit above (it currently fails on aarch64-linux since OpenSSL isn't available). It'd be nice if rb_sys provided a way to cross-compile locally as well (if it doesn't already, like rake-compiler-dock).

Thanks again for all the work you've done on Ruby + Rust!

ankane commented 1 year ago

Just released 0.2.0 with precompiled versions for Mac and x86-64 Linux.

ianks commented 1 year ago

🔥 🔥 🔥

Added precompiling for a few platforms in the commit above (it currently fails on aarch64-linux since OpenSSL isn't available). It'd be nice if rb_sys provided a way to cross-compile locally as well (if it doesn't already, like rake-compiler-dock).

Just literally published a fix for that today in v0.9.47. Hopefully that fixed it!

. It'd be nice if rb_sys provided a way to cross-compile locally as well (if it doesn't already, like rake-compiler-dock).

Ah yes, it is actually just rake-compiler-dock under the hood so it should work fine. Admittedly, I don't do it often because currently arm64 builds for rcd aren't published. And since I'm on an M1, they are slowwwwww. You can see how it's done here.

ankane commented 1 year ago

Thanks @ianks!

Tried v0.9.47, but think it may solve a different issue (the aarch64-linux failure is due to a missing header required by openssl-sys - need to spend more time on it).

ianks commented 1 year ago

Oof yeah that’s a bit of a doozy. Providing a cross platform version of OpenSSL should be possible in the setup of cross gem action, but ideally the dep on openssl-sys could be removed entirely in favor of rustls. I think this may be coming from the http feature of the tokenizers crate? Not sure though.