Cannot install tokenizer 0.3.2

[pribadi1st]

Hi i am new to this gem,

i tried to install gem with release tag 0.3.2 and this issue occurs

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/tokenizers-0.3.2/ext/tokenizers
/Users/pribadiridwan/.rbenv/versions/3.0.0/bin/ruby -I
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0 extconf.rb
checking for cargo... no

current directory: /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/tokenizers-0.3.2/ext/tokenizers
make DESTDIR\= sitearchdir\=./.gem.20230307-17603-x8spx4 sitelibdir\=./.gem.20230307-17603-x8spx4 clean

current directory: /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/tokenizers-0.3.2/ext/tokenizers
make DESTDIR\= sitearchdir\=./.gem.20230307-17603-x8spx4 sitelibdir\=./.gem.20230307-17603-x8spx4
info: downloading installer
info: profile set to 'minimal'
info: default host triple is aarch64-apple-darwin
info: skipping toolchain installation

Rust is installed now. Great!

To get started you need Cargo's bin directory
(/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/tokenizers-0.3.2/ext/tokenizers/.rb-sys/stable/cargo/bin)
in your PATH
environment variable. This has not been done automatically.

To configure your current shell, run:
source
"/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/tokenizers-0.3.2/ext/tokenizers/.rb-sys/stable/cargo/env"
make: rustup: No such file or directory
make: ***
[/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/tokenizers-0.3.2/ext/tokenizers/.rb-sys/stable/cargo/bin/cargo]
Error 1

make failed, exit code 2

Gem files will remain installed in /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/tokenizers-0.3.2
for inspection.
Results logged to
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/extensions/arm64-darwin-21/3.0.0/tokenizers-0.3.2/gem_make.out

  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:102:in `run'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:51:in `block in make'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:43:in `each'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:43:in `make'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:42:in `build'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:170:in `build_extension'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:204:in `block in
build_extensions'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:201:in `each'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/ext/builder.rb:201:in `build_extensions'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/site_ruby/3.0.0/rubygems/installer.rb:843:in `build_extensions'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/source/rubygems.rb:200:in
`install'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/installer/gem_installer.rb:54:in
`install'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/installer/parallel_installer.rb:167:in
`do_install'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/installer/parallel_installer.rb:158:in
`block in worker_pool'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/worker.rb:62:in
`apply_func'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/worker.rb:57:in `block
in process_queue'
  /Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/worker.rb:54:in `loop'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/worker.rb:54:in
`process_queue'
/Users/pribadiridwan/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/bundler-2.4.7/lib/bundler/worker.rb:90:in `block
(2 levels) in create_threads'

An error occurred while installing tokenizers (0.3.2), and Bundler cannot continue.

In Gemfile:
  tokenizers

however if i downgrade it to 0.3.2 its working

my application system ruby 3.0.0 rails 7.+ rbenv 1.2.0

do you have any idea why ? thank you

[ankane]

Hey @pribadi1st, it looks like it's trying to install the source version, which requires a Rust compiler (although it looks like rb-sys is trying to install and use a local one and failing). Can you paste the output of bundle platform and gem list tokenizers?

[MichaelHoste]

Hi @ankane,

I have the same issue with 0.3.2 and downgrading doesn't work. I had to install rust (brew install rust) for bundle install to pass.

---

bundle platform

Your platform is: arm64-darwin22

Your app has gems that work on these platforms:
* ruby
* x86_64-linux

Your Gemfile specifies a Ruby version requirement:
* ruby 3.1.3p185

Your current platform satisfies the Ruby version requirement.

gem list tokenizers

*** LOCAL GEMS ***

---

Maybe this ticket should be opened again?

[ankane]

Hey @MichaelHoste, I've reported the issue to rb_sys: https://github.com/oxidize-rb/rb-sys/issues/183

[ankane]

Also, if you run bundle lock --add-platform arm64-darwin, it will install the precompiled version instead of building from source.

[MichaelHoste]

Thanks for the tips @ankane ! It seems to work.

I guess this issue can stay closed then if it's related to rb_sys.