search

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust
MIT License
125 stars 26 forks source link

Can't compile on Ruby 3.0: No builder for extension 'ext/tiktoken_ruby/Cargo.toml' #1

Closed taf2 closed 1 year ago

taf2 commented 1 year ago

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

No builder for extension 'ext/tiktoken_ruby/Cargo.toml'

Gem files will remain installed in /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/tiktoken_ruby-0.0.3 for inspection. Results logged to /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/extensions/arm64-darwin-22/3.0.0/tiktoken_ruby-0.0.3/gem_make.out

/Users/taf2/.rvm/rubies/ruby-3.0.5/lib/ruby/3.0.0/rubygems/ext/builder.rb:144:in build_error' /Users/taf2/.rvm/rubies/ruby-3.0.5/lib/ruby/3.0.0/rubygems/ext/builder.rb:125:inbuilder_for' /Users/taf2/.rvm/rubies/ruby-3.0.5/lib/ruby/3.0.0/rubygems/ext/builder.rb:150:in build_extension' /Users/taf2/.rvm/rubies/ruby-3.0.5/lib/ruby/3.0.0/rubygems/ext/builder.rb:193:inblock in build_extensions' /Users/taf2/.rvm/rubies/ruby-3.0.5/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in each' /Users/taf2/.rvm/rubies/ruby-3.0.5/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:inbuild_extensions' /Users/taf2/.rvm/rubies/ruby-3.0.5/lib/ruby/3.0.0/rubygems/installer.rb:837:in build_extensions' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/rubygems_gem_installer.rb:72:inbuild_extensions' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/rubygems_gem_installer.rb:28:in install' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/source/rubygems.rb:200:ininstall' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/installer/gem_installer.rb:54:in install' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/installer/gem_installer.rb:16:ininstall_from_spec' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:155:in do_install' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:146:inblock in worker_pool' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:62:in apply_func' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:57:inblock in process_queue' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:54:in loop' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:54:inprocess_queue' /Users/taf2/.rvm/gems/ruby-3.0.5@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing tiktoken_ruby (0.0.3), and Bundler cannot continue.

In Gemfile: tiktoken_ruby

taf2 commented 1 year ago

Looks like need ruby 3.2?

IAPark commented 1 year ago

Whops, thanks for letting me know. I think the precompiled native extension should work fine with Ruby 3.0, but I can reproduce a problem installing with gem install tiktoken_ruby --platform ruby and that was meant to work. I think if I switch back to rb_sys it will work in earlier version so I'll try to get a version out with that change here relatively soon

IAPark commented 1 year ago

This should be fixed in 0.0.4, let me know if it seems like it isn't

taf2 commented 1 year ago

It's fantastic that you provided the precompiled binaries for x86, arm for both linux and mac! I'm having an issue on ruby 3.2 however adding this to bundler via bundle add tiktoken_ruby, i am getting:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby
/Users/taf2/.rvm/rubies/ruby-3.2.2/bin/ruby extconf.rb
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --without-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/Users/taf2/.rvm/rubies/ruby-3.2.2/bin/$(RUBY_BASE_NAME)
<internal:/Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file --
rb_sys/mkmf (LoadError)
    from <internal:/Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from extconf.rb:4:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4 for inspection.
Results logged to /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/extensions/arm64-darwin-22/3.2.0-static/tiktoken_ruby-0.0.4/gem_make.out

  /Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:119:in `run'
  /Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:27:in `build'
  /Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:187:in `build_extension'
  /Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:221:in `block in build_extensions'
  /Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in `each'
  /Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in `build_extensions'
  /Users/taf2/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/installer.rb:843:in `build_extensions'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/rubygems_gem_installer.rb:72:in `build_extensions'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/rubygems_gem_installer.rb:28:in `install'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/source/rubygems.rb:200:in `install'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/gem_installer.rb:54:in `install'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:155:in `do_install'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:146:in `block in worker_pool'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:62:in `apply_func'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:57:in `block in process_queue'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:54:in `loop'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:54:in `process_queue'
  /Users/taf2/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing tiktoken_ruby (0.0.4), and Bundler cannot continue.

In Gemfile:
  tiktoken_ruby

shell returned 5

What is interesting is i have downloaded the binary versions and they work fine for all 4 environments. However bundler seems to insist on the source version even when i try to add it to my local cache and use --local it still insists on the source version... any suggestions?

Thanks!

taf2 commented 1 year ago

Looks like we need rb_sys gem as well in the Gemfile...

IAPark commented 1 year ago

Interesting, rb_sys is marked as a dependency of this project in the gemfile so I'm a bit confused. Are you saying adding rb_sys to your project's gemfile fixed the issue? Did mkmf.log have any more details?

taf2 commented 1 year ago

Here's an updated on what I'm seeing:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby
/home/deployer/.rvm/rubies/ruby-3.2.2/bin/ruby extconf.rb
checking for cargo... no

current directory:
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby
make DESTDIR\= sitearchdir\=./.gem.20230501-18612-s2102z
sitelibdir\=./.gem.20230501-18612-s2102z clean

current directory:
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby
make DESTDIR\= sitearchdir\=./.gem.20230501-18612-s2102z
sitelibdir\=./.gem.20230501-18612-s2102z
info: downloading installer
info: profile set to 'minimal'
info: default host triple is x86_64-unknown-linux-gnu
info: skipping toolchain installation

Rust is installed now. Great!

To get started you need Cargo's bin directory
(/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
in your PATH
environment variable. This has not been done automatically.

To configure your current shell, run:
source
"/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2023-04-20, rust version 1.69.0 (84c898d65 2023-04-16)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-std'
info: installing component 'rustc'

stable-x86_64-unknown-linux-gnu installed - rustc 1.69.0 (84c898d65
2023-04-16)

info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
info: checking for self-update
info: using existing install for 'stable-x86_64-unknown-linux-gnu'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

stable-x86_64-unknown-linux-gnu unchanged - rustc 1.69.0 (84c898d65
2023-04-16)

info: note that the toolchain 'stable-x86_64-unknown-linux-gnu' is currently in
use (environment override by RUSTUP_TOOLCHAIN)
generating target/release/libtiktoken_ruby.so (release)
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo
rustc  --manifest-path ./Cargo.toml --target-dir target --lib --release -- -C
linker=gcc -L native=/home/deployer/.rvm/rubies/ruby-3.2.2/lib -C link-arg=-lm
-l pthread
    Updating crates.io index
    Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
    Updating git submodule `https://github.com/zurawiki/tiktoken`
 Downloading crates ...
  Downloaded magnus-macros v0.3.0
  Downloaded peeking_take_while v0.1.2
  Downloaded cfg-if v1.0.0
  Downloaded shlex v1.1.0
  Downloaded rustc-hash v1.1.0
  Downloaded rb-sys v0.9.68
  Downloaded autocfg v1.1.0
  Downloaded rb-sys-env v0.1.2
  Downloaded scopeguard v1.1.0
  Downloaded bit-set v0.5.3
  Downloaded shell-words v1.1.0
  Downloaded lazycell v1.3.0
  Downloaded smallvec v1.10.0
  Downloaded lock_api v0.4.9
  Downloaded rb-sys-build v0.9.68
  Downloaded libloading v0.7.4
  Downloaded glob v0.3.1
  Downloaded quote v1.0.26
  Downloaded lazy_static v1.4.0
  Downloaded bit-vec v0.6.3
  Downloaded bitflags v1.3.2
  Downloaded proc-macro2 v1.0.52
  Downloaded once_cell v1.17.1
  Downloaded cexpr v0.6.0
  Downloaded parking_lot v0.12.1
  Downloaded parking_lot_core v0.9.7
  Downloaded clang-sys v1.6.0
  Downloaded anyhow v1.0.70
  Downloaded unicode-ident v1.0.8
  Downloaded serde v1.0.157
  Downloaded memchr v2.5.0
  Downloaded minimal-lexical v0.2.1
  Downloaded base64 v0.21.0
  Downloaded aho-corasick v0.7.20
  Downloaded magnus v0.4.4
  Downloaded fancy-regex v0.11.0
  Downloaded nom v7.1.3
  Downloaded regex-automata v0.1.10
  Downloaded bindgen v0.60.1
  Downloaded regex v1.7.1
  Downloaded syn v1.0.109
  Downloaded regex-syntax v0.6.28
  Downloaded libc v0.2.140
  Downloaded bstr v1.4.0
   Compiling memchr v2.5.0
   Compiling proc-macro2 v1.0.52
   Compiling quote v1.0.26
   Compiling glob v0.3.1
   Compiling unicode-ident v1.0.8
   Compiling clang-sys v1.6.0
   Compiling libc v0.2.140
   Compiling cfg-if v1.0.0
   Compiling syn v1.0.109
   Compiling minimal-lexical v0.2.1
   Compiling nom v7.1.3
   Compiling libloading v0.7.4
   Compiling aho-corasick v0.7.20
   Compiling bindgen v0.60.1
   Compiling regex-syntax v0.6.28
   Compiling cexpr v0.6.0
   Compiling regex v1.7.1
   Compiling peeking_take_while v0.1.2
   Compiling rustc-hash v1.1.0
   Compiling lazycell v1.3.0
   Compiling bitflags v1.3.2
   Compiling shlex v1.1.0
   Compiling lazy_static v1.4.0
   Compiling shell-words v1.1.0
   Compiling rb-sys-build v0.9.68
   Compiling autocfg v1.1.0
   Compiling lock_api v0.4.9
   Compiling rb-sys v0.9.68
   Compiling parking_lot_core v0.9.7
error: failed to run custom build command for `rb-sys v0.9.68`

Caused by:
process didn't exit successfully:
`/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby/target/release/build/rb-sys-a89efa31a6642183/build-script-main`
(exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-env-changed=RBCONFIG_ruby_version
  cargo:rerun-if-env-changed=RBCONFIG_platform
  cargo:rerun-if-env-changed=RUBY_ROOT
  cargo:rerun-if-env-changed=RUBY_VERSION
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-changed=build/main.rs
  cargo:rerun-if-changed=build/version.rs
  cargo:rerun-if-changed=build/features.rs
  cargo:rerun-if-changed=build/ruby_macros.rs
  cargo:rerun-if-env-changed=RUBY_STATIC
  cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
  cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS

  --- stderr
Using bindgen with clang args:
["-I/home/deployer/.rvm/rubies/ruby-3.2.2/include/ruby-3.2.0",
"-I/home/deployer/.rvm/rubies/ruby-3.2.2/include/ruby-3.2.0/x86_64-linux",
"-fms-extensions", "-O3", "-fno-fast-math", "-ggdb3", "-Wall", "-Wextra",
"-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond",
"-Wimplicit-function-declaration", "-Wimplicit-int", "-Wmisleading-indentation",
"-Wpointer-arith", "-Wwrite-strings", "-Wold-style-definition",
"-Wimplicit-fallthrough=0", "-Wmissing-noreturn", "-Wno-cast-function-type",
"-Wno-constant-logical-operand", "-Wno-long-long",
"-Wno-missing-field-initializers", "-Wno-overlength-strings",
"-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign",
"-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value",
"-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn",
"-Wunused-variable", "-Wundef"]
thread 'main' panicked at 'Unable to find libclang: "couldn't find any valid
shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*',
'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where
one of these files can be found (invalid: [])"',
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/github.com-1ecc6299db9ec823/bindgen-0.60.1/src/lib.rs:2172:31
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
make: *** [target/release/libtiktoken_ruby.so] Error 101

make failed, exit code 2

Gem files will remain installed in
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/tiktoken_ruby-0.0.4 for
inspection.
Results logged to
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/extensions/x86_64-linux/3.2.0/tiktoken_ruby-0.0.4/gem_make.out

/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:119:in
`run'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:51:in
`block in make'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:43:in
`each'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:43:in
`make'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:41:in
`build'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:187:in
`build_extension'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:221:in
`block in build_extensions'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in
`each'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in
`build_extensions'
/home/deployer/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/installer.rb:843:in
`build_extensions'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/source/rubygems.rb:200:in
`install'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/gem_installer.rb:54:in
`install'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:155:in
`do_install'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:146:in
`block in worker_pool'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:62:in
`apply_func'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:57:in
`block in process_queue'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:54:in
`loop'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:54:in
`process_queue'
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/worker.rb:90:in
`block (2 levels) in create_threads'

An error occurred while installing tiktoken_ruby (0.0.4), and Bundler cannot
continue.

In Gemfile:
  tiktoken_ruby
checking deploy state
/home/deployer/.rvm/gems/ruby-3.2.2@ctm3/gems/bundler-2.4.4/lib/bundler/definition.rb:524:in `materialize': Could not find tiktoken_ruby-0.0.4 in locally installed gems (Bundler::GemNotFound)
taf2 commented 1 year ago

Quick update thought my issue might* be related to this bundler issue: https://github.com/rubygems/rubygems/issues/4269

diaclavijo commented 1 year ago

Following answer by @taf2

I am using Apple M1, so running this solved the issue

bundle lock --add-platform arm64-darwin-21