search

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust
MIT License
109 stars 26 forks source link

Install failing on Ruby 3.3.0 and Heroku #20

Closed kapso closed 4 months ago

kapso commented 6 months ago

I am using Ruby 3.3.0-rc1, it works great with v3.2.2

Error....

Fetching tiktoken_ruby 0.0.6
       Installing tiktoken_ruby 0.0.6 with native extensions
       Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

       current directory:
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/bin/ruby extconf.rb
       checking for gcc... yes
       checking for g++... yes
       checking for gcc-ar... yes
       checking for cargo... no

       current directory:
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
       make DESTDIR\= sitearchdir\=./.gem.20231212-4291-rgiys4
       sitelibdir\=./.gem.20231212-4291-rgiys4 clean

       current directory:
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
       make DESTDIR\= sitearchdir\=./.gem.20231212-4291-rgiys4
       sitelibdir\=./.gem.20231212-4291-rgiys4
       info: downloading installer
       info: profile set to 'minimal'
       info: default host triple is x86_64-unknown-linux-gnu
       info: skipping toolchain installation

       Rust is installed now. Great!

       To get started you need Cargo's bin directory 
       (/tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
       in your PATH
       environment variable. This has not been done automatically.

       To configure your current shell, run:
       source 
       "/tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
       info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
       info: latest update on 2023-12-07, rust version 1.74.1 (a28077b28 2023-12-04)
       info: downloading component 'cargo'
       info: downloading component 'rust-std'
       info: downloading component 'rustc'
       info: installing component 'cargo'
       info: installing component 'rust-std'
       info: installing component 'rustc'

       stable-x86_64-unknown-linux-gnu installed - rustc 1.74.1 (a28077b28
       2023-12-04)

       info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
       info: checking for self-update
       info: using existing install for 'stable-x86_64-unknown-linux-gnu'
       info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

       stable-x86_64-unknown-linux-gnu unchanged - rustc 1.74.1 (a28077b28
       2023-12-04)

       info: note that the toolchain 'stable-x86_64-unknown-linux-gnu' is currently in
       use (environment override by RUSTUP_TOOLCHAIN)
       generating target/release/libtiktoken_ruby.so (release)
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo
       rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release
       -- -C linker=gcc -L native=/tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib -C
       link-arg=-lm -l pthread
           Updating crates.io index
           Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
           Updating git submodule `https://github.com/zurawiki/tiktoken`
        Downloading crates ...
         Downloaded bit-set v0.5.3
         Downloaded parking_lot v0.12.1
         Downloaded parking_lot_core v0.9.7
         Downloaded bindgen v0.66.1
         Downloaded proc-macro2 v1.0.66
         Downloaded lock_api v0.4.9
         Downloaded once_cell v1.17.1
         Downloaded memchr v2.5.0
         Downloaded scopeguard v1.1.0
         Downloaded magnus-macros v0.6.0
         Downloaded serde v1.0.157
         Downloaded bitflags v2.4.0
         Downloaded shell-words v1.1.0
         Downloaded shlex v1.1.0
         Downloaded peeking_take_while v0.1.2
         Downloaded regex-automata v0.1.10
         Downloaded libloading v0.7.4
         Downloaded fancy-regex v0.11.0
         Downloaded cfg-if v1.0.0
         Downloaded cexpr v0.6.0
         Downloaded libc v0.2.140
         Downloaded seq-macro v0.3.5
         Downloaded rustc-hash v1.1.0
         Downloaded rb-sys-env v0.1.2
         Downloaded rb-sys-build v0.9.81
         Downloaded rb-sys v0.9.81
         Downloaded lazycell v1.3.0
         Downloaded clang-sys v1.6.0
         Downloaded smallvec v1.10.0
         Downloaded regex-syntax v0.6.28
         Downloaded magnus v0.6.1
         Downloaded regex v1.7.1
         Downloaded syn v2.0.31
         Downloaded nom v7.1.3
         Downloaded unicode-ident v1.0.8
         Downloaded minimal-lexical v0.2.1
         Downloaded glob v0.3.1
         Downloaded bstr v1.4.0
         Downloaded lazy_static v1.4.0
         Downloaded autocfg v1.1.0
         Downloaded base64 v0.21.0
         Downloaded quote v1.0.33
         Downloaded bit-vec v0.6.3
         Downloaded anyhow v1.0.70
         Downloaded aho-corasick v0.7.20
          Compiling memchr v2.5.0
          Compiling proc-macro2 v1.0.66
          Compiling unicode-ident v1.0.8
          Compiling glob v0.3.1
          Compiling clang-sys v1.6.0
          Compiling libc v0.2.140
          Compiling quote v1.0.33
          Compiling cfg-if v1.0.0
          Compiling minimal-lexical v0.2.1
          Compiling syn v2.0.31
          Compiling nom v7.1.3
          Compiling libloading v0.7.4
          Compiling aho-corasick v0.7.20
          Compiling regex-syntax v0.6.28
          Compiling bindgen v0.66.1
          Compiling regex v1.7.1
          Compiling cexpr v0.6.0
          Compiling lazy_static v1.4.0
          Compiling bitflags v2.4.0
          Compiling lazycell v1.3.0
          Compiling peeking_take_while v0.1.2
          Compiling shlex v1.1.0
          Compiling rustc-hash v1.1.0
          Compiling shell-words v1.1.0
          Compiling autocfg v1.1.0
          Compiling lock_api v0.4.9
          Compiling parking_lot_core v0.9.7
          Compiling rb-sys-build v0.9.81
          Compiling rb-sys v0.9.81
          Compiling smallvec v1.10.0
          Compiling scopeguard v1.1.0
          Compiling rb-sys-env v0.1.2
          Compiling anyhow v1.0.70
       error: failed to run custom build command for `rb-sys v0.9.81`

       Caused by:
       process didn't exit successfully:
       `/tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/build/rb-sys-f895743fce8c911c/build-script-main`
       (exit status: 101)
         --- stdout
         cargo:rerun-if-env-changed=RUBY
         cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
         cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
         cargo:rerun-if-env-changed=RBCONFIG_platform
         cargo:rerun-if-env-changed=RUBY_ROOT
         cargo:rerun-if-env-changed=RUBY_VERSION
         cargo:rerun-if-env-changed=RUBY
         cargo:rerun-if-changed=build/main.rs
         cargo:rerun-if-changed=build/stable_api_config.rs
         cargo:rerun-if-changed=build/version.rs
         cargo:rerun-if-changed=build/features.rs
         cargo:rerun-if-env-changed=RUBY_STATIC
         cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
         cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
         cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
         cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
         cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
         cargo:rerun-if-env-changed=TARGET
         cargo:rerun-if-env-changed=TARGET
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS

         --- stderr
       INFO: using bindgen with clang args:
       ["-I/tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/include/ruby-3.3.0+0",
       "-I/tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/include/ruby-3.3.0+0/x86_64-linux",
       "-fms-extensions", "-O3", "-fno-fast-math", "-g", "-Wall", "-Wextra",
       "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond",
       "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wpointer-arith",
       "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0",
       "-Wmissing-noreturn", "-Wno-cast-function-type",
       "-Wno-constant-logical-operand", "-Wno-long-long",
       "-Wno-missing-field-initializers", "-Wno-overlength-strings",
       "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign",
       "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value",
       "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn",
       "-Wunused-variable", "-Wmisleading-indentation", "-Wundef"]
       thread 'main' panicked at
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31:
       Unable to find libclang: "couldn't find any valid shared libraries matching:
       ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the
       `LIBCLANG_PATH` environment variable to a path where one of these files can be
       found (invalid: [])"
         note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
       warning: build failed, waiting for other jobs to finish...
       make: *** [Makefile:566: target/release/libtiktoken_ruby.so] Error 101

       make failed, exit code 2

       Gem files will remain installed in
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6 for
       inspection.
       Results logged to
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/extensions/x86_64-linux/3.3.0+0/tiktoken_ruby-0.0.6/gem_make.out

       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:118:in
       `run'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:51:in
       `block in make'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:43:in
       `each'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:43:in
       `make'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/ext_conf_builder.rb:42:in
       `build'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:186:in
       `build_extension'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:220:in
       `block in build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:217:in
       `each'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:217:in
       `build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/installer.rb:861:in
       `build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/rubygems_gem_installer.rb:76:in
       `build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/rubygems_gem_installer.rb:28:in
       `install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/source/rubygems.rb:205:in
       `install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/gem_installer.rb:54:in
       `install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/gem_installer.rb:16:in
       `install_from_spec'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/parallel_installer.rb:129:in
       `do_install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/parallel_installer.rb:120:in
       `block in worker_pool'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:62:in
       `apply_func'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:57:in
       `block in process_queue'
         <internal:kernel>:187:in `loop'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:54:in
       `process_queue'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:90:in
       `block (2 levels) in create_threads'

       An error occurred while installing tiktoken_ruby (0.0.6), and Bundler cannot
       continue.

       In Gemfile:
         tiktoken_ruby
       Bundler Output: Fetching gem metadata from https://rubygems.org/........
       Fetching gem metadata from https://gems.contribsys.com/..
       Fetching rb_sys 0.9.83
       Installing rb_sys 0.9.83
       Fetching tiktoken_ruby 0.0.6
       Installing tiktoken_ruby 0.0.6 with native extensions
       Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

       current directory:
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/bin/ruby extconf.rb
       checking for gcc... yes
       checking for g++... yes
       checking for gcc-ar... yes
       checking for cargo... no

       current directory:
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
       make DESTDIR\= sitearchdir\=./.gem.20231212-4291-rgiys4
       sitelibdir\=./.gem.20231212-4291-rgiys4 clean

       current directory:
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
       make DESTDIR\= sitearchdir\=./.gem.20231212-4291-rgiys4
       sitelibdir\=./.gem.20231212-4291-rgiys4
       info: downloading installer
       info: profile set to 'minimal'
       info: default host triple is x86_64-unknown-linux-gnu
       info: skipping toolchain installation

       Rust is installed now. Great!

       To get started you need Cargo's bin directory 
       (/tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
       in your PATH
       environment variable. This has not been done automatically.

       To configure your current shell, run:
       source 
       "/tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
       info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
       info: latest update on 2023-12-07, rust version 1.74.1 (a28077b28 2023-12-04)
       info: downloading component 'cargo'
       info: downloading component 'rust-std'
       info: downloading component 'rustc'
       info: installing component 'cargo'
       info: installing component 'rust-std'
       info: installing component 'rustc'

       stable-x86_64-unknown-linux-gnu installed - rustc 1.74.1 (a28077b28
       2023-12-04)

       info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
       info: checking for self-update
       info: using existing install for 'stable-x86_64-unknown-linux-gnu'
       info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

       stable-x86_64-unknown-linux-gnu unchanged - rustc 1.74.1 (a28077b28
       2023-12-04)

       info: note that the toolchain 'stable-x86_64-unknown-linux-gnu' is currently in
       use (environment override by RUSTUP_TOOLCHAIN)
       generating target/release/libtiktoken_ruby.so (release)
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo
       rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release
       -- -C linker=gcc -L native=/tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib -C
       link-arg=-lm -l pthread
           Updating crates.io index
           Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
           Updating git submodule `https://github.com/zurawiki/tiktoken`
        Downloading crates ...
         Downloaded bit-set v0.5.3
         Downloaded parking_lot v0.12.1
         Downloaded parking_lot_core v0.9.7
         Downloaded bindgen v0.66.1
         Downloaded proc-macro2 v1.0.66
         Downloaded lock_api v0.4.9
         Downloaded once_cell v1.17.1
         Downloaded memchr v2.5.0
         Downloaded scopeguard v1.1.0
         Downloaded magnus-macros v0.6.0
         Downloaded serde v1.0.157
         Downloaded bitflags v2.4.0
         Downloaded shell-words v1.1.0
         Downloaded shlex v1.1.0
         Downloaded peeking_take_while v0.1.2
         Downloaded regex-automata v0.1.10
         Downloaded libloading v0.7.4
         Downloaded fancy-regex v0.11.0
         Downloaded cfg-if v1.0.0
         Downloaded cexpr v0.6.0
         Downloaded libc v0.2.140
         Downloaded seq-macro v0.3.5
         Downloaded rustc-hash v1.1.0
         Downloaded rb-sys-env v0.1.2
         Downloaded rb-sys-build v0.9.81
         Downloaded rb-sys v0.9.81
         Downloaded lazycell v1.3.0
         Downloaded clang-sys v1.6.0
         Downloaded smallvec v1.10.0
         Downloaded regex-syntax v0.6.28
         Downloaded magnus v0.6.1
         Downloaded regex v1.7.1
         Downloaded syn v2.0.31
         Downloaded nom v7.1.3
         Downloaded unicode-ident v1.0.8
         Downloaded minimal-lexical v0.2.1
         Downloaded glob v0.3.1
         Downloaded bstr v1.4.0
         Downloaded lazy_static v1.4.0
         Downloaded autocfg v1.1.0
         Downloaded base64 v0.21.0
         Downloaded quote v1.0.33
         Downloaded bit-vec v0.6.3
         Downloaded anyhow v1.0.70
         Downloaded aho-corasick v0.7.20
          Compiling memchr v2.5.0
          Compiling proc-macro2 v1.0.66
          Compiling unicode-ident v1.0.8
          Compiling glob v0.3.1
          Compiling clang-sys v1.6.0
          Compiling libc v0.2.140
          Compiling quote v1.0.33
          Compiling cfg-if v1.0.0
          Compiling minimal-lexical v0.2.1
          Compiling syn v2.0.31
          Compiling nom v7.1.3
          Compiling libloading v0.7.4
          Compiling aho-corasick v0.7.20
          Compiling regex-syntax v0.6.28
          Compiling bindgen v0.66.1
          Compiling regex v1.7.1
          Compiling cexpr v0.6.0
          Compiling lazy_static v1.4.0
          Compiling bitflags v2.4.0
          Compiling lazycell v1.3.0
          Compiling peeking_take_while v0.1.2
          Compiling shlex v1.1.0
          Compiling rustc-hash v1.1.0
          Compiling shell-words v1.1.0
          Compiling autocfg v1.1.0
          Compiling lock_api v0.4.9
          Compiling parking_lot_core v0.9.7
          Compiling rb-sys-build v0.9.81
          Compiling rb-sys v0.9.81
          Compiling smallvec v1.10.0
          Compiling scopeguard v1.1.0
          Compiling rb-sys-env v0.1.2
          Compiling anyhow v1.0.70
       error: failed to run custom build command for `rb-sys v0.9.81`

       Caused by:
       process didn't exit successfully:
       `/tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/build/rb-sys-f895743fce8c911c/build-script-main`
       (exit status: 101)
         --- stdout
         cargo:rerun-if-env-changed=RUBY
         cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
         cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
         cargo:rerun-if-env-changed=RBCONFIG_platform
         cargo:rerun-if-env-changed=RUBY_ROOT
         cargo:rerun-if-env-changed=RUBY_VERSION
         cargo:rerun-if-env-changed=RUBY
         cargo:rerun-if-changed=build/main.rs
         cargo:rerun-if-changed=build/stable_api_config.rs
         cargo:rerun-if-changed=build/version.rs
         cargo:rerun-if-changed=build/features.rs
         cargo:rerun-if-env-changed=RUBY_STATIC
         cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
         cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
         cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
         cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
         cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
         cargo:rerun-if-env-changed=TARGET
         cargo:rerun-if-env-changed=TARGET
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
         cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS

         --- stderr
       INFO: using bindgen with clang args:
       ["-I/tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/include/ruby-3.3.0+0",
       "-I/tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/include/ruby-3.3.0+0/x86_64-linux",
       "-fms-extensions", "-O3", "-fno-fast-math", "-g", "-Wall", "-Wextra",
       "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond",
       "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wpointer-arith",
       "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0",
       "-Wmissing-noreturn", "-Wno-cast-function-type",
       "-Wno-constant-logical-operand", "-Wno-long-long",
       "-Wno-missing-field-initializers", "-Wno-overlength-strings",
       "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign",
       "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value",
       "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn",
       "-Wunused-variable", "-Wmisleading-indentation", "-Wundef"]
       thread 'main' panicked at
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31:
       Unable to find libclang: "couldn't find any valid shared libraries matching:
       ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the
       `LIBCLANG_PATH` environment variable to a path where one of these files can be
       found (invalid: [])"
         note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
       warning: build failed, waiting for other jobs to finish...
       make: *** [Makefile:566: target/release/libtiktoken_ruby.so] Error 101

       make failed, exit code 2

       Gem files will remain installed in
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.6 for
       inspection.
       Results logged to
       /tmp/build_6a57c05e/vendor/bundle/ruby/3.3.0+0/extensions/x86_64-linux/3.3.0+0/tiktoken_ruby-0.0.6/gem_make.out

       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:118:in
       `run'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:51:in
       `block in make'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:43:in
       `each'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:43:in
       `make'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/ext_conf_builder.rb:42:in
       `build'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:186:in
       `build_extension'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:220:in
       `block in build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:217:in
       `each'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/ext/builder.rb:217:in
       `build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/rubygems/installer.rb:861:in
       `build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/rubygems_gem_installer.rb:76:in
       `build_extensions'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/rubygems_gem_installer.rb:28:in
       `install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/source/rubygems.rb:205:in
       `install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/gem_installer.rb:54:in
       `install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/gem_installer.rb:16:in
       `install_from_spec'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/parallel_installer.rb:129:in
       `do_install'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/installer/parallel_installer.rb:120:in
       `block in worker_pool'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:62:in
       `apply_func'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:57:in
       `block in process_queue'
         <internal:kernel>:187:in `loop'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:54:in
       `process_queue'
       /tmp/build_6a57c05e/vendor/ruby-3.3.0.rc1/lib/ruby/3.3.0+0/bundler/worker.rb:90:in
       `block (2 levels) in create_threads'

       An error occurred while installing tiktoken_ruby (0.0.6), and Bundler cannot
       continue.

       In Gemfile:
         tiktoken_ruby

 !
 !     Failed to install gems via Bundler.
 !
 !     Push rejected, failed to compile Ruby app.
 !     Push failed
bvalentino commented 6 months ago

+1 on Render. Seems like libslang is missing to be declared as a dependency.

tritonrc commented 6 months ago

I ran into this on another gem migrating to 3.3.0. The rb-sys-build crate needs to be at least 0.9.85 (https://github.com/oxidize-rb/rb-sys/pull/290). I put together a fork ( https://github.com/tritonrc/tiktoken_ruby) where I ran cargo update and that did the trick.

One thing to note: I needed to add libclang-dev to our build images as lack of libclang was causing an issue.

bmulholland commented 6 months ago

I guess that means https://github.com/IAPark/tiktoken_ruby/pull/21 fixes this?

prdanelli commented 5 months ago

Having the same issue with installing for Ruby 3.3.0 on bullseye - build is fine until I try to add the gem: 

root@055c189799d9:/home/api# bundle install tiktoken_ruby
ERROR: "bundle install" was called with arguments ["tiktoken_ruby"]
Usage: "bundle install [OPTIONS]"
root@055c189799d9:/home/api# gem install tiktoken_ruby
Fetching rb_sys-0.9.86.gem
Fetching tiktoken_ruby-0.0.6.gem
Successfully installed rb_sys-0.9.86
Building native extensions. This could take a while...
ERROR:  Error installing tiktoken_ruby:
        ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
/usr/local/bin/ruby extconf.rb
checking for gcc... yes
checking for g++... yes
checking for gcc-ar... yes
checking for cargo... no

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
make DESTDIR\= sitearchdir\=./.gem.20240118-10-3jdy4v sitelibdir\=./.gem.20240118-10-3jdy4v clean

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
make DESTDIR\= sitearchdir\=./.gem.20240118-10-3jdy4v sitelibdir\=./.gem.20240118-10-3jdy4v
info: downloading installer
info: profile set to 'minimal'
info: default host triple is aarch64-unknown-linux-gnu
info: skipping toolchain installation

Rust is installed now. Great!

To get started you need Cargo's bin directory
(/usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
in your PATH
environment variable. This has not been done automatically.

To configure your current shell, run:
source
"/usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-std'
info: installing component 'rustc'

  stable-aarch64-unknown-linux-gnu installed - rustc 1.75.0 (82e1608df 2023-12-21)

info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'
info: checking for self-update
info: using existing install for 'stable-aarch64-unknown-linux-gnu'
info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'

  stable-aarch64-unknown-linux-gnu unchanged - rustc 1.75.0 (82e1608df 2023-12-21)

info: note that the toolchain 'stable-aarch64-unknown-linux-gnu' is currently in use (environment override by RUSTUP_TOOLCHAIN)
generating target/release/libtiktoken_ruby.so (release)
/usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release -- -C linker=gcc -L native=/usr/local/lib -C link-arg=-lm -l pthread
    Updating crates.io index
    Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
    Updating git submodule `https://github.com/zurawiki/tiktoken`
 Downloading crates ...
  Downloaded shlex v1.1.0
  Downloaded cfg-if v1.0.0
  Downloaded aho-corasick v0.7.20
  Downloaded bit-set v0.5.3
  Downloaded peeking_take_while v0.1.2
  Downloaded scopeguard v1.1.0
  Downloaded rb-sys-env v0.1.2
  Downloaded lazycell v1.3.0
  Downloaded rb-sys v0.9.81
  Downloaded rustc-hash v1.1.0
  Downloaded rb-sys-build v0.9.81
  Downloaded parking_lot_core v0.9.7
  Downloaded lock_api v0.4.9
  Downloaded clang-sys v1.6.0
  Downloaded parking_lot v0.12.1
  Downloaded libloading v0.7.4
  Downloaded memchr v2.5.0
  Downloaded fancy-regex v0.11.0
  Downloaded minimal-lexical v0.2.1
  Downloaded regex-automata v0.1.10
  Downloaded magnus v0.6.1
  Downloaded syn v2.0.31
  Downloaded bstr v1.4.0
  Downloaded bindgen v0.66.1
  Downloaded regex-syntax v0.6.28
  Downloaded libc v0.2.140
  Downloaded regex v1.7.1
  Downloaded nom v7.1.3
  Downloaded base64 v0.21.0
  Downloaded serde v1.0.157
  Downloaded unicode-ident v1.0.8
  Downloaded once_cell v1.17.1
  Downloaded smallvec v1.10.0
  Downloaded anyhow v1.0.70
  Downloaded cexpr v0.6.0
  Downloaded shell-words v1.1.0
  Downloaded seq-macro v0.3.5
  Downloaded quote v1.0.33
  Downloaded bitflags v2.4.0
  Downloaded glob v0.3.1
  Downloaded lazy_static v1.4.0
  Downloaded bit-vec v0.6.3
  Downloaded autocfg v1.1.0
  Downloaded magnus-macros v0.6.0
  Downloaded proc-macro2 v1.0.66
   Compiling memchr v2.5.0
   Compiling proc-macro2 v1.0.66
   Compiling glob v0.3.1
   Compiling unicode-ident v1.0.8
   Compiling libc v0.2.140
   Compiling cfg-if v1.0.0
   Compiling minimal-lexical v0.2.1
   Compiling regex-syntax v0.6.28
   Compiling libloading v0.7.4
   Compiling bindgen v0.66.1
   Compiling shlex v1.1.0
   Compiling rustc-hash v1.1.0
   Compiling peeking_take_while v0.1.2
   Compiling lazy_static v1.4.0
   Compiling bitflags v2.4.0
   Compiling lazycell v1.3.0
   Compiling shell-words v1.1.0
   Compiling clang-sys v1.6.0
   Compiling autocfg v1.1.0
   Compiling aho-corasick v0.7.20
   Compiling nom v7.1.3
   Compiling quote v1.0.33
   Compiling lock_api v0.4.9
   Compiling syn v2.0.31
   Compiling parking_lot_core v0.9.7
   Compiling anyhow v1.0.70
   Compiling scopeguard v1.1.0
   Compiling rb-sys-env v0.1.2
   Compiling bit-vec v0.6.3
   Compiling smallvec v1.10.0
   Compiling regex v1.7.1
   Compiling bit-set v0.5.3
   Compiling cexpr v0.6.0
   Compiling magnus v0.6.1
   Compiling once_cell v1.17.1
   Compiling regex-automata v0.1.10
   Compiling parking_lot v0.12.1
   Compiling bstr v1.4.0
   Compiling base64 v0.21.0
   Compiling fancy-regex v0.11.0
   Compiling seq-macro v0.3.5
   Compiling tiktoken-rs v0.3.2 (https://github.com/IAPark/tiktoken-rs.git#5231fbf4)
   Compiling magnus-macros v0.6.0
   Compiling rb-sys-build v0.9.81
   Compiling rb-sys v0.9.81
error: failed to run custom build command for `rb-sys v0.9.81`

Caused by:
  process didn't exit successfully: `/usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/build/rb-sys-6ba708d6c34f9ae4/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
  cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
  cargo:rerun-if-env-changed=RBCONFIG_platform
  cargo:rerun-if-env-changed=RUBY_ROOT
  cargo:rerun-if-env-changed=RUBY_VERSION
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-changed=build/stable_api_config.rs
  cargo:rerun-if-changed=build/main.rs
  cargo:rerun-if-changed=build/features.rs
  cargo:rerun-if-changed=build/version.rs
  cargo:rerun-if-env-changed=RUBY_STATIC
  cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
  cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
  cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS

  --- stderr
  INFO: using bindgen with clang args: ["-I/usr/local/include/ruby-3.3.0", "-I/usr/local/include/ruby-3.3.0/aarch64-linux", "-fms-extensions", "-O3", "-fno-fast-math", "-ggdb3", "-Wall", "-Wextra", "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond", "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wpointer-arith", "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0", "-Wmissing-noreturn", "-Wno-cast-function-type", "-Wno-constant-logical-operand", "-Wno-long-long", "-Wno-missing-field-initializers", "-Wno-overlength-strings", "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign", "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value", "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn", "-Wunused-variable", "-Wmisleading-indentation", "-Wundef"]
  thread 'main' panicked at /usr/local/bundle/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31:
  Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
make: *** [Makefile:566: target/release/libtiktoken_ruby.so] Error 101

make failed, exit code 2

Gem files will remain installed in /usr/local/bundle/gems/tiktoken_ruby-0.0.6 for inspection.
Results logged to /usr/local/bundle/extensions/aarch64-linux/3.3.0/tiktoken_ruby-0.0.6/gem_make.out

A new release of RubyGems is available: 3.5.3 → 3.5.5!
Run `gem update --system 3.5.5` to update your installation.
d4rky-pl commented 5 months ago

From what I've noticed, the gem is not trying to build on Ruby 3.2 which tells me there's already a valid binary version in the RubyGems for that. Would be great if we could have a prebuilt one for 3.3 in there as well as this is also a blocker for us

andreaslillebo commented 5 months ago

Looking at the code, it looks like pushing a new version will trigger the cross-compile-action, which will compile the gem for these platforms and these Ruby-versions, as can be seen in this build for the current version.

Ruby 3.3 was not released when the current version of this gem was released, hence there is no precompiled version for Ruby 3.3.

Tldr; Releasing a new version should solve this issue.

d4rky-pl commented 4 months ago

Unfortunately version 0.0.7 of the gem does not include binaries in rubygems as the build process has failed - https://github.com/IAPark/tiktoken_ruby/actions/runs/7868320647

FYI @IAPark

ScotterC commented 4 months ago

I took a shot at fixing that the other day in my fork. 

Error response from daemon: manifest for rbsys/x86_64-linux-musl:0.9.87 not found: manifest unknown: manifest unknown

I couldn't pull the rbsys container locally or the latest

docker pull rbsys/x86_64-linux-musl:0.9.87
manifest for rbsys/x86_64-linux-musl:0.9.87 not found: manifest unknown: manifest unknown

docker pull rbsys/x86_64-linux:latest
latest: Pulling from rbsys/x86_64-linux
no matching manifest for linux/arm64/v8 in the manifest list entries

But the tags seem to be available on dockerhub.

Ah, this issue seems to be a resolution? https://github.com/oxidize-rb/rb-sys/issues/324.

@IAPark I got my fork to cross-compile successfully now that the docker containers are in place. Should work here just by re-running the action.

ScotterC commented 4 months ago

Latest release (0.0.7) now has the compiled binaries and worked on my heroku deploy. Please re-open if not resolved for you.

prdanelli commented 4 months ago

@ScotterC I'm seeing the following installing the Gem for Ruby 3.2.3, inside a Debian Bulleye Docker container, running on an Arm Mac. 

Installing tiktoken_ruby 0.0.7 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby
/usr/local/bin/ruby extconf.rb
checking for gcc... yes
checking for g++... yes
checking for gcc-ar... yes
checking for cargo... no

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby
make DESTDIR\= sitearchdir\=./.gem.20240215-1-simyyu sitelibdir\=./.gem.20240215-1-simyyu clean

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby
make DESTDIR\= sitearchdir\=./.gem.20240215-1-simyyu sitelibdir\=./.gem.20240215-1-simyyu
info: downloading installer
Cannot execute /tmp/tmp.skww56pn8N/rustup-init (likely because of mounting /tmp as noexec).
Please copy the file to a location where you can execute binaries and run ./rustup-init.
make: *** [Makefile:550: /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo] Error 1

make failed, exit code 2

Gem files will remain installed in /usr/local/bundle/gems/tiktoken_ruby-0.0.7 for inspection.
Results logged to /usr/local/bundle/extensions/aarch64-linux/3.2.0/tiktoken_ruby-0.0.7/gem_make.out

  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:125:in `run'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:51:in `block in make'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:43:in `each'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:43:in `make'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:42:in `build'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:193:in `build_extension'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:227:in `block in build_extensions'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:224:in `each'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:224:in `build_extensions'
  /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:852:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:76:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:28:in `install'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/source/rubygems.rb:203:in `install'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:54:in `install'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:130:in `do_install'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:121:in `block in worker_pool'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:62:in `apply_func'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:57:in `block in process_queue'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:54:in `loop'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:54:in `process_queue'
  /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing tiktoken_ruby (0.0.7), and Bundler cannot continue.

Additionally, when trying to build a Ruby 3.2.2 container and install this Gem, i see the following: 

### ... 
99.73 Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
99.73
99.73 current directory:
99.73 /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby
99.73 /usr/local/bin/ruby extconf.rb
99.73 checking for gcc... yes
99.73 checking for g++... yes
99.73 checking for gcc-ar... yes
99.73 checking for cargo... no
99.73
99.73 current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby
99.73 make DESTDIR\= sitearchdir\=./.gem.20240215-28-riomdj
99.73 sitelibdir\=./.gem.20240215-28-riomdj clean
99.73
99.73 current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby
99.73 make DESTDIR\= sitearchdir\=./.gem.20240215-28-riomdj
99.73 sitelibdir\=./.gem.20240215-28-riomdj
99.73 info: downloading installer
99.73 info: profile set to 'minimal'
99.73 info: default host triple is aarch64-unknown-linux-gnu
99.73 info: skipping toolchain installation
99.73
99.73
99.73 Rust is installed now. Great!
99.73
99.73 To get started you need Cargo's bin directory
99.73 (/usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
99.73 in your PATH
99.73 environment variable. This has not been done automatically.
99.73
99.73 To configure your current shell, run:
99.73 source
99.73 "/usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
99.73 info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
99.73 info: latest update on 2024-02-08, rust version 1.76.0 (07dca489a 2024-02-04)
99.73 info: downloading component 'cargo'
99.73 info: downloading component 'rust-std'
99.73 info: downloading component 'rustc'
99.73 info: installing component 'cargo'
99.73 info: installing component 'rust-std'
99.73 info: installing component 'rustc'
99.73
99.73 stable-aarch64-unknown-linux-gnu installed - rustc 1.76.0 (07dca489a
99.73 2024-02-04)
99.73
99.73 info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'
99.73 info: checking for self-update
99.73 info: using existing install for 'stable-aarch64-unknown-linux-gnu'
99.73 info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'
99.73
99.73 stable-aarch64-unknown-linux-gnu unchanged - rustc 1.76.0 (07dca489a
99.73 2024-02-04)
99.73
99.73 info: note that the toolchain 'stable-aarch64-unknown-linux-gnu' is currently in
99.73 use (environment override by RUSTUP_TOOLCHAIN)
99.73 generating target/release/libtiktoken_ruby.so (release)
99.73 /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo
99.73 rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release
99.73 -- -C linker=gcc -L native=/usr/local/lib -C link-arg=-lm -l pthread
99.73     Updating crates.io index
99.73     Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
99.73     Updating git submodule `https://github.com/zurawiki/tiktoken`
# ... 
99.73 Compiling tiktoken-rs v0.3.2
99.73 (https://github.com/IAPark/tiktoken-rs.git#5231fbf4)
99.73    Compiling magnus-macros v0.6.0
99.73    Compiling rb-sys-build v0.9.81
99.73    Compiling rb-sys v0.9.81
99.73 error: failed to run custom build command for `rb-sys v0.9.81`
99.73
99.73 Caused by:
99.73 process didn't exit successfully:
99.73 `/usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby/target/release/build/rb-sys-cf5d654371f43146/build-script-main`
99.73 (exit status: 101)
99.73   --- stdout
99.73   cargo:rerun-if-env-changed=RUBY
99.73   cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
99.73   cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
99.73   cargo:rerun-if-env-changed=RBCONFIG_platform
99.73   cargo:rerun-if-env-changed=RUBY_ROOT
99.73   cargo:rerun-if-env-changed=RUBY_VERSION
99.73   cargo:rerun-if-env-changed=RUBY
99.73   cargo:rerun-if-changed=build/stable_api_config.rs
99.73   cargo:rerun-if-changed=build/main.rs
99.73   cargo:rerun-if-changed=build/features.rs
99.73   cargo:rerun-if-changed=build/version.rs
99.73   cargo:rerun-if-env-changed=RUBY_STATIC
99.73   cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
99.73   cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
99.73   cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
99.73   cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
99.73   cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
99.73   cargo:rerun-if-env-changed=TARGET
99.73   cargo:rerun-if-env-changed=TARGET
99.73   cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
99.73   cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
99.73   cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
99.73   cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
99.73   cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
99.73   cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
99.73
99.73   --- stderr
99.73 INFO: using bindgen with clang args: ["-I/usr/local/include/ruby-3.2.0",
99.73 "-I/usr/local/include/ruby-3.2.0/aarch64-linux", "-fms-extensions", "-O3",
99.73 "-fno-fast-math", "-ggdb3", "-Wall", "-Wextra", "-Wdeprecated-declarations",
99.73 "-Wdiv-by-zero", "-Wduplicated-cond", "-Wimplicit-function-declaration",
99.73 "-Wimplicit-int", "-Wmisleading-indentation", "-Wpointer-arith",
99.73 "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0",
99.73 "-Wmissing-noreturn", "-Wno-cast-function-type",
99.73 "-Wno-constant-logical-operand", "-Wno-long-long",
99.73 "-Wno-missing-field-initializers", "-Wno-overlength-strings",
99.73 "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign",
99.73 "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value",
99.73 "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn",
99.73 "-Wunused-variable", "-Wundef"]
99.73 thread 'main' panicked at
99.73 /usr/local/bundle/gems/tiktoken_ruby-0.0.7/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31:
99.73 Unable to find libclang: "couldn't find any valid shared libraries matching:
99.73 ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the
99.73 `LIBCLANG_PATH` environment variable to a path where one of these files can be
99.73 found (invalid: [])"
99.73   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
99.73 make: *** [Makefile:565: target/release/libtiktoken_ruby.so] Error 101
99.73
99.73 make failed, exit code 2
99.73
99.73 Gem files will remain installed in /usr/local/bundle/gems/tiktoken_ruby-0.0.7
99.73 for inspection.
99.73 Results logged to
99.73 /usr/local/bundle/extensions/aarch64-linux/3.2.0/tiktoken_ruby-0.0.7/gem_make.out
99.73
99.73   /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:125:in `run'
99.73 /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:51:in `block in
99.73 make'
99.73   /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:43:in `each'
99.73   /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:43:in `make'
99.73 /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:42:in
99.73 `build'
99.73 /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:193:in
99.73 `build_extension'
99.73 /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:227:in `block in
99.73 build_extensions'
99.73   /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:224:in `each'
99.73 /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:224:in
99.73 `build_extensions'
99.73 /usr/local/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:852:in
99.73 `build_extensions'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:76:in
99.73 `build_extensions'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/rubygems_gem_installer.rb:28:in
99.73 `install'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/source/rubygems.rb:203:in
99.73 `install'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:54:in
99.73 `install'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/gem_installer.rb:16:in
99.73 `install_from_spec'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:130:in
99.73 `do_install'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/installer/parallel_installer.rb:121:in
99.73 `block in worker_pool'
99.73   /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:62:in `apply_func'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:57:in `block in
99.73 process_queue'
99.73   /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:54:in `loop'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:54:in
99.73 `process_queue'
99.73 /usr/local/bundle/gems/bundler-2.4.22/lib/bundler/worker.rb:90:in `block (2
99.73 levels) in create_threads'
99.73
99.73 An error occurred while installing tiktoken_ruby (0.0.7), and Bundler cannot
99.73 continue.
99.73
99.73 In Gemfile:
99.73   langchainrb was resolved to 0.9.2, which depends on
99.73     tiktoken_ruby

Are there any new steps that need to be taken before this uses the compiled binaries?

ScotterC commented 4 months ago

@prdanelli hmm AFAIK there's nothing else that should be needed to use the compiled binaries so I'm confused by the errors you're getting. The only thing that stands out to me is rb-sys v0.9.81 in the latter stacktrace where the version should be >0.9.86

bmulholland commented 4 months ago

On my end I confirm that 0.0.7 deployed on Heroku without clang, decreasing our slug size by ~110MB. Thank you!!

eiriklied commented 4 months ago

On my end I confirm that 0.0.7 deployed on Heroku without clang, decreasing our slug size by ~110MB. Thank you!!

@bmulholland , I'm struggling to deploy 0.0.7 on Heroku without clang. Both on Ruby 3.3 and 3.2.2. Not sure there are any particular steps I've missed?

bmulholland commented 4 months ago

It was always installing fine on 3.2.2 so you’ve got a different issue than this one. Check your build logs and work through it from there.