Update rbsys to support Ruby 3.3

[djrodgerspryor]

The tests pass and this seems to work perfectly as far as I can tell.

[bmulholland]

Fixes https://github.com/IAPark/tiktoken_ruby/issues/20

[karlentwistle]

@bmulholland Maybe I've done something wrong but I'm not sure this fixes https://github.com/IAPark/tiktoken_ruby/issues/20

• I created a new Rails app here https://github.com/karlentwistle/tiktoken_heroku_example using Ruby 3.3.0
• Applied this commit https://github.com/karlentwistle/tiktoken_heroku_example/commit/5888659e95baa9ba3bb99063346e68ec861aa781
• Attempted to push to Heroku and got the following error

Full deploy output including error

``` Enumerating objects: 7, done. Counting objects: 100% (7/7), done. Delta compression using up to 8 threads Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 603 bytes | 603.00 KiB/s, done. Total 4 (delta 3), reused 0 (delta 0), pack-reused 0 remote: Updated 82 paths from f5b0d5a remote: Compressing source files... done. remote: Building source: remote: remote: -----> Building on the Heroku-22 stack remote: -----> Using buildpack: heroku/ruby remote: -----> Ruby app detected remote: -----> Installing bundler 2.3.25 remote: -----> Removing BUNDLED WITH version in the Gemfile.lock remote: -----> Compiling Ruby/Rails remote: -----> Using Ruby version: ruby-3.3.0 remote: -----> Installing dependencies using bundler 2.3.25 remote: Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4 remote: Fetching https://github.com/StileEducation/tiktoken_ruby.git remote: Fetching https://github.com/StileEducation/tiktoken_ruby.git remote: Fetching gem metadata from https://rubygems.org/......... remote: Fetching rb_sys 0.9.86 remote: Installing rb_sys 0.9.86 remote: Gem::Ext::BuildError: ERROR: Failed to build gem native extension. remote: remote: current directory: remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/bin/ruby extconf.rb remote: checking for gcc... yes remote: checking for g++... yes remote: checking for gcc-ar... yes remote: checking for cargo... no remote: remote: current directory: remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby remote: make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m remote: sitelibdir\=./.gem.20240108-198-77bm6m clean remote: remote: current directory: remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby remote: make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m remote: sitelibdir\=./.gem.20240108-198-77bm6m remote: info: downloading installer remote: info: profile set to 'minimal' remote: info: default host triple is x86_64-unknown-linux-gnu remote: info: skipping toolchain installation remote: remote: remote: Rust is installed now. Great! remote: remote: To get started you need Cargo's bin directory remote: (/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin) remote: in your PATH remote: environment variable. This has not been done automatically. remote: remote: To configure your current shell, run: remote: source remote: "/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/env" remote: info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu' remote: info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21) remote: info: downloading component 'cargo' remote: info: downloading component 'rust-std' remote: info: downloading component 'rustc' remote: info: installing component 'cargo' remote: info: installing component 'rust-std' remote: info: installing component 'rustc' remote: remote: stable-x86_64-unknown-linux-gnu installed - rustc 1.75.0 (82e1608df remote: 2023-12-21) remote: remote: info: default toolchain set to 'stable-x86_64-unknown-linux-gnu' remote: info: checking for self-update remote: info: using existing install for 'stable-x86_64-unknown-linux-gnu' remote: info: default toolchain set to 'stable-x86_64-unknown-linux-gnu' remote: remote: stable-x86_64-unknown-linux-gnu unchanged - rustc 1.75.0 (82e1608df remote: 2023-12-21) remote: remote: info: note that the toolchain 'stable-x86_64-unknown-linux-gnu' is currently in remote: use (environment override by RUSTUP_TOOLCHAIN) remote: generating target/release/libtiktoken_ruby.so (release) remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo remote: rustc --manifest-path ./Cargo.toml --target-dir target --lib --profile release remote: -- -C linker=gcc -L native=/tmp/build_91a6107a/vendor/ruby-3.3.0/lib -C remote: link-arg=-lm -l pthread remote: Updating crates.io index remote: Updating git repository `https://github.com/IAPark/tiktoken-rs.git` remote: Updating git submodule `https://github.com/zurawiki/tiktoken` remote: Downloading crates ... remote: Downloaded base64 v0.21.0 remote: Downloaded once_cell v1.17.1 remote: Downloaded bstr v1.4.0 remote: Downloaded smallvec v1.10.0 remote: Downloaded bit-vec v0.6.3 remote: Downloaded cexpr v0.6.0 remote: Downloaded lock_api v0.4.9 remote: Downloaded syn v2.0.31 remote: Downloaded shlex v1.1.0 remote: Downloaded serde v1.0.157 remote: Downloaded seq-macro v0.3.5 remote: Downloaded proc-macro2 v1.0.66 remote: Downloaded unicode-ident v1.0.8 remote: Downloaded shell-words v1.1.0 remote: Downloaded scopeguard v1.1.0 remote: Downloaded regex-syntax v0.6.28 remote: Downloaded rb-sys v0.9.81 remote: Downloaded regex v1.7.1 remote: Downloaded nom v7.1.3 remote: Downloaded libc v0.2.140 remote: Downloaded bit-set v0.5.3 remote: Downloaded bindgen v0.66.1 remote: Downloaded rb-sys-env v0.1.2 remote: Downloaded rb-sys-build v0.9.81 remote: Downloaded quote v1.0.33 remote: Downloaded peeking_take_while v0.1.2 remote: Downloaded parking_lot_core v0.9.7 remote: Downloaded parking_lot v0.12.1 remote: Downloaded minimal-lexical v0.2.1 remote: Downloaded memchr v2.5.0 remote: Downloaded magnus-macros v0.6.0 remote: Downloaded magnus v0.6.1 remote: Downloaded libloading v0.7.4 remote: Downloaded lazycell v1.3.0 remote: Downloaded lazy_static v1.4.0 remote: Downloaded glob v0.3.1 remote: Downloaded clang-sys v1.6.0 remote: Downloaded bitflags v2.4.0 remote: Downloaded autocfg v1.1.0 remote: Downloaded anyhow v1.0.70 remote: Downloaded aho-corasick v0.7.20 remote: Downloaded fancy-regex v0.11.0 remote: Downloaded rustc-hash v1.1.0 remote: Downloaded regex-automata v0.1.10 remote: Downloaded cfg-if v1.0.0 remote: Compiling memchr v2.5.0 remote: Compiling proc-macro2 v1.0.66 remote: Compiling glob v0.3.1 remote: Compiling unicode-ident v1.0.8 remote: Compiling clang-sys v1.6.0 remote: Compiling libc v0.2.140 remote: Compiling quote v1.0.33 remote: Compiling cfg-if v1.0.0 remote: Compiling minimal-lexical v0.2.1 remote: Compiling syn v2.0.31 remote: Compiling nom v7.1.3 remote: Compiling libloading v0.7.4 remote: Compiling aho-corasick v0.7.20 remote: Compiling regex-syntax v0.6.28 remote: Compiling bindgen v0.66.1 remote: Compiling regex v1.7.1 remote: Compiling cexpr v0.6.0 remote: Compiling bitflags v2.4.0 remote: Compiling lazy_static v1.4.0 remote: Compiling rustc-hash v1.1.0 remote: Compiling shlex v1.1.0 remote: Compiling peeking_take_while v0.1.2 remote: Compiling lazycell v1.3.0 remote: Compiling shell-words v1.1.0 remote: Compiling autocfg v1.1.0 remote: Compiling lock_api v0.4.9 remote: Compiling rb-sys-build v0.9.81 remote: Compiling parking_lot_core v0.9.7 remote: Compiling rb-sys v0.9.81 remote: Compiling smallvec v1.10.0 remote: error: failed to run custom build command for `rb-sys v0.9.81` remote: remote: Caused by: remote: process didn't exit successfully: remote: `/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/target/release/build/rb-sys-0b34e8f38d6837b7/build-script-main` remote: (exit status: 101) remote: --- stdout remote: cargo:rerun-if-env-changed=RUBY remote: cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING remote: cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION remote: cargo:rerun-if-env-changed=RBCONFIG_platform remote: cargo:rerun-if-env-changed=RUBY_ROOT remote: cargo:rerun-if-env-changed=RUBY_VERSION remote: cargo:rerun-if-env-changed=RUBY remote: cargo:rerun-if-changed=build/features.rs remote: cargo:rerun-if-changed=build/version.rs remote: cargo:rerun-if-changed=build/stable_api_config.rs remote: cargo:rerun-if-changed=build/main.rs remote: cargo:rerun-if-env-changed=RUBY_STATIC remote: cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED remote: cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir remote: cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir remote: cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS remote: cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir remote: cargo:rerun-if-env-changed=TARGET remote: cargo:rerun-if-env-changed=TARGET remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS remote: remote: --- stderr remote: INFO: using bindgen with clang args: remote: ["-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0", remote: "-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0/x86_64-linux", remote: "-fms-extensions", "-O3", "-fno-fast-math", "-g", "-Wall", "-Wextra", remote: "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond", remote: "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wpointer-arith", remote: "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0", remote: "-Wmissing-noreturn", "-Wno-cast-function-type", remote: "-Wno-constant-logical-operand", "-Wno-long-long", remote: "-Wno-missing-field-initializers", "-Wno-overlength-strings", remote: "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign", remote: "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value", remote: "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn", remote: "-Wunused-variable", "-Wmisleading-indentation", "-Wundef"] remote: thread 'main' panicked at remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31: remote: Unable to find libclang: "couldn't find any valid shared libraries matching: remote: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the remote: `LIBCLANG_PATH` environment variable to a path where one of these files can be remote: found (invalid: [])" remote: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace remote: warning: build failed, waiting for other jobs to finish... remote: make: *** [Makefile:566: target/release/libtiktoken_ruby.so] Error 101 remote: remote: make failed, exit code 2 remote: remote: Gem files will remain installed in remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c remote: for inspection. remote: Results logged to remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/extensions/x86_64-linux/3.3.0/tiktoken_ruby-d7bdeb57336c/gem_make.out remote: remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:125:in remote: `run' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:51:in remote: `block in make' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in remote: `each' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in remote: `make' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/ext_conf_builder.rb:42:in remote: `build' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:193:in remote: `build_extension' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:227:in remote: `block in build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in remote: `each' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in remote: `build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/installer.rb:852:in remote: `build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:76:in remote: `build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path/installer.rb:28:in remote: `post_install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path.rb:245:in remote: `generate_bin' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/git.rb:202:in remote: `install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:54:in remote: `install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:16:in remote: `install_from_spec' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:132:in remote: `do_install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:123:in remote: `block in worker_pool' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:62:in remote: `apply_func' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:57:in remote: `block in process_queue' remote: :187:in `loop' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:54:in remote: `process_queue' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:90:in remote: `block (2 levels) in create_threads' remote: remote: An error occurred while installing tiktoken_ruby (0.0.6), and Bundler cannot remote: continue. remote: remote: In Gemfile: remote: tiktoken_ruby remote: Bundler Output: Fetching https://github.com/StileEducation/tiktoken_ruby.git remote: Fetching https://github.com/StileEducation/tiktoken_ruby.git remote: Fetching gem metadata from https://rubygems.org/......... remote: Fetching rb_sys 0.9.86 remote: Installing rb_sys 0.9.86 remote: Gem::Ext::BuildError: ERROR: Failed to build gem native extension. remote: remote: current directory: remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/bin/ruby extconf.rb remote: checking for gcc... yes remote: checking for g++... yes remote: checking for gcc-ar... yes remote: checking for cargo... no remote: remote: current directory: remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby remote: make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m remote: sitelibdir\=./.gem.20240108-198-77bm6m clean remote: remote: current directory: remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby remote: make DESTDIR\= sitearchdir\=./.gem.20240108-198-77bm6m remote: sitelibdir\=./.gem.20240108-198-77bm6m remote: info: downloading installer remote: info: profile set to 'minimal' remote: info: default host triple is x86_64-unknown-linux-gnu remote: info: skipping toolchain installation remote: remote: remote: Rust is installed now. Great! remote: remote: To get started you need Cargo's bin directory remote: (/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin) remote: in your PATH remote: environment variable. This has not been done automatically. remote: remote: To configure your current shell, run: remote: source remote: "/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/env" remote: info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu' remote: info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21) remote: info: downloading component 'cargo' remote: info: downloading component 'rust-std' remote: info: downloading component 'rustc' remote: info: installing component 'cargo' remote: info: installing component 'rust-std' remote: info: installing component 'rustc' remote: remote: stable-x86_64-unknown-linux-gnu installed - rustc 1.75.0 (82e1608df remote: 2023-12-21) remote: remote: info: default toolchain set to 'stable-x86_64-unknown-linux-gnu' remote: info: checking for self-update remote: info: using existing install for 'stable-x86_64-unknown-linux-gnu' remote: info: default toolchain set to 'stable-x86_64-unknown-linux-gnu' remote: remote: stable-x86_64-unknown-linux-gnu unchanged - rustc 1.75.0 (82e1608df remote: 2023-12-21) remote: remote: info: note that the toolchain 'stable-x86_64-unknown-linux-gnu' is currently in remote: use (environment override by RUSTUP_TOOLCHAIN) remote: generating target/release/libtiktoken_ruby.so (release) remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo remote: rustc --manifest-path ./Cargo.toml --target-dir target --lib --profile release remote: -- -C linker=gcc -L native=/tmp/build_91a6107a/vendor/ruby-3.3.0/lib -C remote: link-arg=-lm -l pthread remote: Updating crates.io index remote: Updating git repository `https://github.com/IAPark/tiktoken-rs.git` remote: Updating git submodule `https://github.com/zurawiki/tiktoken` remote: Downloading crates ... remote: Downloaded base64 v0.21.0 remote: Downloaded once_cell v1.17.1 remote: Downloaded bstr v1.4.0 remote: Downloaded smallvec v1.10.0 remote: Downloaded bit-vec v0.6.3 remote: Downloaded cexpr v0.6.0 remote: Downloaded lock_api v0.4.9 remote: Downloaded syn v2.0.31 remote: Downloaded shlex v1.1.0 remote: Downloaded serde v1.0.157 remote: Downloaded seq-macro v0.3.5 remote: Downloaded proc-macro2 v1.0.66 remote: Downloaded unicode-ident v1.0.8 remote: Downloaded shell-words v1.1.0 remote: Downloaded scopeguard v1.1.0 remote: Downloaded regex-syntax v0.6.28 remote: Downloaded rb-sys v0.9.81 remote: Downloaded regex v1.7.1 remote: Downloaded nom v7.1.3 remote: Downloaded libc v0.2.140 remote: Downloaded bit-set v0.5.3 remote: Downloaded bindgen v0.66.1 remote: Downloaded rb-sys-env v0.1.2 remote: Downloaded rb-sys-build v0.9.81 remote: Downloaded quote v1.0.33 remote: Downloaded peeking_take_while v0.1.2 remote: Downloaded parking_lot_core v0.9.7 remote: Downloaded parking_lot v0.12.1 remote: Downloaded minimal-lexical v0.2.1 remote: Downloaded memchr v2.5.0 remote: Downloaded magnus-macros v0.6.0 remote: Downloaded magnus v0.6.1 remote: Downloaded libloading v0.7.4 remote: Downloaded lazycell v1.3.0 remote: Downloaded lazy_static v1.4.0 remote: Downloaded glob v0.3.1 remote: Downloaded clang-sys v1.6.0 remote: Downloaded bitflags v2.4.0 remote: Downloaded autocfg v1.1.0 remote: Downloaded anyhow v1.0.70 remote: Downloaded aho-corasick v0.7.20 remote: Downloaded fancy-regex v0.11.0 remote: Downloaded rustc-hash v1.1.0 remote: Downloaded regex-automata v0.1.10 remote: Downloaded cfg-if v1.0.0 remote: Compiling memchr v2.5.0 remote: Compiling proc-macro2 v1.0.66 remote: Compiling glob v0.3.1 remote: Compiling unicode-ident v1.0.8 remote: Compiling clang-sys v1.6.0 remote: Compiling libc v0.2.140 remote: Compiling quote v1.0.33 remote: Compiling cfg-if v1.0.0 remote: Compiling minimal-lexical v0.2.1 remote: Compiling syn v2.0.31 remote: Compiling nom v7.1.3 remote: Compiling libloading v0.7.4 remote: Compiling aho-corasick v0.7.20 remote: Compiling regex-syntax v0.6.28 remote: Compiling bindgen v0.66.1 remote: Compiling regex v1.7.1 remote: Compiling cexpr v0.6.0 remote: Compiling bitflags v2.4.0 remote: Compiling lazy_static v1.4.0 remote: Compiling rustc-hash v1.1.0 remote: Compiling shlex v1.1.0 remote: Compiling peeking_take_while v0.1.2 remote: Compiling lazycell v1.3.0 remote: Compiling shell-words v1.1.0 remote: Compiling autocfg v1.1.0 remote: Compiling lock_api v0.4.9 remote: Compiling rb-sys-build v0.9.81 remote: Compiling parking_lot_core v0.9.7 remote: Compiling rb-sys v0.9.81 remote: Compiling smallvec v1.10.0 remote: error: failed to run custom build command for `rb-sys v0.9.81` remote: remote: Caused by: remote: process didn't exit successfully: remote: `/tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/target/release/build/rb-sys-0b34e8f38d6837b7/build-script-main` remote: (exit status: 101) remote: --- stdout remote: cargo:rerun-if-env-changed=RUBY remote: cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING remote: cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION remote: cargo:rerun-if-env-changed=RBCONFIG_platform remote: cargo:rerun-if-env-changed=RUBY_ROOT remote: cargo:rerun-if-env-changed=RUBY_VERSION remote: cargo:rerun-if-env-changed=RUBY remote: cargo:rerun-if-changed=build/features.rs remote: cargo:rerun-if-changed=build/version.rs remote: cargo:rerun-if-changed=build/stable_api_config.rs remote: cargo:rerun-if-changed=build/main.rs remote: cargo:rerun-if-env-changed=RUBY_STATIC remote: cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED remote: cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir remote: cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir remote: cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS remote: cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir remote: cargo:rerun-if-env-changed=TARGET remote: cargo:rerun-if-env-changed=TARGET remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-unknown-linux-gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_unknown_linux_gnu remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS remote: cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS remote: remote: --- stderr remote: INFO: using bindgen with clang args: remote: ["-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0", remote: "-I/tmp/build_91a6107a/vendor/ruby-3.3.0/include/ruby-3.3.0/x86_64-linux", remote: "-fms-extensions", "-O3", "-fno-fast-math", "-g", "-Wall", "-Wextra", remote: "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond", remote: "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wpointer-arith", remote: "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0", remote: "-Wmissing-noreturn", "-Wno-cast-function-type", remote: "-Wno-constant-logical-operand", "-Wno-long-long", remote: "-Wno-missing-field-initializers", "-Wno-overlength-strings", remote: "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign", remote: "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value", remote: "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn", remote: "-Wunused-variable", "-Wmisleading-indentation", "-Wundef"] remote: thread 'main' panicked at remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.66.1/lib.rs:604:31: remote: Unable to find libclang: "couldn't find any valid shared libraries matching: remote: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the remote: `LIBCLANG_PATH` environment variable to a path where one of these files can be remote: found (invalid: [])" remote: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace remote: warning: build failed, waiting for other jobs to finish... remote: make: *** [Makefile:566: target/release/libtiktoken_ruby.so] Error 101 remote: remote: make failed, exit code 2 remote: remote: Gem files will remain installed in remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/tiktoken_ruby-d7bdeb57336c remote: for inspection. remote: Results logged to remote: /tmp/build_91a6107a/vendor/bundle/ruby/3.3.0/bundler/gems/extensions/x86_64-linux/3.3.0/tiktoken_ruby-d7bdeb57336c/gem_make.out remote: remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:125:in remote: `run' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:51:in remote: `block in make' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in remote: `each' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in remote: `make' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/ext_conf_builder.rb:42:in remote: `build' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:193:in remote: `build_extension' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:227:in remote: `block in build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in remote: `each' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in remote: `build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/rubygems/installer.rb:852:in remote: `build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:76:in remote: `build_extensions' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path/installer.rb:28:in remote: `post_install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/path.rb:245:in remote: `generate_bin' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/source/git.rb:202:in remote: `install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:54:in remote: `install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:16:in remote: `install_from_spec' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:132:in remote: `do_install' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:123:in remote: `block in worker_pool' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:62:in remote: `apply_func' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:57:in remote: `block in process_queue' remote: :187:in `loop' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:54:in remote: `process_queue' remote: /tmp/build_91a6107a/vendor/ruby-3.3.0/lib/ruby/3.3.0/bundler/worker.rb:90:in remote: `block (2 levels) in create_threads' remote: remote: An error occurred while installing tiktoken_ruby (0.0.6), and Bundler cannot remote: continue. remote: remote: In Gemfile: remote: tiktoken_ruby remote: remote: ! remote: ! Failed to install gems via Bundler. remote: ! remote: ! Push rejected, failed to compile Ruby app. remote: remote: ! Push failed remote: ! remote: ! ## Warning - The same version of this code has already been built: 5888659e95baa9ba3bb99063346e68ec861aa781 remote: ! remote: ! We have detected that you have triggered a build from source code with version 5888659e95baa9ba3bb99063346e68ec861aa781 remote: ! at least twice. One common cause of this behavior is attempting to deploy code from a different branch. remote: ! remote: ! If you are developing on a branch and deploying via git you must run: remote: ! remote: ! git push heroku :main remote: ! remote: ! This article goes into details on the behavior: remote: ! https://devcenter.heroku.com/articles/duplicate-build-version remote: remote: Verifying deploy... remote: remote: ! Push rejected to tiktoken-ruby. remote: To https://git.heroku.com/tiktoken-ruby.git ! [remote rejected] main -> main (pre-receive hook declined) error: failed to push some refs to 'https://git.heroku.com/tiktoken-ruby.git' ```

[djrodgerspryor]

@karlentwistle you will still need to install libclang

[andreaslillebo]

Does anyone know a good way to install libclang on Heroku? It appears to no longer be included in the build environment on the heroku-22 stack.

[karlentwistle]

@karlentwistle you will still need to install libclang

Ah, I see, thanks very much for letting me know 🙇

Does anyone know a good way to install libclang on Heroku? It appears to no longer be included in the build environment on the heroku-22 stack.

I got it working with https://github.com/karlentwistle/tiktoken_heroku_example/commit/583a5c75c734ded7500e5f0cf7da0196c5a4108f and https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-apt

---

Can also now confirm this PR does indeed fix https://github.com/IAPark/tiktoken_ruby/issues/20 😄

[xfalcox]

Howdy @IAPark, this is blocking projects from moving to Ruby 3.3. Any chance this can be merged?

[ScotterC]

Curious. Why is libclang required now when it wasn't on Ruby 3.2? Did Ruby 3.2 handle it or did Tiktoken not require the dependency?

[bmulholland]

@ScotterC Maybe because there's a prebuilt version for ruby 3.2, but not 3.3? Ref: https://github.com/IAPark/tiktoken_ruby/issues/20#issuecomment-1910245815

[ScotterC]

Yup here's where the builds are created https://github.com/IAPark/tiktoken_ruby/blob/main/.github/workflows/build.yml#L9. If I understood more clearly how that build ends up creating the precompiled version on rubygems or locally then I'd create a PR for it. I'm loathe to add another buildpack to Heroku since I'm feeling the platform lock-in lately.

[xfalcox]

Did anyone here fork the gem for Ruby 3.3 ?

[ScotterC]

@xfalcox only forks updated recently are similar to this PR. Updating rb_sys. Here's thoughtbot's https://github.com/thoughtbot/tiktoken_ruby

[IAPark]

Closed in favor of https://github.com/IAPark/tiktoken_ruby/pull/22 (kinda by accident)

[IAPark]

Also, the truth is I created this gem in a weekend because I was annoyed at work and we've sense moved in a direction where it's irrelevant. It kinda just gives me a bit of stress now when I'm reminded of it. I think I should probably put a notice up asking if anyone wants to take it over

[IAPark]

In theory this should be out now though cross compiling seems broken and I might not be able to get back to it until next weekend at the earliest

[ScotterC]

@IAPark thanks for the work! I'll see if I can help find a maintainer