search

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust
MIT License
118 stars 26 forks source link

Update magnus for ruby-head compatibility #14

Closed casperisfine closed 1 year ago

casperisfine commented 1 year ago

Older versions of magnus fail to compile on Ruby 3.3.0-dev with:

Compiling tiktoken-rs v0.3.2
(https://github.com/IAPark/tiktoken-rs.git#5231fbf4)
error[E0609]: no field `len` on type `RString__bindgen_ty_1__bindgen_ty_1`
-->
/tmp/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.5/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/magnus-0.4.4/src/r_string.rs:366:57
    |
366 |             slice::from_raw_parts(h.ptr as *const u8, h.len as usize)
    |                                                         ^^^ unknown field
    |
    = note: available fields are: `ptr`, `aux`
error[E0609]: no field `len` on type `RString__bindgen_ty_1__bindgen_ty_1`
-->
/tmp/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.5/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/magnus-0.4.4/src/r_string.rs:954:19
    |
954 |                 h.len as usize
    |                   ^^^ unknown field
    |
    = note: available fields are: `ptr`, `aux`
error[E0609]: no field `len` on type `RString__bindgen_ty_1__bindgen_ty_2`
-->
/tmp/bundle/ruby/3.3.0+0/gems/tiktoken_ruby-0.0.5/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/magnus-0.4.4/src/r_string.rs:996:44
    |
996 |     value.as_internal().as_ref().as_.embed.len
    |                                            ^^^ unknown field
    |
    = note: available fields are: `ary`
For more information about this error, try `rustc --explain E0609`.
error: could not compile `magnus` (lib) due to 3 previous errors
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:564: target/release/libtiktoken_ruby.so] Error 101

cc @IAPark

casperisfine commented 1 year ago

This seem to not be quite enough:

  INFO: using bindgen with clang args: ["-I/home/runner/.rubies/ruby-head/include/ruby-3.3.0+0", "-I/home/runner/.rubies/ruby-head/include/ruby-3.3.0+0/x86_64-linux", "-fms-extensions", "-O3", "-fno-fast-math", "-ggdb3", "-Wall", "-Wextra", "-Wdeprecated-declarations", "-Wdiv-by-zero", "-Wduplicated-cond", "-Wimplicit-function-declaration", "-Wimplicit-int", "-Wmisleading-indentation", "-Wpointer-arith", "-Wwrite-strings", "-Wold-style-definition", "-Wimplicit-fallthrough=0", "-Wmissing-noreturn", "-Wno-cast-function-type", "-Wno-constant-logical-operand", "-Wno-long-long", "-Wno-missing-field-initializers", "-Wno-overlength-strings", "-Wno-packed-bitfield-compat", "-Wno-parentheses-equality", "-Wno-self-assign", "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value", "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn", "-Wunused-variable", "-Wundef", "-DENABLE_PATH_CHECK=0"]
  clang diag: warning: unknown warning option '-Wduplicated-cond' [-Wunknown-warning-option]
  clang diag: warning: unknown warning option '-Wimplicit-fallthrough=0'; did you mean '-Wimplicit-fallthrough'? [-Wunknown-warning-option]
  clang diag: warning: unknown warning option '-Wno-packed-bitfield-compat' [-Wunknown-warning-option]
  clang diag: warning: unknown warning option '-Wsuggest-attribute=format'; did you mean '-Wproperty-attribute-mismatch'? [-Wunknown-warning-option]
  clang diag: warning: unknown warning option '-Wsuggest-attribute=noreturn' [-Wunknown-warning-option]
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "Stable API is needed but could not find a candidate. Try enabling the `stable-api-compiled-fallback` feature in rb-sys."', /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rb-sys-0.9.81/build/main.rs:64:59
casperisfine commented 1 year ago

Ok, solved in 4ea7ced

ianks commented 1 year ago

LGTM, thanks for this

casperisfine commented 1 year ago

❤️

waldnzwrld commented 9 months ago

I'm still seeing issues trying to build against 3.3.0.dev

tiktoken_ruby-0.0.6-x86_64-linux requires ruby version >= 3.0, < 3.3.dev, which
is incompatible with the current version, 3.3.0.dev
ianks commented 9 months ago

@waldnzwrld can you post your logs of gem install tiktoken_ruby --verbose?

casperisfine commented 9 months ago

tiktoken_ruby-0.0.6-x86_64-linux requires ruby version >= 3.0, < 3.3.dev

That's trying to install the precompiled gem, which indeed can't be forward compatible since until yesterday the ABI for 3.3 wasn't frozen. You need to redo the Gemfile.lock resolution so it doesn't list that native gem as an option (e.g ruby ">= 3.3.0.dev" in Gemfile).

That's an annoying think about bundler when running multiple ruby versions concurrently.

waldnzwrld commented 9 months ago

Hi @ianks

This is what I'm seeing in logs when installing

Installing tiktoken_ruby 0.0.6 with native extensions
  Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

  current directory:
  /workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
  /workspace/fakename/vendor/ruby/eac197c7c45f9e9763ebd36933e5571c4c22d523/bin/ruby
  extconf.rb
  checking for gcc... yes
  checking for g++... yes
  checking for gcc-ar... yes
  checking for cargo... yes

  current directory:
  /workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
  make DESTDIR\= sitearchdir\=./.gem.20231212-3458-wj2vu6
  sitelibdir\=./.gem.20231212-3458-wj2vu6 clean

  current directory:
  /workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
  make DESTDIR\= sitearchdir\=./.gem.20231212-3458-wj2vu6
  sitelibdir\=./.gem.20231212-3458-wj2vu6
  generating target/release/libtiktoken_ruby.so (release)
  cargo rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile
  release -- -C linker=gcc -L
  native=/workspace/fakename/vendor/ruby/eac197c7c45f9e9763ebd36933e5571c4c22d523/lib
  -C link-arg=-lm -l pthread
      Updating crates.io index
      Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
      Updating git submodule `https://github.com/zurawiki/tiktoken`
   Downloading crates ...
    Downloaded parking_lot_core v0.9.7
    Downloaded base64 v0.21.0
    Downloaded glob v0.3.1
    Downloaded bit-set v0.5.3
    Downloaded rb-sys-build v0.9.81
    Downloaded shlex v1.1.0
    Downloaded peeking_take_while v0.1.2
    Downloaded magnus-macros v0.6.0
    Downloaded lazy_static v1.4.0
    Downloaded lazycell v1.3.0
    Downloaded shell-words v1.1.0
    Downloaded seq-macro v0.3.5
    Downloaded cfg-if v1.0.0
    Downloaded smallvec v1.10.0
    Downloaded quote v1.0.33
    Downloaded scopeguard v1.1.0
    Downloaded rb-sys-env v0.1.2
    Downloaded once_cell v1.17.1
    Downloaded rustc-hash v1.1.0
    Downloaded bit-vec v0.6.3
    Downloaded cexpr v0.6.0
    Downloaded proc-macro2 v1.0.66
    Downloaded autocfg v1.1.0
    Downloaded rb-sys v0.9.81
    Downloaded lock_api v0.4.9
    Downloaded libloading v0.7.4
    Downloaded bitflags v2.4.0
    Downloaded unicode-ident v1.0.8
    Downloaded clang-sys v1.6.0
    Downloaded parking_lot v0.12.1
    Downloaded memchr v2.5.0
    Downloaded serde v1.0.157
    Downloaded anyhow v1.0.70
    Downloaded regex-automata v0.1.10
    Downloaded fancy-regex v0.11.0
    Downloaded aho-corasick v0.7.20
    Downloaded minimal-lexical v0.2.1
    Downloaded nom v7.1.3
    Downloaded magnus v0.6.1
    Downloaded bindgen v0.66.1
    Downloaded syn v2.0.31
    Downloaded regex v1.7.1
    Downloaded regex-syntax v0.6.28
    Downloaded libc v0.2.140
    Downloaded bstr v1.4.0
     Compiling memchr v2.5.0
     Compiling proc-macro2 v1.0.66
     Compiling unicode-ident v1.0.8
     Compiling glob v0.3.1
     Compiling libc v0.2.140
     Compiling minimal-lexical v0.2.1
     Compiling cfg-if v1.0.0
     Compiling regex-syntax v0.6.28
     Compiling bindgen v0.66.1
     Compiling peeking_take_while v0.1.2
     Compiling lazy_static v1.4.0
     Compiling shlex v1.1.0
     Compiling rustc-hash v1.1.0
     Compiling lazycell v1.3.0
     Compiling bitflags v2.4.0
     Compiling shell-words v1.1.0
     Compiling autocfg v1.1.0
     Compiling parking_lot_core v0.9.7
     Compiling anyhow v1.0.70
     Compiling rb-sys-env v0.1.2
     Compiling smallvec v1.10.0
     Compiling bit-vec v0.6.3
     Compiling scopeguard v1.1.0
     Compiling once_cell v1.17.1
     Compiling regex-automata v0.1.10
     Compiling seq-macro v0.3.5
     Compiling base64 v0.21.0
     Compiling libloading v0.7.4
     Compiling bit-set v0.5.3
     Compiling magnus v0.6.1
     Compiling lock_api v0.4.9
     Compiling clang-sys v1.6.0
     Compiling aho-corasick v0.7.20
     Compiling bstr v1.4.0
     Compiling nom v7.1.3
     Compiling quote v1.0.33
     Compiling syn v2.0.31
     Compiling parking_lot v0.12.1
     Compiling regex v1.7.1
     Compiling fancy-regex v0.11.0
     Compiling cexpr v0.6.0
  Compiling tiktoken-rs v0.3.2
  (https://github.com/IAPark/tiktoken-rs.git#5231fbf4)
     Compiling magnus-macros v0.6.0
     Compiling rb-sys-build v0.9.81
     Compiling rb-sys v0.9.81
  Compiling tiktoken_ruby v0.1.0
  (/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby)
      Finished release [optimized] target(s) in 1m 02s

  current directory:
  /workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby
  make DESTDIR\= sitearchdir\=./.gem.20231212-3458-wj2vu6
  sitelibdir\=./.gem.20231212-3458-wj2vu6 install
  generating target/release/libtiktoken_ruby.so (release)
  cargo rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile
  release -- -C linker=gcc -L
  native=/workspace/fakename/vendor/ruby/eac197c7c45f9e9763ebd36933e5571c4c22d523/lib
  -C link-arg=-lm -l pthread
  Cleaning gem artifacts
     Compiling memchr v2.5.0
     Compiling proc-macro2 v1.0.66
     Compiling glob v0.3.1
     Compiling unicode-ident v1.0.8
     Compiling libc v0.2.140
     Compiling minimal-lexical v0.2.1
     Compiling cfg-if v1.0.0
     Compiling regex-syntax v0.6.28
     Compiling bindgen v0.66.1
     Compiling lazycell v1.3.0
     Compiling lazy_static v1.4.0
     Compiling bitflags v2.4.0
     Compiling peeking_take_while v0.1.2
     Compiling shlex v1.1.0
     Compiling rustc-hash v1.1.0
     Compiling shell-words v1.1.0
     Compiling autocfg v1.1.0
     Compiling parking_lot_core v0.9.7
     Compiling anyhow v1.0.70
     Compiling smallvec v1.10.0
     Compiling bit-vec v0.6.3
     Compiling scopeguard v1.1.0
     Compiling rb-sys-env v0.1.2
     Compiling once_cell v1.17.1
     Compiling regex-automata v0.1.10
     Compiling base64 v0.21.0
     Compiling seq-macro v0.3.5
  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/lazycell-348ec4e2743dfef2.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/autocfg-c2415f55f331390e.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/glob-44b64c14ef7c2ac3.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/unicode_ident-d482513fcea9d8f0.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/peeking_take_while-0c138926807536b3.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/shell_words-16933ed96576beb1.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/shlex-0151b85e4c6b41cc.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/minimal_lexical-833c3202e6748fd1.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/cfg_if-b19be2397e9c0806.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/bitflags-63c780c60828b175.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/rustc_hash-eca03913daa032a7.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/lazy_static-5f5ebbc577c8631f.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/smallvec-8a8345790498d659.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/cfg_if-5b8144915620281b.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/scopeguard-90fd033898a2182b.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/rb_sys_env-242b2c605794729b.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/bit_vec-f76b04a53f6afdd7.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/rustc_hash-b98cf1ecf3add1fe.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/lazy_static-e4c33e1ba5d50820.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/once_cell-bc46301c8cc3f1fd.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/base64-a225d6b0ec15c65a.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/seq_macro-2b4ccd6b39cb5899.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/regex_automata-34f7efc9f37d51b3.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/regex_syntax-b38ffb6a9e01c281.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/regex_syntax-f71c2fb7ebf99074.d`:
  No such file or directory (os error 2)

  error: could not compile `cfg-if` due to previous error
  warning: build failed, waiting for other jobs to finish...
  error: could not compile `unicode-ident` due to previous error
  error: could not compile `rustc-hash` due to previous error
  error: could not compile `shlex` due to previous error
  error: could not compile `cfg-if` due to previous error
  error: could not compile `scopeguard` due to previous error
  error: could not compile `lazycell` due to previous error
  error: could not compile `peeking_take_while` due to previous error
  error: could not compile `lazy_static` due to previous error
  error: could not compile `rustc-hash` due to previous error
  error: could not compile `lazy_static` due to previous error
  error: could not compile `seq-macro` due to previous error
  error: could not compile `rb-sys-env` due to previous error
  error: could not compile `shell-words` due to previous error
  error: could not compile `smallvec` due to previous error
  error: could not compile `bitflags` due to previous error
  error: could not compile `bit-vec` due to previous error
  error: could not compile `glob` due to previous error
  error: could not compile `autocfg` due to previous error
  error: could not compile `once_cell` due to previous error
  error: could not compile `minimal-lexical` due to previous error
  error: could not compile `base64` due to previous error
  error: could not compile `regex-automata` due to previous error
  error: could not compile `regex-syntax` due to previous error
  error: could not compile `regex-syntax` due to previous error
  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/memchr-6d62734455d7b383.d`:
  No such file or directory (os error 2)

  error: error writing dependencies to
  `/workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6/ext/tiktoken_ruby/target/release/deps/memchr-b86df8f71a178eb7.d`:
  No such file or directory (os error 2)

  error: could not compile `memchr` due to previous error
  error: could not compile `memchr` due to previous error
  make: *** [Makefile:565: target/release/libtiktoken_ruby.so] Error 101

  make install failed, exit code 2

  Gem files will remain installed in
  /workspace/fakename/vendor/gems/3.2.2/ruby/3.2.0/gems/tiktoken_ruby-0.0.6 for
  inspection.

Any thoughts?

ianks commented 9 months ago

Are you on Ruby 3.3? Rubygems is trying to write to 3.2.2 directory for some reason…

can you try upgrading rubygems itself as well as rb-sys?