search

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust
MIT License
109 stars 26 forks source link

Error installing gem in MacOS M1 #34

Open AlessandroTolomio opened 2 months ago

AlessandroTolomio commented 2 months ago 
Installing tiktoken_ruby 0.0.8 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.0.0 -r ./siteconf20240418-10855-35ug4t.rb extconf.rb
checking for gcc... yes
checking for g++... yes
checking for gcc-ar... yes
checking for cargo... no

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby
make DESTDIR\= clean

current directory: /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby
make DESTDIR\=
info: downloading installer
info: profile set to 'minimal'
info: default host triple is aarch64-unknown-linux-gnu
info: skipping toolchain installation

Rust is installed now. Great!

To get started you need Cargo's bin directory 
(/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin)
in your PATH
environment variable. This has not been done automatically.

To configure your current shell, you need to source
the corresponding env file under 
/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo.

This is usually done by running one of the following (note the leading DOT):
. 
"/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/env"
# For sh/bash/zsh/ash/dash/pdksh
source 
"/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/env.fish"
# For fish
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
info: latest update on 2024-04-09, rust version 1.77.2 (25ef9e3d8 2024-04-09)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-std'
info: installing component 'rustc'

  stable-aarch64-unknown-linux-gnu installed - rustc 1.77.2 (25ef9e3d8 2024-04-09)

info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'
info: checking for self-update
info: using existing install for 'stable-aarch64-unknown-linux-gnu'
info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'

  stable-aarch64-unknown-linux-gnu unchanged - rustc 1.77.2 (25ef9e3d8 2024-04-09)

info: note that the toolchain 'stable-aarch64-unknown-linux-gnu' is currently in use (environment override by RUSTUP_TOOLCHAIN)
generating target/release/libtiktoken_ruby.so (release)
/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/bin/cargo rustc  --manifest-path ./Cargo.toml --target-dir target --lib --profile release -- -C linker=gcc -L native=/usr/local/lib -C link-arg=-lm
    Updating crates.io index
    Updating git repository `https://github.com/IAPark/tiktoken-rs.git`
    Updating git submodule `https://github.com/zurawiki/tiktoken`
 Downloading crates ...
  Downloaded aho-corasick v0.7.20
  Downloaded magnus-macros v0.6.0
  Downloaded shlex v1.1.0
  Downloaded autocfg v1.1.0
  Downloaded bit-set v0.5.3
  Downloaded either v1.10.0
  Downloaded cfg-if v1.0.0
  Downloaded glob v0.3.1
  Downloaded rb-sys-env v0.1.2
  Downloaded seq-macro v0.3.5
  Downloaded rustc-hash v1.1.0
  Downloaded scopeguard v1.1.0
  Downloaded once_cell v1.17.1
  Downloaded smallvec v1.10.0
  Downloaded quote v1.0.33
  Downloaded parking_lot_core v0.9.7
  Downloaded parking_lot v0.12.1
  Downloaded unicode-ident v1.0.8
  Downloaded serde v1.0.157
  Downloaded minimal-lexical v0.2.1
  Downloaded regex-automata v0.1.10
  Downloaded syn v2.0.31
  Downloaded bstr v1.4.0
  Downloaded libc v0.2.140
  Downloaded regex-syntax v0.6.28
  Downloaded regex v1.7.1
  Downloaded nom v7.1.3
  Downloaded magnus v0.6.1
  Downloaded itertools v0.12.1
  Downloaded bindgen v0.69.4
  Downloaded fancy-regex v0.11.0
  Downloaded base64 v0.21.0
  Downloaded proc-macro2 v1.0.66
  Downloaded memchr v2.5.0
  Downloaded lock_api v0.4.9
  Downloaded clang-sys v1.6.0
  Downloaded anyhow v1.0.70
  Downloaded shell-words v1.1.0
  Downloaded libloading v0.7.4
  Downloaded bitflags v2.4.0
  Downloaded rb-sys-build v0.9.87
  Downloaded cexpr v0.6.0
  Downloaded rb-sys v0.9.87
  Downloaded lazycell v1.3.0
  Downloaded lazy_static v1.4.0
  Downloaded bit-vec v0.6.3
   Compiling memchr v2.5.0
   Compiling proc-macro2 v1.0.66
   Compiling unicode-ident v1.0.8
   Compiling glob v0.3.1
   Compiling libc v0.2.140
   Compiling cfg-if v1.0.0
   Compiling minimal-lexical v0.2.1
   Compiling libloading v0.7.4
   Compiling clang-sys v1.6.0
   Compiling aho-corasick v0.7.20
   Compiling nom v7.1.3
   Compiling quote v1.0.33
   Compiling syn v2.0.31
   Compiling regex-syntax v0.6.28
   Compiling either v1.10.0
   Compiling bindgen v0.69.4
   Compiling itertools v0.12.1
   Compiling cexpr v0.6.0
   Compiling lazycell v1.3.0
   Compiling rustc-hash v1.1.0
   Compiling shlex v1.1.0
   Compiling regex v1.7.1
   Compiling bitflags v2.4.0
   Compiling lazy_static v1.4.0
   Compiling shell-words v1.1.0
   Compiling autocfg v1.1.0
   Compiling lock_api v0.4.9
   Compiling parking_lot_core v0.9.7
   Compiling anyhow v1.0.70
   Compiling scopeguard v1.1.0
   Compiling bit-vec v0.6.3
   Compiling smallvec v1.10.0
   Compiling rb-sys-env v0.1.2
   Compiling magnus v0.6.1
   Compiling bit-set v0.5.3
   Compiling regex-automata v0.1.10
   Compiling once_cell v1.17.1
   Compiling magnus-macros v0.6.0
   Compiling bstr v1.4.0
   Compiling rb-sys-build v0.9.87
   Compiling fancy-regex v0.11.0
   Compiling parking_lot v0.12.1
   Compiling base64 v0.21.0
   Compiling rb-sys v0.9.87
   Compiling seq-macro v0.3.5
   Compiling tiktoken-rs v0.3.2 (https://github.com/IAPark/tiktoken-rs.git#5231fbf4)
error: failed to run custom build command for `rb-sys v0.9.87`

Caused by:
  process didn't exit successfully: `/usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/target/release/build/rb-sys-c65a4afa838c9b06/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-env-changed=RBCONFIG_CROSS_COMPILING
  cargo:rerun-if-env-changed=RBCONFIG_RUBY_PROGRAM_VERSION
  cargo:rerun-if-env-changed=RBCONFIG_platform
  cargo:rerun-if-env-changed=RBCONFIG_arch
  cargo:rerun-if-env-changed=RUBY_ROOT
  cargo:rerun-if-env-changed=RUBY_VERSION
  cargo:rerun-if-env-changed=RUBY
  cargo:rerun-if-changed=build/version.rs
  cargo:rerun-if-changed=build/features.rs
  cargo:rerun-if-changed=build/stable_api_config.rs
  cargo:rerun-if-changed=build/main.rs
  cargo:rerun-if-env-changed=RUBY_STATIC
  cargo:rerun-if-env-changed=RBCONFIG_ENABLE_SHARED
  cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_rubyarchhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_CPPFLAGS
  cargo:rerun-if-env-changed=RBCONFIG_rubyhdrdir
  cargo:rerun-if-env-changed=RBCONFIG_MAJOR
  cargo:rerun-if-env-changed=RBCONFIG_MINOR
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS

  --- stderr
INFO: using bindgen with clang args: ["-I/usr/local/include/ruby-3.0.0", "-I/usr/local/include/ruby-3.0.0/aarch64-linux", "-fms-extensions", "-O3", "-ggdb3", "-Wall", "-Wextra", "-Wdeprecated-declarations", "-Wduplicated-cond", "-Wimplicit-function-declaration", "-Wimplicit-int",
"-Wmisleading-indentation", "-Wpointer-arith", "-Wwrite-strings", "-Wimplicit-fallthrough=0", "-Wmissing-noreturn", "-Wno-cast-function-type", "-Wno-constant-logical-operand", "-Wno-long-long", "-Wno-missing-field-initializers", "-Wno-overlength-strings", "-Wno-packed-bitfield-compat",
"-Wno-parentheses-equality", "-Wno-self-assign", "-Wno-tautological-compare", "-Wno-unused-parameter", "-Wno-unused-value", "-Wsuggest-attribute=format", "-Wsuggest-attribute=noreturn", "-Wunused-variable"]
  #include "ruby.h"

  #ifdef HAVE_RUBY_DEBUG_H
  #include "ruby/debug.h"
  #endif
  #ifdef HAVE_RUBY_DEFINES_H
  #include "ruby/defines.h"
  #endif
  #ifdef HAVE_RUBY_ENCODING_H
  #include "ruby/encoding.h"
  #endif
  #ifdef HAVE_RUBY_FIBER_SCHEDULER_H
  #include "ruby/fiber/scheduler.h"
  #endif
  #ifdef HAVE_RUBY_INTERN_H
  #include "ruby/intern.h"
  #endif
  #ifdef HAVE_RUBY_IO_H
  #include "ruby/io.h"
  #endif
  #ifdef HAVE_RUBY_MEMORY_VIEW_H
  #include "ruby/memory_view.h"
  #endif
  #ifdef HAVE_RUBY_MISSING_H
  #include "ruby/missing.h"
  #endif
  #ifdef HAVE_RUBY_ONIGMO_H
  #include "ruby/onigmo.h"
  #endif
  #ifdef HAVE_RUBY_ONIGURUMA_H
  #include "ruby/oniguruma.h"
  #endif
  #ifdef HAVE_RUBY_RACTOR_H
  #include "ruby/ractor.h"
  #endif
  #ifdef HAVE_RUBY_RANDOM_H
  #include "ruby/random.h"
  #endif
  #ifdef HAVE_RUBY_RE_H
  #include "ruby/re.h"
  #endif
  #ifdef HAVE_RUBY_REGEX_H
  #include "ruby/regex.h"
  #endif
  #ifdef HAVE_RUBY_RUBY_H
  #include "ruby/ruby.h"
  #endif
  #ifdef HAVE_RUBY_ST_H
  #include "ruby/st.h"
  #endif
  #ifdef HAVE_RUBY_THREAD_H
  #include "ruby/thread.h"
  #endif
  #ifdef HAVE_RUBY_THREAD_NATIVE_H
  #include "ruby/thread_native.h"
  #endif
  #ifdef HAVE_RUBY_UTIL_H
  #include "ruby/util.h"
  #endif
  #ifdef HAVE_RUBY_VERSION_H
  #include "ruby/version.h"
  #endif
  #ifdef HAVE_RUBY_VM_H
  #include "ruby/vm.h"
  #endif
  #ifdef HAVE_RUBY_WIN32_H
  #include "ruby/win32.h"
  #endif
  #ifdef HAVE_RUBY_IO_BUFFER_H
  #include "ruby/io/buffer.h"
  #endif
  #ifdef HAVE_RUBY_ATOMIC_H
  #include "ruby/atomic.h"
  #endif
  struct rb_sys__Opaque__RString { struct RString dummy; };
  struct rb_sys__Opaque__RArray { struct RArray dummy; };
  thread 'main' panicked at /usr/local/bundle/gems/tiktoken_ruby-0.0.8/ext/tiktoken_ruby/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.69.4/lib.rs:622:31:
  Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:552: target/release/libtiktoken_ruby.so] Error 101

make failed, exit code 2

Gem files will remain installed in /usr/local/bundle/gems/tiktoken_ruby-0.0.8 for inspection.
Results logged to /usr/local/bundle/extensions/aarch64-linux/3.0.0/tiktoken_ruby-0.0.8/gem_make.out

  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:93:in `run'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:44:in `block in make'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:36:in `each'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:36:in `make'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:63:in `block in build'
  /usr/local/lib/ruby/3.0.0/tempfile.rb:317:in `open'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:26:in `build'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:159:in `build_extension'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:193:in `block in build_extensions'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in `each'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in `build_extensions'
  /usr/local/lib/ruby/3.0.0/rubygems/installer.rb:837:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/rubygems_gem_installer.rb:76:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/rubygems_gem_installer.rb:28:in `install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/source/rubygems.rb:205:in `install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/gem_installer.rb:54:in `install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/parallel_installer.rb:132:in `do_install'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/installer/parallel_installer.rb:123:in `block in worker_pool'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:62:in `apply_func'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:57:in `block in process_queue'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:54:in `loop'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:54:in `process_queue'
  /usr/local/bundle/gems/bundler-2.5.7/lib/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing tiktoken_ruby (0.0.8), and Bundler cannot continue.