Rails bootsnap Cannot load such file -- tiktoken_ruby

[night91]

Describe the bug Cannot load such file -- tiktoken_ruby

File: https://github.com/patterns-ai-core/langchainrb/blob/main/lib/langchain/utils/token_length/openai_validator.rb has a require for require "tiktoken_ruby".

This happen to me in Travis build running specs, launching rails server or running specs doesn't fail in my localhost.

To Reproduce Langchain::Utils::TokenLength::OpenAIValidator

Expected behavior An error about a missing dependency should not happen.

Terminal commands & output

Failure/Error: require_relative '../config/environment'
LoadError:
  cannot load such file -- tiktoken_ruby
# ./vendor/bundle/ruby/3.2.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:27:in `require'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/kernel.rb:34:in `require'
# ./vendor/bundle/ruby/3.2.0/gems/langchainrb-0.13.3/lib/langchain/utils/token_length/openai_validator.rb:3:in `<main>'
# ./vendor/bundle/ruby/3.2.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/kernel.rb:26:in `require'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/helpers.rb:148:in `const_get'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/helpers.rb:148:in `cget'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/eager_load.rb:175:in `block in actual_eager_load_dir'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/helpers.rb:42:in `block in ls'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/helpers.rb:25:in `each'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/helpers.rb:25:in `ls'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/eager_load.rb:170:in `actual_eager_load_dir'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/eager_load.rb:17:in `block (2 levels) in eager_load'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/eager_load.rb:16:in `each'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/eager_load.rb:16:in `block in eager_load'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/eager_load.rb:10:in `synchronize'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader/eager_load.rb:10:in `eager_load'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader.rb:413:in `block in eager_load_all'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader.rb:411:in `each'
# ./vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.15/lib/zeitwerk/loader.rb:411:in `eager_load_all'
# ./vendor/bundle/ruby/3.2.0/gems/railties-7.1.3.4/lib/rails/application/finisher.rb:80:in `block in <module:Finisher>'
# ./vendor/bundle/ruby/3.2.0/gems/railties-7.1.3.4/lib/rails/initializable.rb:32:in `instance_exec'
# ./vendor/bundle/ruby/3.2.0/gems/railties-7.1.3.4/lib/rails/initializable.rb:32:in `run'
# ./vendor/bundle/ruby/3.2.0/gems/railties-7.1.3.4/lib/rails/initializable.rb:61:in `block in run_initializers'
# ./vendor/bundle/ruby/3.2.0/gems/railties-7.1.3.4/lib/rails/initializable.rb:60:in `run_initializers'
# ./vendor/bundle/ruby/3.2.0/gems/railties-7.1.3.4/lib/rails/application.rb:426:in `initialize!'
# ./config/environment.rb:5:in `<top (required)>'
# ./spec/rails_helper.rb:4:in `require_relative'
# ./spec/rails_helper.rb:4:in `<top (required)>'
# ./spec/components/meta_title/component_spec.rb:3:in `<top (required)>'

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: OS X
• Ruby version 3.2.2
• Langchain.rb version 0.13.3

Additional context Add any other context about the problem here.

[bknoles]

I ran into this yesterday within Github Actions. The gem specifies ruby_tiktoken as an optional dependency, but it looks like if your code is using the OpenAI adapter, ruby_tiktoken is required. The workaround/fix for me was just to explicitly add it to my Gemfile: bundle add tiktoken

[night91]

@bknoles That would do it, buy then it should be in the docs.

[bknoles]

Yeah, I wasn't suggesting it as a long term fix. Just wanted to document the workaround I used to keep my side project moving!

https://xkcd.com/979/