search

volition-co / tiktoken

MIT License
3 stars 2 forks source link

Invalid model error when calling `encoding_for_model` on Heroku #1

Open developius opened 1 year ago

developius commented 1 year ago

Hi, thanks for publishing the gem! I'm using it on Heroku with stack heroku-22 but getting an error when calling encoding_for_model. Locally it works just fine.

Ruby: 3.2.2 OS: Ubuntu 22.04

Here's a Dockerfile that exhibits the issue (tested on play-with-docker):

FROM heroku/heroku:22
RUN gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
RUN curl -sSL https://get.rvm.io | bash -s stable
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && rvm install 3.2.2 && rvm use 3.2.2"
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && ruby -v"
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && gem install tiktoken"
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && ruby -e 'require \"tiktoken\"; Tiktoken.encoding_for_model(\"text-embedding-ada-002\")'"

And the error:

/usr/local/rvm/gems/ruby-3.2.2/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:16:in `_encoding_for_model': Invalid model:  (ArgumentError)

    Tiktoken._encoding_for_model(model, extra_special_tokens)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        from /usr/local/rvm/gems/ruby-3.2.2/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:16:in `encoding_for_model'
        from -e:1:in `<main>'

On my Mac (M1), the Tiktoken.encoding_for_model("text-embedding-ada-002") works just fine.

Any ideas what might be causing this? Perhaps an architecture problem?

developius commented 1 year ago

I got it to segfault on Heroku too, when calling encoding_for_model:

Segfault ``` irb(main):001:0> Tiktoken.encoding_for_model("gpt-3.5-turbo") /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:16:in `_encoding_for_model': Invalid model: (ArgumentError) Tiktoken._encoding_for_model(model, extra_special_tokens) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ irb(main):002:0> Tiktoken.get_encoding("cl100k_base") /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12: [BUG] Segmentation fault at 0x0000000000000010 ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] -- Control frame information ----------------------------------------------- c:0033 p:---- s:0208 e:000207 CFUNC :_get_encoding c:0032 p:0012 s:0202 e:000201 METHOD /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12 c:0031 p:0006 s:0196 e:000195 EVAL (irb):2 [FINISH] c:0030 p:---- s:0193 e:000192 CFUNC :eval c:0029 p:0020 s:0185 e:000184 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/workspace.rb:119 c:0028 p:0153 s:0177 e:000175 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/context.rb:502 c:0027 p:0119 s:0164 e:000163 BLOCK /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:588 c:0026 p:0024 s:0158 e:000157 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:777 c:0025 p:0007 s:0152 e:000151 BLOCK /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:567 c:0024 p:0126 s:0147 e:000146 BLOCK /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:267 [FINISH] c:0023 p:---- s:0143 e:000142 CFUNC :loop c:0022 p:0005 s:0139 e:000138 BLOCK /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:249 [FINISH] c:0021 p:---- s:0136 e:000135 CFUNC :catch c:0020 p:0010 s:0131 e:000130 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:248 c:0019 p:0050 s:0126 E:000bf8 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:566 c:0018 p:0003 s:0121 e:000120 BLOCK /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:500 [FINISH] c:0017 p:---- s:0118 e:000117 CFUNC :catch c:0016 p:0050 s:0113 E:000ba0 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:499 c:0015 p:0069 s:0107 e:000106 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:421 c:0014 p:0114 s:0101 e:000100 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:70 c:0013 p:0009 s:0097 e:000096 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:19 c:0012 p:0033 s:0092 e:000091 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:102 c:0011 p:0054 s:0088 e:000087 METHOD /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/command.rb:27 c:0010 p:0040 s:0080 e:000079 METHOD /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/invocation.rb:127 c:0009 p:0213 s:0073 e:000072 METHOD /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor.rb:392 c:0008 p:0030 s:0060 e:000059 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command/base.rb:87 c:0007 p:0147 s:0053 e:000052 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command.rb:48 c:0006 p:0038 s:0041 e:000040 TOP /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands.rb:18 [FINISH] c:0005 p:---- s:0036 e:000035 CFUNC :require c:0004 p:0011 s:0031 e:000030 METHOD :37 c:0003 p:0158 s:0016 e:000015 METHOD /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32 c:0002 p:0024 s:0006 e:000005 EVAL /app/bin/rails:4 [FINISH] c:0001 p:0000 s:0003 E:001c00 DUMMY [FINISH] -- Ruby level backtrace information ---------------------------------------- /app/bin/rails:4:in `
' /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require' :37:in `require' :37:in `require' /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands.rb:18:in `
' /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command.rb:48:in `invoke' /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command/base.rb:87:in `perform' /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch' /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command' /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run' /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:102:in `perform' /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:19:in `start' /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:70:in `start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:421:in `start' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:499:in `run' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:499:in `catch' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:500:in `block in run' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:566:in `eval_input' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:248:in `each_top_level_statement' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:248:in `catch' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:249:in `block in each_top_level_statement' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:249:in `loop' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:267:in `block (2 levels) in each_top_level_statement' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:567:in `block in eval_input' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:777:in `signal_status' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:588:in `block (2 levels) in eval_input' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/context.rb:502:in `evaluate' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/workspace.rb:119:in `evaluate' /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/workspace.rb:119:in `eval' (irb):2:in `
' /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12:in `get_encoding' /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12:in `_get_encoding' -- Machine register context ------------------------------------------------ RIP: 0x00007f68d79d3610 RBP: 0x00007fff769f7558 RSP: 0x00007fff769f7028 RAX: 0x00007f68d7e17bc0 RBX: 0x0000000000000008 RCX: 0x00007f68d73c0b80 RDX: 0x00007f68d73c0000 RDI: 0x0000000000000008 RSI: 0x00007f68d3c5e930 R8: 0x0000000000000000 R9: 0x0000000000000000 R10: 0x0000000000000007 R11: 0x0000000000000007 R12: 0x00007fff769f7280 R13: 0x00007fff769f76f0 R14: 0x00007fff769f70a8 R15: 0x0000000000000004 EFL: 0x0000000000010202 -- C level backtrace information ------------------------------------------- /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_print_backtrace+0xd) [0x7f68d7b2bc9f] /ruby-3.2.2/vm_dump.c:785 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_bugreport) /ruby-3.2.2/vm_dump.c:1080 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_bug_for_fatal_signal+0xf4) [0x7f68d7922154] /ruby-3.2.2/error.c:813 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(sigsegv+0x4d) [0x7f68d7a7b23d] /ruby-3.2.2/signal.c:964 /lib/x86_64-linux-gnu/libc.so.6(0x7f68d7424520) [0x7f68d7424520] /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(RBASIC_CLASS+0x0) [0x7f68d79d3610] /ruby-3.2.2/object.c:885 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_class_of) ./include/ruby/internal/globals.h:175 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_obj_is_kind_of) /ruby-3.2.2/object.c:848 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_set_errinfo+0x3f) [0x7f68d792dacf] /ruby-3.2.2/eval.c:1878 /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d02296db) [0x7f68d02296db] /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d022aae5) [0x7f68d022aae5] /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d0225df6) [0x7f68d0225df6] /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d0223a3d) [0x7f68d0223a3d] /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d02243e4) [0x7f68d02243e4] /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d022145d) [0x7f68d022145d] /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d021ce3b) [0x7f68d021ce3b] /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d0222312) [0x7f68d0222312] /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0x97) [0x7f68d7b108e4] /ruby-3.2.2/vm_insnhelper.c:5080 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core) /ruby-3.2.2/insns.def:820 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x7f68d7b16213] /ruby-3.2.2/vm.c:2374 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_f_eval+0x160) [0x7f68d7b170e0] /ruby-3.2.2/vm_eval.c:1791 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0x97) [0x7f68d7b108e4] /ruby-3.2.2/vm_insnhelper.c:5080 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core) /ruby-3.2.2/insns.def:820 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0x5b2) [0x7f68d7b166f2] /ruby-3.2.2/vm.c:2383 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(invoke_block_from_c_bh+0x212) [0x7f68d7b176cf] /ruby-3.2.2/vm.c:1398 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_yield_with_cref) /ruby-3.2.2/vm.c:1509 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_yield) /ruby-3.2.2/vm.c:1517 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_yield_0) /ruby-3.2.2/vm_eval.c:1348 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(loop_i) /ruby-3.2.2/vm_eval.c:1447 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vrescue2+0x11e) [0x7f68d792b5be] /ruby-3.2.2/eval.c:917 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_rescue2+0x8e) [0x7f68d792b84e] /ruby-3.2.2/eval.c:898 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0xd2) [0x7f68d7b060f2] /ruby-3.2.2/vm_insnhelper.c:5080 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core+0x1e2) [0x7f68d7b109c2] /ruby-3.2.2/insns.def:801 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x7f68d7b16213] /ruby-3.2.2/vm.c:2374 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(catch_i+0x2d8) [0x7f68d7b189e8] /ruby-3.2.2/vm.c:1398 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_catch_protect+0xee) [0x7f68d7b0907e] /ruby-3.2.2/vm_eval.c:2393 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_catch_obj+0x51) [0x7f68d7b09221] /ruby-3.2.2/vm_eval.c:2419 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0xd2) [0x7f68d7b060f2] /ruby-3.2.2/vm_insnhelper.c:5080 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core+0x1e2) [0x7f68d7b109c2] /ruby-3.2.2/insns.def:801 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x7f68d7b16213] /ruby-3.2.2/vm.c:2374 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(catch_i+0x2d8) [0x7f68d7b189e8] /ruby-3.2.2/vm.c:1398 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_catch_protect+0xee) [0x7f68d7b0907e] /ruby-3.2.2/vm_eval.c:2393 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_catch_obj+0x51) [0x7f68d7b09221] /ruby-3.2.2/vm_eval.c:2419 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0xd2) [0x7f68d7b060f2] /ruby-3.2.2/vm_insnhelper.c:5080 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core+0x1e2) [0x7f68d7b109c2] /ruby-3.2.2/insns.def:801 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0x5b2) [0x7f68d7b166f2] /ruby-3.2.2/vm.c:2383 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(load_iseq_eval+0x3c) [0x7f68d7994401] /ruby-3.2.2/load.c:697 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(require_internal) /ruby-3.2.2/load.c:1203 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_require_string+0x42) [0x7f68d7994e62] /ruby-3.2.2/load.c:1294 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_alias+0x7d) [0x7f68d7b25fbd] /ruby-3.2.2/vm_insnhelper.c:3447 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0x97) [0x7f68d7b108e4] /ruby-3.2.2/vm_insnhelper.c:5080 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core) /ruby-3.2.2/insns.def:820 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0x5b2) [0x7f68d7b166f2] /ruby-3.2.2/vm.c:2383 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_ec_exec_node+0xb1) [0x7f68d7927441] /ruby-3.2.2/eval.c:289 /app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(ruby_run_node+0x93) [0x7f68d792d903] /ruby-3.2.2/eval.c:330 /app/vendor/ruby-3.2.2/bin/ruby(rb_main+0x21) [0x559404d55187] ./main.c:38 /app/vendor/ruby-3.2.2/bin/ruby(main) ./main.c:57 /lib/x86_64-linux-gnu/libc.so.6(0x7f68d740bd90) [0x7f68d740bd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f68d740be40] [0x559404d551d5] ```
lstone commented 1 year ago

Ever solve this issue?