Experiencing segmentations faults

[aweis89]

I'm experiencing segfaults when using copilot in Neovim, I'm not sure if it's specific to my setup or what it might be

[afrhan-repo]

can you show the error message?

[gptlang]

Do you happen to have tiktoken.lua installed? It's difficult to imagine Lua would segfault.

[tfriedel]

I also experienced segfaults. They originated in tiktoken. I had tried installing binaries and compiling it on ubuntu 24.04. Both same result (segfault). What worked was uninstalling neovim and luarocks with apt and installing them with brew. Then compiling tiktoken using the luarocks that was installed in brew.

edit: turns out when I compiled it with brew it was installed, but not found. So copilotchat now works without it. Not sure what is even the benefit of counting the tokens?

[deathbeam]

Yea i experienced same issue when I installed tiktoken binary on new notebook, definitely big issue, @gptlang any idea?

[deathbeam]

Output from env RUST_BACKTRACE=1 nvim --headless:

thread '<unnamed>' panicked at src/lib.rs:222:41:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: tiktoken_core::new
   4: mlua::lua::Lua::create_function::{{closure}}
   5: mlua::lua::Lua::create_callback::call_callback
   6: <unknown>
   7: lua_pcall
   8: nlua_pcall
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: uv_run
  14: <unknown>
  15: loop_poll_events
  16: process_wait
  17: <unknown>
  18: os_call_shell
  19: call_shell
  20: os_expand_wildcards
  21: gen_expand_wildcards
  22: expand_wildcards
  23: expand_wildcards_eval
  24: <unknown>
  25: <unknown>
  26: <unknown>
  27: ExpandOne
  28: <unknown>
  29: call_internal_func
  30: call_func
  31: nlua_call
  32: <unknown>
  33: <unknown>
  34: <unknown>
  35: lua_pcall
  36: nlua_pcall
  37: nlua_exec_file
  38: do_source
  39: <unknown>
  40: <unknown>
  41: main
  42: <unknown>
  43: __libc_start_main
  44: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[1]    3847543 segmentation fault (core dumped)  env RUST_BACKTRACE=1 nvim --headless

[tfriedel]

This is what I got, for RUST_BACKTRACE=full:

       at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7fe7076288bb - std::sys_common::backtrace::_print_fmt::hf9c799a68b2477fc
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x7fe7076288bb - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf2d48dfeb5948660
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7fe707660fd0 - core::fmt::rt::Argument::fmt::h507ca91fbb1ef494
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/core/src/fmt/rt.rs:142:9
   5:     0x7fe707660fd0 - core::fmt::write::h4d5f6025aa566322
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/core/src/fmt/mod.rs:1120:17
   6:     0x7fe70762661d - std::io::Write::write_fmt::hc088b3c8cc6dcb9f
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/io/mod.rs:1762:15
   7:     0x7fe7076286a4 - std::sys_common::backtrace::_print::h231201f6cff8f56a
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7fe7076286a4 - std::sys_common::backtrace::print::h478d6be486f8cd2d
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7fe7076440ca - std::panicking::default_hook::{{closure}}::hf1c55106b4d45a90
  10:     0x7fe707643d6d - std::panicking::default_hook::h7b29000c631cc719
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/panicking.rs:292:9
  11:     0x7fe707644408 - std::panicking::rust_panic_with_hook::h2a988919dd121561
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/panicking.rs:779:13
  12:     0x7fe707628c9e - std::panicking::begin_panic_handler::{{closure}}::h3a9eaa1d23ecf555
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/panicking.rs:657:13
  13:     0x7fe707628ad6 - std::sys_common::backtrace::__rust_end_short_backtrace::h813561fdec15f700
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/sys_common/backtrace.rs:170:18
  14:     0x7fe707644132 - rust_begin_unwind
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/panicking.rs:645:5
  15:     0x7fe707512e55 - core::panicking::panic_fmt::hef0307862026e6f9
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/core/src/panicking.rs:72:14
  16:     0x7fe707513563 - core::result::unwrap_failed::h8393ef8d489acb45
                               at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/core/src/result.rs:1653:5
  17:     0x7fe707515fbf - tiktoken_core::new::hbcb3333be3b3fc91
  18:     0x7fe70751cee4 - mlua::lua::Lua::create_function::{{closure}}::heed2363ff805704c
  19:     0x7fe7075295d8 - mlua::lua::Lua::create_callback::call_callback::h2157948e4b34a838
  20:     0x7fe707f46a36 - <unknown>
  21:     0x7fe707fa2637 - lua_pcall
  22:     0x55ecdd0a365c - nlua_pcall
  23:     0x55ecdd0abc29 - nlua_luv_cfpcall
  24:     0x7fe70805f415 - luv_async_cb
  25:     0x7fe707e0be94 - uv__async_io.part.0
  26:     0x7fe707e1fa53 - uv__io_poll
  27:     0x7fe707e0cb96 - uv_run
  28:     0x55ecdd00bc3b - loop_poll_events
  29:     0x55ecdd0188bc - process_wait
  30:     0x55ecdd13dd52 - do_os_system.lto_priv.0
  31:     0x55ecdd13e489 - os_call_shell
  32:     0x55ecdd13e924 - call_shell
  33:     0x55ecdd13f385 - os_expand_wildcards
  34:     0x55ecdd144c0c - gen_expand_wildcards.part.0
  35:     0x55ecdd14543d - expand_wildcards
  36:     0x55ecdd145682 - expand_wildcards_eval
  37:     0x55ecdcf92266 - ExpandFromContext
  38:     0x55ecdcf951bb - ExpandOne
  39:     0x55ecdcfe733e - f_expand.lto_priv.0
  40:     0x55ecdcfdf756 - call_internal_func
  41:     0x55ecdd00a16e - call_func
  42:     0x55ecdd0b2357 - nlua_call
  43:     0x7fe707f46a36 - <unknown>
  44:     0x7fe707fa2637 - lua_pcall
  45:     0x55ecdd0a365c - nlua_pcall
  46:     0x55ecdd0b1559 - nlua_typval_exec.part.0
  47:     0x55ecdd0b3278 - ex_lua
  48:     0x55ecdd253c01 - execute_cmd0.constprop.0
  49:     0x55ecdd02a3ca - do_cmdline
  50:     0x55ecdd18e8b7 - do_source
  51:     0x55ecdcf2f937 - main
  52:     0x7fe707c131ca - __libc_start_call_main
                               at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  53:     0x7fe707c1328b - __libc_start_main_impl
                               at ./csu/../csu/libc-start.c:360:3
  54:     0x55ecdcf30465 - _start
  55:                0x0 - <unknown>

[gptlang]

I haven't had that issue. I'll try to reproduce it

[gptlang]

Ok I've found the issue. It was newly introduced because of vim.schedule & concurrency issues such that we falsely expect the tiktoken file to be downloaded despite it not being so