Kakoune becomes unresponsive when using kak-tree-sitter with golang

[garymjr]

I am currently testing kak-tree-sitter with golang, and it’s working great! However, I am encountering an issue where kakoune becomes completely unresponsive after running for ~30-45 minutes. In order to resolve this, I have to terminate the shell session it’s running in and then kill both the kak and kak-tree-sitter processes before restarting kakoune.

I would be more than happy to provide any information you need to assist with troubleshooting. It’s worth noting that I usually have a split open with multiple buffers loaded, all of which have tree-sitter enabled when this issue occurs.

[hadronized]

Were you able to find a way to reproduce it?

[garymjr]

It happens every time. The conditions are sometimes different, for example I've had it happen with a kak-lsp autocomplete menu open as well, but it's always with kak-tree-sitter turned on and multiple buffers loaded.

[hadronized]

Okay, I’ll try to see what’s going on. If you happen to have logs, which should be located in $XDG_RUNTIME_DIR/kak-tree-sitter, I’d be very grateful.

[garymjr]

Here's a pastebin of the stdout.txt file after the latest freeze. The stderr.txt file was empty.

[garymjr]

It’s possible this could have been from a conflict between kak-lsp semantic highlighting and kak-tree-sitter. I turned off semantic highlighting and it seems to have stop happening.

[garymjr]

It’s possible this could have been from a conflict between kak-lsp semantic highlighting and kak-tree-sitter. I turned off semantic highlighting and it seems to have stop happening.

It turns out I was mistaken and it happened again even with semantic tokens turned off.

[hadronized]

Thank you for the feedback. I’m wondering whether it’s a gopls problem, or something with tree-sitter. Do you have a module / file I could try it in? An empty .go file yields the problem? I’m currently working on enhancing error handling so the next version should be easier to debug that kind of problem.

Also, how did you install kak-tree-sitter?

[hadronized]

I just reproduced at work.

[hadronized]

I have more news about that. The problem seems to be related to too many forks, which increases the number of tasks running. There is basically a fork leak.

[hadronized]

I have completely changed the backend used for IO (from tokio to mio) and the problem can still happen. I think it’s either a daemonize bug, or something relating to tree-sitter and/or tree-sitter-highlight.

[hadronized]

Actually, I was using an old build. With the current HEAD the bug seems fixed. I assume it was a misuse of tokio’s blocking threads.

Feel free to reopen if you think the problem is not solved. I have noticed some I/O errors sometimes but that was (I think?) related to my machine that was in a weird state.

[hadronized]

I’m reopening as I noticed the problem again.

It’s probably a daemonize thing as I noticed I never have that problem when running as a standalone server.

[hadronized]

From #119: it looks like we are spawning hundreds / thousands of kak processes. That happens wherever we are using --daemon or not.

[hadronized]

hint: could it be a kak -p bug?

[refi64]

I've started randomly hitting this pretty quickly into editing lately. It appears to happen while kakoune is trying to write to the fifo (this is heavily trimmed down and excluding me trying to instinctively print x64 registers lol):

$ gdb -p $(pidof kak)
[...]
(gdb) bt
#0  0x0000ffffa0518b3c in open64 () from /nix/store/7zii1yvdbwchx06qz6sd82d1p53jx86y-glibc-2.37-8/lib/libc.so.6
#1  0x000000000050e450 in Kakoune::write_buffer_to_file(Kakoune::Buffer&, Kakoune::StringView, Kakoune::WriteMethod, Kakoune::WriteFlags) ()
#2  0x00000000004ced0c in Kakoune::(anonymous namespace)::do_write_buffer(Kakoune::Context&, Kakoune::Optional<Kakoune::String>, Kakoune::WriteFlags, Kakoune::Optional<Kakoune::WriteMethod>) ()
#3  0x00000000004cf37c in void Kakoune::(anonymous namespace)::write_buffer<false>(Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&) ()
#4  0x00000000004b0214 in Kakoune::CommandManager::execute_single_command(Kakoune::ArrayView<Kakoune::String const>, Kakoune::Context&, Kakoune::ShellContext const&) ()
#5  0x00000000004b1df8 in Kakoune::CommandManager::execute(Kakoune::StringView, Kakoune::Context&, Kakoune::ShellContext const&) ()
#6  0x00000000004d0698 in Kakoune::(anonymous namespace)::evaluate_commands_cmd::{lambda(Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&)#1}::operator()(Kakoune::ParametersParser, Kakoune::ParametersParse
#7  0x00000000004de0a8 in void Kakoune::(anonymous namespace)::context_wrap<Kakoune::(anonymous namespace)::evaluate_commands_cmd::{lambda(Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&)#1}::operator()(Kakoune::ParametersParser, Kakoune::ParametersParser const&, Kakoune::ShellContext) const::{lambda(Kakoune::ParametersParser, Kakoune::ParametersParser const&)#1}>(Kakoune::ParametersParser, Kakoune::ParametersParser const&, Kakoune::StringView, Kakoune::(anonymous namespace)::evaluate_commands_cmd::{lambda(Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&)#1}::operator()(Kakoune::ParametersParser, Kakoune::ParametersParser const&, Kakoune::ShellContext) const::{lambda(Kakoune::ParametersParser, Kakoune::ParametersParser const&)#1}) [clone .constprop.0] [clone .isra.0] ()
#8  0x00000000004b0214 in Kakoune::CommandManager::execute_single_command(Kakoune::ArrayView<Kakoune::String const>, Kakoune::Context&, Kakoune::ShellContext const&) ()
#9  0x00000000004b1df8 in Kakoune::CommandManager::execute(Kakoune::StringView, Kakoune::Context&, Kakoune::ShellContext const&) ()
#10 0x00000000004cad84 in std::_Function_handler<void (Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&), Kakoune::(anonymous namespace)::define_command(Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&)::{lambda(Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&)#2}>::_M_invoke(std::_Any_data const&, Kakoune::ParametersParser const&, Kakoune::Context&, Kakoune::ShellContext const&) ()
#11 0x00000000004b0214 in Kakoune::CommandManager::execute_single_command(Kakoune::ArrayView<Kakoune::String const>, Kakoune::Context&, Kakoune::ShellContext const&) ()
#12 0x00000000004b1df8 in Kakoune::CommandManager::execute(Kakoune::StringView, Kakoune::Context&, Kakoune::ShellContext const&) ()
#13 0x0000000000544624 in Kakoune::HookManager::run_hook(Kakoune::Hook, Kakoune::StringView, Kakoune::Context&) ()
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) p $x8
$3 = 56  [this is openat]
[...]
(gdb) p (char*)$x1
$6 = 0x2f791fb0 "/run/user/1000/kak-tree-sitter/buffers/88505"
(gdb) p $x2
$7 = 577  [this is O_WRONLY | O_CREAT | O_TRUNC]
(gdb) p/o $x3
$8 = 0644

During this time, kak-tree-sitter seems to be...just waiting for events?

$ gdb -p $(pidof kak-tree-sitter)
[...]
(gdb) bt
#0  0x0000ffffad077184 in epoll_pwait () from /nix/store/7zii1yvdbwchx06qz6sd82d1p53jx86y-glibc-2.37-8/lib/libc.so.6
#1  0x0000aaaae9d28e54 in mio::sys::unix::selector::epoll::Selector::select::h9a02f6e9016cb701 ()
#2  0x0000aaaae9cf14b0 in kak_tree_sitter::server::Server::bootstrap::hb2b506cc16237e41 ()
#3  0x0000aaaae9d17690 in kak_tree_sitter::main::h7e1c0f19fa18c8a0 ()
#4  0x0000aaaae9d0e84c in std::sys_common::backtrace::__rust_begin_short_backtrace::hf753d16413dca37e ()
#5  0x0000aaaae9d0e860 in std::rt::lang_start::{{closure}}::h1606e8cb68522d6b ()
#6  0x0000aaaae9e71c48 in std::rt::lang_start_internal::hd4f9ead014c19d1a ()
#7  0x0000aaaae9d17bb0 in main ()

Apparently FIFOs block when the other end isn't being used?

Once you have created a FIFO special file in this way, any process can open it for reading or writing, in the same way as an ordinary file. However, it has to be open at both ends simultaneously before you can proceed to do any input or output operations on it.

And as it turns out, this gives the perfect workaround! You can just cat the FIFO, and kakoune gets unblocked:

$ cat /run/user/1000/kak-tree-sitter/buffers/[the-session-id]
[contents of file here]

After doing this...kak-tree-sitter also continued to work? I have no idea why. Maybe it's some deadlock where kak-tree-sitter isn't listening to the FIFO because it's waiting for something from Kakoune, but Kakoune is too busy writing to the FIFO? It might be useful in the future for kak-tree-sitter's kakoune integration to use a shell command to write to the FIFO instead of write directly, that way it can be interrupted? Or use an intermediate Rust CLI in order to set a timeout, etc etc.

As an aside, I don't think the extra kak processes are actually tied to this; they're all just zombies that were never waited on:

$ ps -A | rg kak | head
  88505 pts/4    00:00:02 kak
  88553 ?        00:00:00 kak-lsp
  88559 ?        00:00:02 kak-tree-sitter
  88744 ?        00:00:00 kak <defunct>
  89050 ?        00:00:00 kak <defunct>
  89177 ?        00:00:00 kak <defunct>
  89198 ?        00:00:00 kak <defunct>
  89211 ?        00:00:00 kak <defunct>
  89218 ?        00:00:00 kak <defunct>
  89229 ?        00:00:00 kak <defunct>
$ ps -A | rg kak | rg -v 'defunct'
  88505 pts/4    00:00:02 kak
  88553 ?        00:00:00 kak-lsp
  88559 ?        00:00:02 kak-tree-sitter

Still arguably a bug, but a largely unrelated one.

[hadronized]

Yes, that FIFO bug was recently fixed — I’m in the process of releasing kak-tree-sitter v0.4.5 to fix exactly that.

I’m also investigating this zombie bug. If it’s just a join / wait problem, I should be able to fix it on the fly.

Btw, as a workaround, I simply used to resize my tmux window lol. I guess it made Kakoune read something else and abort the FIFO write.

After doing this...kak-tree-sitter also continued to work? I have no idea why. Maybe it's some deadlock where kak-tree-sitter isn't listening to the FIFO because it's waiting for something from Kakoune, but Kakoune is too busy writing to the FIFO?

Yes, basically, before my very last fix from today, it worked in an awkward way. The « command FIFO » is open in non-blocking mode, but buffer content is streamed with a blocking FIFO. And I didn’t support EAGAIN / EWOULDBLOCK correctly, so every time we read a command from a FIFO, we would block on the buffer content, and since it’s the next operation in the kak command… it blocks the editor.

The fix I made turns all FIFO non-blocking and adds a state machine to handle all this world in an asynchronous way, so that I can properly handle EAGAIN / EWOULDBLOCK. For instance, something else that surfaced: if you want to highlight a big buffer, the whole write operation by Kakoune is probably slightly slower than the read from KTS. Hence, even when a reader and writer have opened the FIFO, reading from KTS eventually exhaust the pipe, setting the EWOULDBLOCK error, which is not something I had planned in the first place.

All of that is fixed on master and will be in v0.4.5 very soon!

[mikelu92]

Hi not sure if anyone else is having trouble with this but I'm still having trouble (I think?) with blocked fifos. I'm on kak-tree-sitter 0.4.6-dev-4b7ca5e and kakoune v2022.10.31-371-g14b60ab9

[hadronized]

Hello @mikelu92. It’s a regression I introduced a few months ago, and I fixed it in #172.