Crashing (possibly while editing markdown)

[lf-]

Context of the crash:

I was editing some markdown containing haskell, while also having pretty big haskell files open.

~ » coredumpctl debug
           PID: 436573 (nvim)
           UID: 1000 (jade)
           GID: 100 (users)
        Signal: 6 (ABRT)
     Timestamp: Wed 2022-05-18 12:35:22 PDT (2min 21s ago)
  Command Line: /run/current-system/sw/bin/nvim --cmd $'let g:loaded_node_provider=0' --cmd $'let g:loaded_python_provider=0' --cmd $'let g:python3_host_prog=\'/nix/store/mv12ajfnyndzdc1isj0kgmwdjm61n023-neovim-0.7.0/bin/nvim-python3\'' --cmd $'let g:ruby_host_prog=\'/nix/store/mv12ajfnyndzdc1isj0kgmwdjm61n023-neovim-0.7.0/bin/nvim-ruby\'' -S a.vim
    Executable: /nix/store/02z6kwgc1ma1ra9ir2x1mnvm3qlz8s6l-neovim-unwrapped-0.7.0/bin/nvim
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-alacritty-ec1cd60ff21046dd880e5716a8652bdd.scope
          Unit: user@1000.service
     User Unit: app-alacritty-ec1cd60ff21046dd880e5716a8652bdd.scope
         Slice: user-1000.slice
     Owner UID: 1000 (jade)
       Boot ID: 350d1f0cacfd430f83ff6cb345ec866e
    Machine ID: 4fc42215004f4b53bc919a5207a4b10e
      Hostname: chonkpad
       Storage: /var/lib/systemd/coredump/core.nvim.1000.350d1f0cacfd430f83ff6cb345ec866e.436573.1652902522000000.zst (present)
     Disk Size: 25.9M
       Message: Process 436573 (nvim) of user 1000 dumped core.

                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/typescript.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/ruby.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/yaml.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/markdown.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/bash.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/make.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/jsdoc.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/nix.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/comment.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/haskell.so without build-id.
                Module /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/json.so without build-id.
                Module linux-vdso.so.1 with build-id 4d8f4ed93b54cb340698507a4a43e87763b45c66
                Module libstdc++.so.6 without build-id.
                Module libgcc_s.so.1 without build-id.
                Module ld-linux-x86-64.so.2 with build-id b9f66b930ff8f91e4f0c5a5166a2a646b8dd7392
                Module libpthread.so.0 with build-id 0fb27e00574442bff3b8e065ea25ee63a2a0a9a7
                Module libc.so.6 with build-id 3f866b74dd769cad8eb7a7cad6229ee4a6824184
                Module libluajit-5.1.so.2 without build-id.
                Module libutil.so.1 with build-id aa9275b88f13303064d81bd40899c4a86e5aa694
                Module libm.so.6 with build-id 995265d7140c8259c70e0e4ceef5651d8c37ab54
                Module libtree-sitter.so.0 without build-id.
                Module libunibilium.so.4 without build-id.
                Module libtermkey.so.1 without build-id.
                Module libvterm.so.0 without build-id.
                Module libmsgpackc.so.2 without build-id.
                Module librt.so.1 with build-id 51805a6bde589e18188284277aba28e598ed5020
                Module libdl.so.2 with build-id 6c0e4c7d7e709d6d0b6a41dd881875f8a3dafd80
                Module libuv.so.1 without build-id.
                Module libluv.so.1 without build-id.
                Module nvim without build-id.
                Stack trace of thread 436573:
                #0  0x00007f8a7c113adf __pthread_kill_implementation (libc.so.6 + 0x8badf)
                #1  0x00007f8a7c0c9062 raise (libc.so.6 + 0x41062)
                #2  0x00007f8a7c0b445c abort (libc.so.6 + 0x2c45c)
                #3  0x00007f8a7c0b4395 __assert_fail_base.cold.0 (libc.so.6 + 0x2c395)
                #4  0x00007f8a7c0c2082 __assert_fail (libc.so.6 + 0x3a082)
                #5  0x00007f8a6da1823f n/a (/home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/haskell.so + 0x923f)
                ELF object binary architecture: AMD x86-64

Reading symbols from /nix/store/02z6kwgc1ma1ra9ir2x1mnvm3qlz8s6l-neovim-unwrapped-0.7.0/bin/nvim...
(No debugging symbols found in /nix/store/02z6kwgc1ma1ra9ir2x1mnvm3qlz8s6l-neovim-unwrapped-0.7.0/bin/nvim)

warning: Can't open file /run/nscd/dboLhMRf (deleted) during file-backed mapping note processing
[New LWP 436573]
[New LWP 436973]
[New LWP 436972]
[New LWP 436971]
[New LWP 436974]
[New LWP 436574]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/ybkkrhdwdj227kr20vk8qnzqnmj7a06x-glibc-2.34-115/lib/libthread_db.so.1".
Core was generated by `/run/current-system/sw/bin/nvim --cmd let g:loaded_node_provider=0 --cmd let g:'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f8a7c113adf in __pthread_kill_implementation ()
   from /nix/store/ybkkrhdwdj227kr20vk8qnzqnmj7a06x-glibc-2.34-115/lib/libc.so.6
[Current thread is 1 (Thread 0x7f8a7c064740 (LWP 436573))]
warning: File "/nix/store/69brclzxp7mg927k6986hrfzyd1hpqgd-gcc-11.2.0-lib/lib/libstdc++.so.6.0.29-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/nix/store/pv1vnwdlqscmyvv1yqgpdw3hbh0flnrh-gcc-11.3.0-lib".
To enable execution of this file add
    add-auto-load-safe-path /nix/store/69brclzxp7mg927k6986hrfzyd1hpqgd-gcc-11.2.0-lib/lib/libstdc++.so.6.0.29-gdb.py
line to your configuration file "/home/jade/.config/gdb/gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "/home/jade/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path"
(gdb) c
The program is not being run.
(gdb) bt
#0  0x00007f8a7c113adf in __pthread_kill_implementation ()
   from /nix/store/ybkkrhdwdj227kr20vk8qnzqnmj7a06x-glibc-2.34-115/lib/libc.so.6
#1  0x00007f8a7c0c9062 in raise () from /nix/store/ybkkrhdwdj227kr20vk8qnzqnmj7a06x-glibc-2.34-115/lib/libc.so.6
#2  0x00007f8a7c0b445c in abort () from /nix/store/ybkkrhdwdj227kr20vk8qnzqnmj7a06x-glibc-2.34-115/lib/libc.so.6
#3  0x00007f8a7c0b4395 in __assert_fail_base.cold.0 ()
   from /nix/store/ybkkrhdwdj227kr20vk8qnzqnmj7a06x-glibc-2.34-115/lib/libc.so.6
#4  0x00007f8a7c0c2082 in __assert_fail () from /nix/store/ybkkrhdwdj227kr20vk8qnzqnmj7a06x-glibc-2.34-115/lib/libc.so.6
#5  0x00007f8a6da1823f in tree_sitter_haskell_external_scanner_serialize ()
   from /home/jade/.local/share/nvim/site/pack/packer/start/nvim-treesitter/parser/haskell.so
#6  0x00007f8a7c404f34 in ts_parser_parse ()
   from /nix/store/b0zvjj1gf6axy2sdqk1j2ddnf6zm55x6-tree-sitter-0.20.6/lib/libtree-sitter.so.0
#7  0x00000000005b11ce in parser_parse ()
#8  0x00007f8a7c292a36 in ?? ()
   from /nix/store/qcynznv8nr1kk0zrwjbw90pfik1yv0hs-luajit-2.1.0-2022-04-05-env/lib/libluajit-5.1.so.2
#9  0x00007f8a7c2ed334 in lua_pcall ()
   from /nix/store/qcynznv8nr1kk0zrwjbw90pfik1yv0hs-luajit-2.1.0-2022-04-05-env/lib/libluajit-5.1.so.2
#10 0x00000000005a0d48 in nlua_pcall.lto_priv ()
#11 0x00000000005af198 in nlua_call_ref ()
#12 0x0000000000736aaf in decor_provider_invoke.constprop ()
#13 0x00000000004b1d8a in decor_providers_invoke_buf ()
#14 0x0000000000678d4a in update_screen ()
#15 0x00000000004c43a2 in ins_compl_show_pum ()
#16 0x00000000004c4894 in ins_compl_new_leader.lto_priv ()
#17 0x00000000004c7a4e in insert_execute ()
#18 0x00000000006c84f0 in state_enter ()
#19 0x00000000004c3796 in edit ()
#20 0x00000000005fa64d in invoke_edit ()
#21 0x00000000005f210d in normal_execute.lto_priv ()
#22 0x00000000006c84f0 in state_enter ()
#23 0x00000000005edabb in normal_enter ()
#24 0x000000000044dcda in main ()
(gdb)

I can't provide the coredump as it contains confidential information, but let me know if there's anything I can get out of it that would be useful to debug.

I just ran :TSUpdate, so I should be on the latest version of the parser. nvim-treesitter was version 9069849, and my nvim is this:

:ver
NVIM v0.7.0
Build type: Release
LuaJIT 2.1.0-beta3
Compiled by nixbld

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/nix/store/02z6kwgc1ma1ra9ir2x1mnvm3qlz8s6l-neovim-unwrapped-0.7.0/share/nvim"

Run :checkhealth for more info

[tek]

hey @414owen, what's the reason for this being an assert?

  assert(to_copy <= TREE_SITTER_SERIALIZATION_BUFFER_SIZE);

I'd say there's a good chance that's what you're hitting here, @lf-

and also

chonkpad

:grin:

[414owen]

Huh, that's cool. I didn't know editors highlighted markdown blocks at all. I'll have a proper look tomorrow evening, but is it possible a realloc failed?

[tek]

@414owen I assumed it's not related to the markdown, but the "pretty big haskell file". though that size limit should only be hit as a consequence of nesting, shouldn't it?

in any case, the backtrace shows the error to be caused by __assert_fail, called from scanner_serialize, where this buffer size assert is, so I think that's a safe bet

[tek]

the scanner in the Python grammar simply cuts off the buffer at the size limit, so should we maybe replace the assert with that?

[414owen]

Right, gotcha.

https://github.com/tree-sitter/tree-sitter-haskell/pull/62/files#diff-bbcf6c785b007d9655b443abc204cc3cc32aa03cea190cb865fe44252dc6f1dfL1618

What we had before was presumably undefined behaviour, so I added the assert.

A few other projects seem to not copy the buffer at all if it's too large.
The docs say For your parser to work correctly, the serialize function must store its entire state, and deserialize must restore the entire state. Whether or not his means we'll be buggy either way, I'm not sure. @maxbrunsfeld would it be possible to add a line to the docs about what to do if you simply don't have enough buffer space?

One thing we could do, is we could store uint8_ts for the indents, instead of uint16_t. That way, we'd be able to support files that have twice as many lexically nested layout-dependent constructs, but we'd only support indents only up to 255 chars.

I'll make a PR where we just don't copy the buffer at all. Not sure what @lf- will see visually in their markdown file after that.

@lf- I know you can't reveal the contents of your file, but maybe you could you tell us what the many layout-defining constructs are?

[lf-]

The odd thing is that it is what I would call a pretty normal markdown file. I could lorem ipsum it and mechanically rename everything in the code blocks if you'd like. Frustratingly there's not an obvious repro, it just happened while writing in this file. Also, I'm very intrigued at why this parser is getting invoked with the particular stack trace that it is: called under pop-up menu completion (i use coc, which shouldn't do anything with treesitter there, so I'm very intrigued).

I would kill to have symbols or at least context? parser state? anything for all this, so I might figure out how to convince Nix to build a neovim and luajit and the treesitter plugin with symbols. No promises though, given it's work code and kinda a yak shave.

[tek]

mysterious!