Neovim treesitter plugin conflicts with bundled treesitter grammer

[lucc]

Describe the bug

When opening a help file in neovim some red error messages are printed on random lines in the buffer.

Steps To Reproduce

Steps to reproduce the behavior:

1. Use this default.nix file:

   { pkgs ? import <nixpkgs> { }, ... }:
   let
   neovim-with-treesitter = callback:
   pkgs.neovim.override {
     configure = {
       customRC = ''
         lua <<EOL
         require'nvim-treesitter.configs'.setup {
           highlight = { enable = true }
         }
         EOL
       '';
       packages.foo.start = [
         (pkgs.vimPlugins.nvim-treesitter.withPlugins callback)
       ];
     };
   };
   in
   {
   broken = neovim-with-treesitter (p: [ ]);
   ok = neovim-with-treesitter (p: [ p.vimdoc ]);
   }

2. build the two derivations nix-build -A broken --out-link result-broken && nix-build -A ok --out-link result-ok
3. try result-broken/bin/nvim +h
4. observe the red error message in the corner of the help window

Expected behavior

No error messages, like with result-ok/bin/nvim +h.

Additional context

The problem can also be seen with :checkhealth in the section for nvim-treesitter. Depending on which parsers are explicitly installed to hide the bundled parsers one can see these kind of errors (for the bundled parsers, with the broken derivation from above):

The following errors have been detected: ~
- ERROR c(highlights): ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 242 for language c
  c(highlights) is concatenated from the following files:
  | [ERROR]:"/nix/store/v244gg3vm6hasg25g5r5q8c95cnfmkn0-vim-pack-dir/pack/myNeovimPackages/start/nvim-treesitter/queries/c/highlights.scm", failed to load: ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 242 for language c
- ERROR c(folds): ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 289 for language c
  c(folds) is concatenated from the following files:
  | [ERROR]:"/nix/store/v244gg3vm6hasg25g5r5q8c95cnfmkn0-vim-pack-dir/pack/myNeovimPackages/start/nvim-treesitter/queries/c/folds.scm", failed to load: ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 289 for language c
- ERROR c(indents): ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 1109 for language c
  c(indents) is concatenated from the following files:
  | [ERROR]:"/nix/store/v244gg3vm6hasg25g5r5q8c95cnfmkn0-vim-pack-dir/pack/myNeovimPackages/start/nvim-treesitter/queries/c/indents.scm", failed to load: ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 1109 for language c
- ERROR lua(highlights): ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 3670 for language lua
  lua(highlights) is concatenated from the following files:
  | [ERROR]:"/nix/store/v244gg3vm6hasg25g5r5q8c95cnfmkn0-vim-pack-dir/pack/myNeovimPackages/start/nvim-treesitter/queries/lua/highlights.scm", failed to load: ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 3670 for language lua
- ERROR lua(injections): ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 3422 for language lua
  lua(injections) is concatenated from the following files:
  | [ERROR]:"/nix/store/v244gg3vm6hasg25g5r5q8c95cnfmkn0-vim-pack-dir/pack/myNeovimPackages/start/nvim-treesitter/queries/lua/injections.scm", failed to load: ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 3422 for language lua
- ERROR vimdoc(highlights): ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 615 for language vimdoc
  vimdoc(highlights) is concatenated from the following files:
  | [ERROR]:"/nix/store/v244gg3vm6hasg25g5r5q8c95cnfmkn0-vim-pack-dir/pack/myNeovimPackages/start/nvim-treesitter/queries/vimdoc/highlights.scm", failed to load: ...ed-0.9.5/share/nvim/runtime/lua/vim/treesitter/query.lua:259: query: invalid node type at position 615 for language vimdoc

Notify maintainers

maintainer of neovim: @manveru, @rvolosatovs maintainer of nvim-treesitter: @figsoda

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.72, NixOS, 24.05 (Uakari), 24.05.20240119.bbe7d8f`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/nix/store/11zbgb8j7wnnccbbjcq0q556h28g7p4r-source`

---

Add a :+1: reaction to issues you find important.

[teto]

Because I run neovim nightly I am not sure about nixpkgs' status but I had this issue and I installed the treesitter grammars imperatively, via nvim-treesitter's :TSInstall. Make sure the nvim-treesitter's queries appear before neovim's.

The versioning of tree-sitter grammars is really not great but I hope we eventually improve that. The convergence of queries between helix and neovim is already a good first step.

[lucc]

@teto I think you can work around this issue declaratifely like hinted in the initial post: My interpretation of the problem is that neovim itself bundles three parsers: lua, c, vimdoc. But for some reasons these three parsers are not compatible with nvim-treesitter. If you install nvim-treesitter without a lua, c or vimdoc parser it will pick up the bundled one from the neovim derivation and spit out the errors. If you additionally install the respective parser with the nvim-treesitter.withPlugins function, it will find the additional parser (which seems to be compatible) before the bundled one.

[teto]

that's because nvim-treesitter brings its own queries that override neovim's ones. And queries are tied to a grammar. Newer queries with old grammars wreck havoc

[xbreak]

If you additionally install the respective parser with the nvim-treesitter.withPlugins function, it will find the additional parser (which seems to be compatible) before the bundled one.

This is my workaround as well.

[urob]

The cause of the issue and the fix are documented here: https://github.com/nvim-treesitter/nvim-treesitter/issues/3092:

Since 0.8.0, Neovim bundles parsers and queries for c, lua, vim, and help. If you use nvim-treesitter, you must make sure these parsers are installed via nvim-treesitter so that both parser and queries (which are always installed) are taken from nvim-treesitter. (It's important for this that your nvim-treesitter plugin directory comes before both /usr/local/share/nvim/* and /usr/lib/nvim in your runtimepath.)

TL;DR: at a minimum, one must install parsers for c, lua, vim, and help. As mentioned above, nvim-treesitter.withAllGrammars will do - albeit being somewhat overkill.

[nicolasdumitru]

Neovim user here. I find this issue to be particularly weird, as I have only had issues with the vimdoc parser, not with c or lua. For context: I install neovim from nixpkgs (no overlays), I configure it in .config/nvim and I install all parsers in my neovim configuration with ensure_installed = { ... } and update the with :TSUpdate. I ran :TSInstall vimdoc and it fixed the issue completely. Why does this only happen for that particular parser?

[teto]

because it was the only one not compatible ? or you haven't triggered an incompatibility in the other parsers yet

[nicolasdumitru]

@teto sure, but it's not just about the incompatibility. That was the only one that I had to manually/imperatively install by running :TSInstall, i.e. the only one that didn't get done properly by the Neovim config. What I meant to say is that it's (subjectively) hard to diagnose the issue—(1) you'd think that the config does what it's supposed to do, therefore why do it manually? and (2) it only fails for one of multiple parsers; every single other parser had been installed as intended. It's the kind of thing that can drive you insane for a couple hours if you don't know all these details, especially if you previously had a ~perfectly reliable config.

[tmillr]

idk if it's the same issue, but I recently updated my nixpkgs and just noticed today that several parsers were missing, which is strange to me because I'm simply using nvim-treesitter.withAllGrammars as one of my plugins and it was working fine for me before (i.e. I'd get all the parsers and they'd be reachable within nvim at runtime). I think that some of the recent changes may have caused it?

I've now spent 6+ hours trying to get to the root of it and it's driving me absolutely nuts lol. I wasn't able to figure out or fix the root cause, but I did learn a couple things. There seems to be some strange things going on and it's hard to debug. The bug seems to be occurring inside makeNeovimConfig and appears to be due to "conflicting" dependencies. When I remove all of my plugins except nvim-treesitter.withAllGrammars, it seems to work correctly, but with all of my plugins I end up with just the query.so parser instead? So, finally, I stumbled into overrides.nix and found out that that was the playground plugin. Sure enough, when I don't include playground in my plugins, I get all of my parsers back.

I don't think it's an issue on my end because I've had those 2 plugins included for awhile now and didn't seem to have this problem before. Is this the intended behavior? Is this a known issue?

I still don't know what's causing the issue, but I think it might be due to the use of lib.unique (edit: or buildEnv) in the vimUtils.packDir function? Maybe the comparison is done based on the package name (as opposed to all the attributes/fields in the case of a normal attrset). I think this essentially means that it is currently impossible to have 2 or more vim plugins which specify treesitter grammars as a dependency (although the issue could be more nuanced than this).

[!NOTE] I use a flake-based setup which follows nixpkgs-unstable. Neovim and all of my plugins are installed via nix, and I use the newer "unstable" wrapper. When I say that the parsers are missing, I mean that they are entirely missing from the pack dir and nvim fails to load them (e.g. for syntax highlighting) and emits a no parser error message.