nix highlighting + injection is slow

[Pegasust]

Describe the highlighting problem

I'm working on github:nix-community/poetry2nix/overrides/default.nix and it's absurdly slow until I :TSDisable highlight nix and resort to semantic highlighting (thanks oxalica). I'm pretty sure I have no other injections (not sure about highlighting) going on, and I can provide a profiling flamegraph that shows the slowness comes from highlight.lua and injection.

Example snippet that causes the problem

The whole github:nix-community/poetry2nix/overrides/default.nix is a sample, I haven't had an issue with other big Nix files (yet)

Tree-sitter parsing result

It's a 30k lines file and I couldn't figure out how to copy parsed language/injected language vtext, it's all Nix as far as I can manually audit overrides-ts.txt

Example screenshot

colorscheme: catppuccin-frappe

For :TSHighlightCapturesUnderCursor, I think this is not applicable - the highlights are fine, this report focuses on performance

Expected behavior

Should be responsive and optionally fast

Output of :checkhealth nvim-treesitter

nvim-treesitter: require("nvim-treesitter.health").check()

Installation ~
- OK `tree-sitter` found 0.20.8 (parser generator, only needed for :TSInstallFromGrammar)
- OK `node` found v18.18.2 (only needed for :TSInstallFromGrammar)
- OK `git` executable found.
- OK `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
  Version: xcode-select: note: No developer tools were found, requesting install.
- OK Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

OS Info:
{
  machine = "arm64",
  release = "23.1.0",
  sysname = "Darwin",
  version = "Darwin Kernel Version 23.1.0: Mon Oct  9 21:27:24 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T6000"
} ~

Parser/Features         H L F I J
  - ada                 ✓ ✓ ✓ . .
  - agda                ✓ . ✓ . .
  - apex                ✓ ✓ ✓ . .
  - arduino             ✓ ✓ ✓ ✓ ✓
  - astro               ✓ ✓ ✓ ✓ ✓
  - authzed             ✓ . . . ✓
  - awk                 ✓ . . . ✓
  - bash                ✓ ✓ ✓ . ✓
  - bass                ✓ ✓ ✓ ✓ ✓
  - beancount           ✓ . ✓ . .
  - bibtex              ✓ . ✓ ✓ .
  - bicep               ✓ ✓ ✓ ✓ ✓
  - bitbake             ✓ ✓ ✓ ✓ ✓
  - blueprint           ✓ . . . .
  - c                   ✓ ✓ ✓ ✓ ✓
  - c_sharp             ✓ ✓ ✓ . ✓
  - cairo               ✓ ✓ ✓ ✓ ✓
  - capnp               ✓ ✓ ✓ ✓ ✓
  - chatito             ✓ ✓ ✓ ✓ ✓
  - clojure             ✓ ✓ ✓ . ✓
  - cmake               ✓ . ✓ ✓ .
  - comment             ✓ . . . .
  - commonlisp          ✓ ✓ ✓ . .
  - cooklang            ✓ . . . .
  - corn                ✓ ✓ ✓ ✓ .
  - cpon                ✓ ✓ ✓ ✓ ✓
  - cpp                 ✓ ✓ ✓ ✓ ✓
  - css                 ✓ . ✓ ✓ ✓
  - csv                 ✓ . . . .
  - cuda                ✓ ✓ ✓ ✓ ✓
  - cue                 ✓ ✓ ✓ ✓ ✓
  - d                   ✓ . ✓ ✓ ✓
  - dart                ✓ ✓ ✓ ✓ ✓
  - devicetree          ✓ ✓ ✓ ✓ ✓
  - dhall               ✓ . ✓ . ✓
  - diff                ✓ . . . .
  - dockerfile          ✓ . . . ✓
  - dot                 ✓ . . . ✓
  - doxygen             ✓ . . ✓ ✓
  - dtd                 ✓ . ✓ . ✓
  - ebnf                ✓ . . . .
  - eds                 ✓ . ✓ . .
  - eex                 ✓ . . . ✓
  - elixir              ✓ ✓ ✓ ✓ ✓
  - elm                 ✓ . . . ✓
  - elsa                ✓ ✓ ✓ ✓ ✓
  - elvish              ✓ . . . ✓
  - embedded_template   ✓ . . . ✓
  - erlang              ✓ . ✓ . .
  - fennel              ✓ ✓ ✓ . ✓
  - firrtl              ✓ ✓ ✓ ✓ ✓
  - fish                ✓ ✓ ✓ ✓ ✓
  - foam                ✓ ✓ ✓ ✓ ✓
  - forth               ✓ ✓ ✓ ✓ ✓
  - fortran             ✓ . ✓ ✓ .
  - fsh                 ✓ . . . .
  - func                ✓ . . . .
  - fusion              ✓ ✓ ✓ ✓ .
  - gdscript            ✓ ✓ ✓ ✓ ✓
  - git_config          ✓ . . . .
  - git_rebase          ✓ . . . ✓
  - gitattributes       ✓ . . . ✓
  - gitcommit           ✓ . . . ✓
  - gitignore           ✓ . . . .
  - gleam               ✓ ✓ ✓ ✓ ✓
  - glimmer             ✓ ✓ ✓ ✓ .
  - glsl                ✓ ✓ ✓ ✓ ✓
  - gn                  ✓ ✓ ✓ ✓ ✓
  - go                  ✓ ✓ ✓ ✓ ✓
  - godot_resource      ✓ ✓ ✓ . .
  - gomod               ✓ . . . ✓
  - gosum               ✓ . . . .
  - gowork              ✓ . . . ✓
  - gpg                 ✓ . . . ✓
  - graphql             ✓ . . ✓ ✓
  - groovy              ✓ . . . ✓
  - gstlaunch           ✓ . . . .
  - hack                ✓ . . . .
  - hare                ✓ ✓ ✓ ✓ ✓
  - haskell             ✓ . ✓ . ✓
  - haskell_persistent  ✓ . ✓ . .
  - hcl                 ✓ . ✓ ✓ ✓
  - heex                ✓ ✓ ✓ ✓ ✓
  - hjson               ✓ ✓ ✓ ✓ ✓
  - hlsl                ✓ ✓ ✓ ✓ ✓
  - hocon               ✓ . . . ✓
  - hoon                ✓ ✓ ✓ . .
  - html                ✓ ✓ ✓ ✓ ✓
  - htmldjango          ✓ . ✓ ✓ ✓
  - http                ✓ . . . ✓
  - hurl                ✓ . ✓ ✓ ✓
  - ini                 ✓ . ✓ . .
  - ispc                ✓ ✓ ✓ ✓ ✓
  - janet_simple        ✓ ✓ ✓ . ✓
  - java                ✓ ✓ ✓ ✓ ✓
  - javascript          ✓ ✓ ✓ ✓ ✓
  - jq                  ✓ . . . ✓
  - jsdoc               ✓ . . . .
  - json                ✓ ✓ ✓ ✓ .
  - json5               ✓ . . . ✓
  - jsonc               ✓ ✓ ✓ ✓ ✓
  - jsonnet             ✓ ✓ ✓ . .
  - julia               ✓ ✓ ✓ ✓ ✓
  - kconfig             ✓ ✓ ✓ ✓ ✓
  - kdl                 ✓ ✓ ✓ ✓ ✓
  - kotlin              ✓ ✓ ✓ . ✓
  - lalrpop             ✓ ✓ . . ✓
  - latex               ✓ . ✓ . ✓
  - ledger              ✓ . ✓ ✓ ✓
  - leo                 ✓ . . . ✓
  - liquidsoap          ✓ ✓ ✓ ✓ .
  - llvm                ✓ . . . .
  - lua                 ✓ ✓ ✓ ✓ ✓
  - luadoc              ✓ . . . .
  - luap                ✓ . . . .
  - luau                ✓ ✓ ✓ ✓ ✓
  - m68k                ✓ ✓ ✓ . ✓
  - make                ✓ . ✓ . ✓
  - markdown            ✓ . ✓ ✓ ✓
  - markdown_inline     ✓ . . . ✓
  - matlab              ✓ ✓ ✓ ✓ ✓
  - menhir              ✓ . . . ✓
  - mermaid             ✓ . . . .
  - meson               ✓ . ✓ . ✓
  - mlir                ✓ ✓ . . .
  - nasm                ✓ . . . ✓
  - nickel              ✓ . . ✓ .
  - ninja               ✓ . ✓ ✓ .
  - nix                 ✓ ✓ ✓ . ✓
  - norg                . . . . .
  - nqc                 ✓ ✓ ✓ ✓ ✓
  - objc                ✓ ✓ ✓ ✓ ✓
  - objdump             ✓ . . . ✓
  - ocaml               ✓ ✓ ✓ ✓ ✓
  - ocaml_interface     ✓ ✓ ✓ ✓ ✓
  - ocamllex            ✓ . . . ✓
  - odin                ✓ ✓ ✓ ✓ ✓
  - org                 . . . . .
  - pascal              ✓ ✓ ✓ ✓ ✓
  - passwd              ✓ . . . .
  - pem                 ✓ . ✓ . ✓
  - perl                ✓ . ✓ . ✓
  - php                 ✓ ✓ ✓ ✓ ✓
  - phpdoc              ✓ . . . .
  - pioasm              ✓ . . . ✓
  - po                  ✓ . ✓ . ✓
  - pod                 ✓ . . . .
  - poe_filter          ✓ . ✓ ✓ ✓
  - pony                ✓ ✓ ✓ ✓ ✓
  - prisma              ✓ . ✓ . .
  - promql              ✓ . . . ✓
  - proto               ✓ . ✓ . .
  - prql                ✓ . . . ✓
  - psv                 ✓ . . . .
  - pug                 ✓ . . . ✓
  - puppet              ✓ ✓ ✓ ✓ ✓
  - pymanifest          ✓ . . . ✓
  - python              ✓ ✓ ✓ ✓ ✓
  - ql                  ✓ ✓ ✓ ✓ ✓
  - qmldir              ✓ . . . ✓
  - qmljs               ✓ . ✓ . .
  - query               ✓ ✓ ✓ ✓ ✓
  - r                   ✓ ✓ . ✓ ✓
  - racket              ✓ . ✓ . ✓
  - rasi                ✓ ✓ ✓ ✓ .
  - re2c                ✓ ✓ ✓ ✓ ✓
  - regex               ✓ . . . .
  - rego                ✓ . . . ✓
  - requirements        ✓ . . . ✓
  - rnoweb              ✓ . ✓ . ✓
  - robot               ✓ . ✓ ✓ .
  - ron                 ✓ ✓ ✓ ✓ ✓
  - rst                 ✓ ✓ . . ✓
  - ruby                ✓ ✓ ✓ ✓ ✓
  - rust                ✓ ✓ ✓ ✓ ✓
  - scala               ✓ ✓ ✓ . ✓
  - scfg                ✓ . . . ✓
  - scheme              ✓ . ✓ . ✓
  - scss                ✓ . ✓ ✓ .
  - slint               ✓ . . ✓ .
  - smali               ✓ ✓ ✓ ✓ ✓
  - smithy              ✓ . . . .
  - snakemake           ✓ ✓ ✓ . ✓
  - solidity            ✓ . ✓ . .
  - soql                ✓ . . . .
  - sosl                ✓ . . . .
  - sparql              ✓ ✓ ✓ ✓ ✓
  - sql                 ✓ . . ✓ ✓
  - squirrel            ✓ ✓ ✓ ✓ ✓
  - ssh_config          ✓ . ✓ ✓ ✓
  - starlark            ✓ ✓ ✓ ✓ ✓
  - strace              ✓ . . . ✓
  - supercollider       ✓ ✓ ✓ ✓ ✓
  - surface             ✓ . ✓ ✓ ✓
  - svelte              ✓ . ✓ ✓ ✓
  - swift               ✓ ✓ . ✓ .
  - sxhkdrc             ✓ . ✓ . ✓
  - systemtap           ✓ ✓ ✓ . ✓
  - t32                 ✓ ✓ ✓ ✓ ✓
  - tablegen            ✓ ✓ ✓ ✓ ✓
  - teal                ✓ ✓ ✓ ✓ ✓
  - terraform           ✓ . ✓ ✓ ✓
  - textproto           ✓ . ✓ ✓ .
  - thrift              ✓ ✓ ✓ ✓ ✓
  - tiger               ✓ ✓ ✓ ✓ ✓
  - tlaplus             ✓ ✓ ✓ . ✓
  - todotxt             ✓ . . . .
  - toml                ✓ ✓ ✓ ✓ ✓
  - tsv                 ✓ . . . .
  - tsx                 ✓ ✓ ✓ ✓ ✓
  - turtle              ✓ ✓ ✓ ✓ ✓
  - twig                ✓ . . . ✓
  - typescript          ✓ ✓ ✓ ✓ ✓
  - ungrammar           ✓ ✓ ✓ ✓ ✓
  - unison              ✓ . . . ✓
  - usd                 ✓ ✓ ✓ ✓ .
  - uxntal              ✓ ✓ ✓ ✓ ✓
  - v                   ✓ ✓ ✓ ✓ ✓
  - vala                ✓ . ✓ . .
  - verilog             ✓ ✓ ✓ . ✓
  - vhs                 ✓ . . . .
  - vim                 ✓ ✓ ✓ . ✓
  - vimdoc              ✓ . . . ✓
  - vue                 ✓ . ✓ ✓ ✓
  - wgsl                ✓ . ✓ ✓ .
  - wgsl_bevy           ✓ . ✓ ✓ .
  - wing                ✓ ✓ ✓ . .
  - xml                 ✓ . ✓ ✓ ✓
  - yaml                ✓ ✓ ✓ ✓ ✓
  - yang                ✓ . ✓ ✓ ✓
  - yuck                ✓ ✓ ✓ ✓ ✓
  - zig                 ✓ . ✓ ✓ ✓

  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections
         +) multiple parsers found, only one will be used
         x) errors found in the query, try to run :TSUpdate {lang} ~

### Output of `nvim --version`

```text
% ❯ nvim --version                    
NVIM v0.10.0-dev-d2e7cc6
Build type: Release
LuaJIT 2.1.1693350652
Run "nvim -V1 -v" for more info

Additional context

[clason]

Please share the file. And yes, the Nix injections are staggeringly expensive; there's little that can be done here without regressing on functionality until async parsing/timeouts are implemented in Neovim core.

[lucario387]

It's a 4k LOC file ... It should be expected to have some slowdown. But not too slow for me though. I do have bash/python parser injected, only comment is not. So this once again turns into a comment parser slowing down stuffs

[clason]

We should consider just removing that parser (or the default injections).

[Pegasust]

Apologies for not including the file.

https://github.com/nix-community/poetry2nix/blob/3ed06a00cb3cf5ddc16f6776d036658a73a61eba/overrides/default.nix

For me (again, this might be an aarch64-darwin thing), typing a word takes a good 5 seconds so that was my main concern. Maybe a nvim slimming session is needed to see if there was anything in between that could have made this a lot worse.