Markdown files with more than 500 lines become perceptibly slow

[medwatt]

Describe the bug

I noticed recently when editing a large markdown file that has more 1000 lines that the delay between keystrokes while typing becomes perceptible. This only happens when treesitter for markdown is enabled.

Here's a video demonstrating the difference in the typing experience between an empty file and a large file.

https://user-images.githubusercontent.com/17733465/167254657-808a0f73-219f-4531-b878-d4ea5d06c4d7.mp4

To Reproduce

1. Create a large markdown file with many code blocks (how large the file is would probably depend on your computer)
2. Start typing while inside a code block

Expected behavior

It is expected that there should be no lag when typing irrespective of the number of lines in the file.

Output of :checkhealth nvim-treesitter

nvim-treesitter: require("nvim-treesitter.health").check()
========================================================================
## Installation
  - OK: `tree-sitter` found 0.20.6 (parser generator, only needed for :TSInstallFromGrammar)
  - OK: `node` found v17.8.0 (only needed for :TSInstallFromGrammar)
  - OK: `git` executable found.
  - OK: `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
    Version: cc (GCC) 11.2.0
  - OK: Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

## Parser/Features H L F I J
  - hack           ✓ . . . . 
  - norg           . . . . . 
  - svelte         ✓ . ✓ ✓ ✓ 
  - astro          ✓ ✓ ✓ ✓ ✓ 
  - bash           ✓ ✓ ✓ . ✓ 
  - beancount      ✓ . ✓ . . 
  - lalrpop        ✓ ✓ . . . 
  - php            ✓ ✓ ✓ ✓ ✓ 
  - swift          ✓ ✓ . . . 
  - markdown       ✓ . ✓ . ✓ 
  - cooklang       ✓ . . . . 
  - tlaplus        ✓ ✓ ✓ . ✓ 
  - fish           ✓ ✓ ✓ ✓ ✓ 
  - toml           ✓ ✓ ✓ ✓ ✓ 
  - wgsl           ✓ . ✓ . . 
  - proto          ✓ . ✓ . . 
  - m68k           ✓ ✓ ✓ . ✓ 
  - pug            ✓ . . . ✓ 
  - elvish         ✓ . . . ✓ 
  - solidity       ✓ . . . . 
  - glsl           ✓ ✓ ✓ ✓ ✓ 
  - tsx            ✓ ✓ ✓ ✓ ✓ 
  - regex          ✓ . . . . 
  - turtle         ✓ ✓ ✓ ✓ ✓ 
  - go             ✓ ✓ ✓ ✓ ✓ 
  - html           ✓ ✓ ✓ ✓ ✓ 
  - yang           ✓ . ✓ ✓ . 
  - graphql        ✓ . . ✓ ✓ 
  - d              ✓ . ✓ ✓ ✓ 
  - vue            ✓ . ✓ ✓ ✓ 
  - r              ✓ ✓ . ✓ ✓ 
  - make           ✓ . . . ✓ 
  - http           ✓ . . . ✓ 
  - prisma         ✓ . . . . 
  - query          ✓ ✓ ✓ ✓ ✓ 
  - java           ✓ ✓ . ✓ ✓ 
  - cmake          ✓ . ✓ . . 
  - llvm           ✓ . . . . 
  - ruby           ✓ ✓ ✓ ✓ ✓ 
  - css            ✓ . ✓ ✓ ✓ 
  - perl           ✓ . ✓ . . 
  - pioasm         ✓ . . . ✓ 
  - json5          ✓ . . . ✓ 
  - julia          ✓ ✓ ✓ ✓ ✓ 
  - pascal         ✓ ✓ ✓ ✓ ✓ 
  - vim            ✓ ✓ . . ✓ 
  - json           ✓ ✓ ✓ ✓ . 
  - cpp            ✓ ✓ ✓ ✓ ✓ 
  - slint          ✓ . . ✓ . 
  - zig            ✓ . ✓ ✓ ✓ 
  - bibtex         ✓ . ✓ ✓ . 
  - gowork         ✓ . . . ✓ 
  - yaml           ✓ ✓ ✓ ✓ ✓ 
  - jsdoc          ✓ . . . . 
  - hcl            ✓ . ✓ ✓ ✓ 
  - heex           ✓ ✓ ✓ ✓ ✓ 
  - glimmer        ✓ . . . . 
  - sparql         ✓ ✓ ✓ ✓ ✓ 
  - dot            ✓ . . . ✓ 
  - latex          ✓ . ✓ . ✓ 
  - gdscript       ✓ ✓ . ✓ ✓ 
  - devicetree     ✓ ✓ ✓ ✓ ✓ 
  - lua            ✓ ✓ ✓ ✓ ✓ 
  - foam           ✓ ✓ ✓ ✓ ✓ 
  - godot_resource ✓ ✓ ✓ . . 
  - scheme         ✓ . ✓ . ✓ 
  - clojure        ✓ ✓ ✓ . ✓ 
  - gomod          ✓ . . . ✓ 
  - comment        ✓ . . . . 
  - elixir         ✓ ✓ ✓ ✓ ✓ 
  - phpdoc         ✓ . . . . 
  - erlang         . . . . . 
  - verilog        ✓ ✓ ✓ . ✓ 
  - rego           ✓ . . . ✓ 
  - dockerfile     ✓ . . . ✓ 
  - fortran        ✓ . ✓ ✓ . 
  - jsonc          ✓ ✓ ✓ ✓ ✓ 
  - haskell        ✓ . . . ✓ 
  - embedded_template✓ . . . ✓ 
  - javascript     ✓ ✓ ✓ ✓ ✓ 
  - fennel         ✓ ✓ . . ✓ 
  - gleam          ✓ ✓ ✓ ✓ ✓ 
  - commonlisp     ✓ ✓ ✓ . . 
  - kotlin         ✓ ✓ ✓ . ✓ 
  - rst            ✓ ✓ . . ✓ 
  - dart           ✓ ✓ . ✓ ✓ 
  - ocaml          ✓ ✓ ✓ . ✓ 
  - cuda           ✓ ✓ ✓ ✓ ✓ 
  - nix            ✓ ✓ ✓ . ✓ 
  - ninja          ✓ . ✓ ✓ . 
  - help           ✓ . . . . 
  - ocaml_interface✓ ✓ ✓ . ✓ 
  - rust           ✓ ✓ ✓ ✓ ✓ 
  - org            . . . . . 
  - ocamllex       ✓ . . . ✓ 
  - typescript     ✓ ✓ ✓ ✓ ✓ 
  - ql             ✓ ✓ . ✓ ✓ 
  - hjson          ✓ ✓ ✓ ✓ ✓ 
  - scala          ✓ . ✓ . ✓ 
  - fusion         ✓ ✓ ✓ ✓ . 
  - hocon          ✓ . . . ✓ 
  - scss           ✓ . . ✓ . 
  - todotxt        ✓ . . . . 
  - eex            ✓ . . . ✓ 
  - c              ✓ ✓ ✓ ✓ ✓ 
  - python         ✓ ✓ ✓ ✓ ✓ 
  - ledger         ✓ . ✓ ✓ ✓ 
  - vala           ✓ . . . . 
  - surface        ✓ . ✓ ✓ ✓ 
  - elm            ✓ . . . ✓ 
  - supercollider  ✓ ✓ ✓ ✓ ✓ 
  - rasi           ✓ ✓ ✓ ✓ . 
  - c_sharp        ✓ ✓ ✓ . ✓ 
  - teal           ✓ ✓ ✓ ✓ ✓ 

  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections
         +) multiple parsers found, only one will be used
         x) errors found in the query, try to run :TSUpdate {lang}

Output of nvim --version

NVIM v0.7.0
Build type: Release
LuaJIT 2.1.0-beta3
Compiled by builduser

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info

Additional context

No response

[lewis6991]

There's a lot of activity in this thread.

@MDeiml are you able to provide a status update? Especially now https://github.com/neovim/neovim/pull/22309 is merged.

[MDeiml]

Sure!

The split parser I mentioned is implemented and used in neovim for quite some while. This helped a lot in other editors, bur neovim had some problem with injected languages which was improved in https://github.com/neovim/neovim/pull/22309 so I imagine that this helped quite a bit. Since then I improved the "block" parser out of the two to almost not use "conflicts". Conflicts are still what makes the "inline" parser quite slow.

[clason]

Note that we also recently bumped the tree-sitter version in our official builds to include the vastly improved error handling; I suspect that this also helps quite a bit in this regard.

So a test on the latest nightly with one of the bad example files would be very much appreciated.

[Feel-ix-343]

Why does Treesiter cause delays in input latency if it is supposed to be async?

[clason]

It's not async; what made you think that?

[Feel-ix-343]

(I realize this is marked off topic, sorry abt that ig)

Hm. I have been searching around for a while and saw posts from several years ago referencing it as a new async highlighting features specifically for neovim, to improve on vim's synchronous and slow highlighting.

I guess I was wrong! My next question would be why isn't it async ?!