Performance drop in huge files (over 1200 LOC)

[hinell]

Problem

Slow in files with over 1200+ LOCs.

Here is a log of called functions obtained by jit.p. Numbers indicate times the given function is called:

autocmds.lua:25;debounced_refresh;debounced_refresh;refresh;get 504
highlighter.lua:292 461
autocmds.lua:25;debounced_refresh;debounced_refresh;refresh 432
highlighter.lua:292;on_line_impl;for_each_tree;fn;iter 362
autocmds.lua:25;debounced_refresh;debounced_refresh;refresh;get;is_indent 356
highlighter.lua:292;on_line_impl;for_each_tree;fn;iter;match_preds;handler 330
autocmds.lua:25;debounced_refresh;debounced_refresh;refresh;get;tbl_filter 307
highlighter.lua:292;on_line_impl;for_each_tree;fn 302
autocmds.lua:39;debounced_refresh;debounced_refresh;refresh;get 268
autocmds.lua:39;debounced_refresh;debounced_refresh;refresh 215
autocmds.lua:39;debounced_refresh;debounced_refresh;refresh;get;is_indent 215
autocmds.lua:25;debounced_refresh;debounced_refresh;refresh;get;tbl_contains 213
treesitter-context.lua:92;get_context;get_context_query;pcall;_memoize.lua:51;fn;fn 198
init.lua:19 193

debounced_refresh() is called too often.
The issue may be partially mitigated by increasing config.debounce = 768 but it isn't perfect.

Steps to reproduce

1. Config:

   
   -- $HOME/.config/nvim/lua/user/plugins/indent-blankline.lua
   local config = require("ibl.config").default_config
   config.debounce        = 200

require("ibl").setup(config)

3. Open the following file:
```bash
#!/usr/bin/bash
nvim -c "set ft=c" <(curl -L https://github.com/tree-sitter/tree-sitter-bash/raw/master/src/scanner.c)

Expected behavior

Slick & fast

Neovim version (nvim -v)

NVIM v0.10.0-dev-8405649f9

[lukas-reineke]

The only thing that gets slower with bigger files is scope, because of treesitter. You can turn it off in big files, or use debounce to reduce the number of times indent-blankline refreshes. But there isn't really anything else I can do.

[qRoC]

@lukas-reineke the big overhead in this plugin from vim.tbl_* functions (vim.validate call).

Current:

With:

vim.validate = function() end

[lukas-reineke]

Thanks! I didn't think about the internal validation of the vim.tbl_ functions. I made a PR to fix this now. I don't think this was the cause for OPs issues, because it wouldn't scale with buffer size, but any performance improvements are nice.

[hinell]

@qRoC The vim.tbl_* aren't the root cause. It's numer of times they are called with big tables.

[hinell]

@lukas-reineke Great work in https://github.com/lukas-reineke/indent-blankline.nvim/commit/9a1fe189a5b6a63e89717424c4995e09feae36d7

I've updated and profiled it once again. With debounce=512 set it gives much better results. Default debounce is still laggy in big files though.

ibl.512.profile

        
13%  fn
  <- 69%  for_each_tree < on_line_impl < highlighter.lua:292
  <- 20%  fn < _memoize.lua:51 < pcall < get_context_query < get_context < treesitter-context.lua:92
  <-  7%  fn < _memoize.lua:51 < pcall < get_context_query < get_context < f < treesitter-context.lua:21
  <-  2%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:39
  <-  1%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
  <-  0%   < _editor.lua:0
11%  highlighter.lua:292
 9%  iter
  <- 100%  fn < for_each_tree < on_line_impl < highlighter.lua:292
 9%  handler
  <- 98%  match_preds < iter < fn < for_each_tree < on_line_impl < highlighter.lua:292
  <-  1%  apply_directives < iter < fn < for_each_tree < on_line_impl < highlighter.lua:292
  <-  1%  match_preds < (for generator) < fn < for_each_tree < highlight_contexts < open < open < f < treesitter-context.lua:21
 6%  get
  <- 61%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
  <- 21%   < _editor.lua:0
  <- 17%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:39
  <-  1%  bufferline.lua:53
 5%  tbl_contains
  <- 80%  handler < match_preds < iter < fn < for_each_tree < on_line_impl < highlighter.lua:292
  <-  9%  get < refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
  <-  6%  get <  < _editor.lua:0
  <-  2%  refresh < lualine.lua:355
  <-  1%  set < bufferline.lua:53
  <-  1%  get_extension_sections < lualine.lua:282 < nvim_win_call < refresh < lualine.lua:355
  <-  1%  tbl_intersect < init.lua:19
        
        

ibl.default.profile

        

13%  get
  <- 69%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
  <- 24%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:39
  <-  5%   < _editor.lua:0
  <-  1%  sources.lua:46 < get_diagnostics < update_status < draw < draw_section < lualine.lua:163 < pcall < statusline < lualine.lua:282
  <-  1%  update_callargs < statuscol.lua:260
  <-  0%  bufferline.lua:53
  <-  0%  diagnostics.lua:70 < get < get_diagnostics < get_components < bufferline.lua:53
10%  fn
  <- 59%  for_each_tree < on_line_impl < highlighter.lua:292
  <- 28%  fn < _memoize.lua:51 < pcall < get_context_query < get_context < treesitter-context.lua:92
  <-  8%  fn < _memoize.lua:51 < pcall < get_context_query < get_context < f < treesitter-context.lua:21
  <-  4%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
  <-  1%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:39
10%  refresh
  <- 80%  debounced_refresh < debounced_refresh < autocmds.lua:25
  <- 19%  debounced_refresh < debounced_refresh < autocmds.lua:39
  <-  0%  lualine.lua:355
 9%  handler
  <- 99%  match_preds < iter < fn < for_each_tree < on_line_impl < highlighter.lua:292
  <-  1%  match_preds < (for generator) < fn < for_each_tree < highlight_contexts < open < open < treesitter-context.lua:92
 9%  highlighter.lua:292
 6%  iter
  <- 100%  fn < for_each_tree < on_line_impl < highlighter.lua:292
 5%  tbl_contains
  <- 72%  handler < match_preds < iter < fn < for_each_tree < on_line_impl < highlighter.lua:292
  <- 17%  get < refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
  <-  6%  get < refresh < debounced_refresh < debounced_refresh < autocmds.lua:39
  <-  2%  get <  < _editor.lua:0
  <-  1%  tbl_intersect < init.lua:19
  <-  1%  handler < match_preds < (for generator) < fn < for_each_tree < highlight_contexts < open < open < f
  <-  1%  get_extension_sections < lualine.lua:282 < nvim_win_call < refresh < lualine.lua:355
  <-  1%  keymap.lua:44 < to_keymap < listen < prepare < callback < async.lua:163
 5%  is_indent
  <- 75%  get < refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
  <- 21%  get < refresh < debounced_refresh < debounced_refresh < autocmds.lua:39
  <-  3%  get <  < _editor.lua:0
  <-  1%  refresh < debounced_refresh < debounced_refresh < autocmds.lua:25
        
        

[Danielkonge]

@hinell I was looking at the performance of ibl and #795 helped a bit for me since it allows me to set much lower values for min and max in viewport_buffer and still get equally good highlighting (I decreased it to 5 and 10 for my own setup, but 30 and 60 or so might be better if you have a high debounce).

I think part of the problem with large files could be that you often have long indentation guides so large values in viewport_buffer might mean that you go all the way to the current viewport_buffer.max of 500, which is a lot of extra calculation. In smaller files you very rarely can even go 500 lines up, but in large files you can easily do that if you are under line 500.

Have you tried setting much lower values in viewport_buffer and profile it?

Note: This didn't fully fix my problem though, and keeping down j or k can still result in some lag, but subjectively I feel that it did help (I haven't profiled it).

[hinell]

@Danielkonge Thanks. I've narrowed down the issue and made a PR #790 that solved 99% perf problems with huge files but it was turned down by maintainer. I see no reason to use this plugin anymore. It's too unoptimized and have a lot weak points. I've switched to hlchunk instead. Best.