Weird slowdown after compilation on complex projects

[griffinwxk]

Description

This is a weird issue that I did some digging myself but with little success so it might take a while. Also this issue probably won't appear for light LaTeX users so I apologize ahead for not able to produce an MWE at this time but am happy to assist with debugging over time.

So I'm a pretty heavy user of LaTeX and I put all my common settings and commands into a .sty file, about 800+ lines in total. I have this LaTeX document of about 1500+ lines (about the length of a section or a chapter in any of my typical projects, so dividing the file is usually not an option) with lots of math. Here's the list of weird behaviors:

• If I open the file on Neovim without compiling it (or delete all the compilations), everything is fine.
• After compiling it, if the cursor is on some early lines (like less than 1100 or something), things are fine too.
• Then if I jump to some late part of the file (say with G command), the UI suddenly starts to crawl.
• If I jump back to early part of the file (say with gg), things revert back to normal.
• If I manually move up the cursor from, say line 1200 back to some 1100, things also suddenly revert back to normal (so there's a "breaking point" of some sorts past certain line number or file size).
• Moreover, if in the late part of the file I press C-d to scroll down (or some other command to go down, like pressing j enough times), it will also revert the UI back to being responsive again, but if I then move cursor up with C-u or just k enough times (going past the point I started using C-d/j/etc.), UI starts to crawl again until it moves back to the "breaking point" of about line 1100.
• I tried to move stuff from my .sty file into the document instead, and it DOES fix the problem but I use my "personal package" for all my LaTeX project and I don't want to just copy-paste for every new project.
• I tried removing the VimTeX highlighting, and (IIRC) it DOES fix the problem but I don't want to give it up because it's so good.
• I tried the same document with all math removed, and everything is good. So definitely math and syntax highlighting related.
• I tried turning conceal off, no good.

I'm reasonably confident it's related to some interaction between VimTeX syntax (about math) and custom packages, so here are the main things I did with syntax:

• I use conceal on and conceallevel=1, and I defined some new conceals based on my own LaTeX commands. But conceal doesn't seem to be the culprit as I noted above.
• I use kanagawa theme and used their color palette to customize some of the VimTeX highlight groups (but not all)

Finally, I don't believe the said behavior was there early last year. I noticed it between late 2022 and like Feb 2023 or something but I cannot pinpoint the time and a lot has happened in Neovim landscape during the time.

I don't expect this to be immediately solvable so I will keep monitoring this issue and happy to discuss and provide more details. Thanks! (And thanks for the awesome plugin as always!)

Steps to reproduce

Currently I cannot generate a reproducible example without posting some of my personal stuff due to the nature of the issue (being related to some complex customization), but am happy to help with testing potential solutions.

Expected behavior

UI should always be responsive

Actual behavior

UI has weird slowness described in the description.

Do you use a latexmkrc file?

Yes with only aux_dir set to "build"

VimtexInfo

System info:
  OS: macOS 12.6 (21G115)
  Vim version: NVIM v0.9.0
  Has clientserver: true
  Servername: /var/folders/y0/rlskvs3n0tb77fjc8v33zqcc0000gn/T/nvim.(my username)/DsJu9w/nvim.65346.0

VimTeX project: test
  base: test.tex
  root: /Users/(my username)/workbench/LaTeXtmp/mwe
  tex: /Users/(my username)/workbench/LaTeXtmp/mwe/test.tex
  main parser: current file verified
  document class: amsart
  packages: amsbsy amsfonts amsgen amsmath amsopn amsrefs amssymb amstext atbegshi-ltx atveryend-ltx autonum auxhook bigintcalc bitset calc cleveref (my personal package here) ctablestack environ epstopdf-base etex etexcmds etextools etoolbox expl3 fix-cm fontenc fontspec fontspec-luatex geometry gettitlestring graphics graphicx hycolor hyperref ifluatex ifoption iftex ifvtex import infwarerr intcalc keyval kvdefinekeys kvoptions kvsetkeys l3keys2e letltxmacro ltxcmds luacode lualatex-math luatexbase mathscinet mathtools mhsetup nameref pcatcode pdfescape pdftexcmds pgf pgfcomp-version-0-65 pgfcomp-version-1-18 pgfcore pgffor pgfkeys pgfmath pgfrcs pgfsys refcount rerunfilecheck rkeyval stmaryrd stringenc tcolorbox textcmds textpos tikz tikz-cd tikzfill-common tikzfill.image trig trimspaces unicode-math unicode-math-luatex uniquecounter url verbatim xcolor xparse
  compiler: latexmk
    engine: -lualatex
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    build_dir: ./build
    callback: 1
    continuous: 1
    executable: latexmk
    job: 
      jobid: 5
      output: /var/folders/y0/rlskvs3n0tb77fjc8v33zqcc0000gn/T/nvim.(my username)/DsJu9w/0
      cmd: max_print_line=2000 latexmk -verbose -file-line-error -synctex=1 -interaction=nonstopmode -lualatex -outdir=./build -pvc -view=none -e '$compiling_cmd = ($compiling_cmd ? $compiling_cmd . " ; " : "") . "echo vimtex_compiler_callback_compiling"' -e '$success_cmd = ($success_cmd ? $success_cmd . " ; " : "") . "echo vimtex_compiler_callback_success"' -e '$failure_cmd = ($failure_cmd ? $failure_cmd . " ; " : "") . "echo vimtex_compiler_callback_failure"' 'test.tex'
      pid: 65616
  viewer: Skim
  qf method: LaTeX logfile

[lervag]

So I'm a pretty heavy user of LaTeX and I put all my common settings and commands into a .sty file, about 800+ lines in total.

That seems useful as long as you're the sole author, but, for the record, I would strongly advice not to do that for collaborative projects.

I have this LaTeX document of about 1500+ lines (about the length of a section or a chapter in any of my typical projects, so dividing the file is usually not an option) with lots of math.

Ok, so, to reproduce, I would have to have your .sty file and the ~1500 line .tex file?

• If I manually move up the cursor from, say line 1200 back to some 1100, things also suddenly revert back to normal (so there's a "breaking point" of some sorts past certain line number or file size).

That's strange, but also interesting and useful.

• I tried to move stuff from my .sty file into the document instead, and it DOES fix the problem but I use my "personal package" for all my LaTeX project and I don't want to just copy-paste for every new project.

Also strange and interesting.

• I tried removing the VimTeX highlighting, and (IIRC) it DOES fix the problem but I don't want to give it up because it's so good.

How did you "remove" the VimTeX highlighting? Did you use let g:vimtex_syntax_enabled = 0?

Steps to reproduce

Currently I cannot generate a reproducible example without posting some of my personal stuff due to the nature of the issue (being related to some complex customization), but am happy to help with testing potential solutions.

Did you try to reproduce with a minimal init.vim? That's actually more important than a minimal test.tex file. If not, can you try with something like this (obviously with a correct runtimepath!):

" test.vim
set nocompatible
set runtimepath^=~/.local/plugged/vimtex
set runtimepath+=~/.local/plugged/vimtex/after
filetype plugin indent on
syntax enable

Save the above to test.vim where you have your LaTeX file, then open neovim with nvim --clean -u test.vim FILE.tex. Can you still reproduce your problems?

---

If you are comfortable with it, you could share your files with me personally (e.g. by email or similar). It would be much easier for me to debug this if I can reproduce the problem on my side.

[griffinwxk]

Thanks for the quick and detailed reply!

Ok, so, to reproduce, I would have to have your .sty file and the ~1500 line .tex file?

I was hoping to generate a large enough lorem ipsum stuff to reproduce the problem, but I might end up just sending you the files if nothing else would work

How did you "remove" the VimTeX highlighting? Did you use let g:vimtex_syntax_enabled = 0?

Yes

Did you try to reproduce with a minimal init.vim? That's actually more important than a minimal test.tex file. If not, can you try with something like this (obviously with a correct runtimepath!):

I just tested it just now. The problem is still there. The interesting thing is that the "breaking point" I mentioned is still exactly the same line number (1105 good, 1106 bad). So it rules out all the fancy eye candy plugins or lsp or treesitter (I did disable treesitter for tex filetype but good to know there's nothing hidden).

Also I tried your minimal vimrc together with g:vimtex_syntax_enabled = 0 and the problem disappears.

I will try to see if I can reproduce this with some lorem ipsum file by the end of next day. If I couldn't find time then I will send you my files through email or something. Thank you for the patience!

Edit: in my field when people do collaborations they seem just to copy paste a bunch of macros with little structure. I'm not very educated in terms of writing LaTeX packages though so it could be something ungodly I did when I wrote my "package". I also tried copy-pasting codes from the .sty file to main file (save some iftex and kvoption stuff) and the problem disappears too.

[lervag]

Thanks for the quick and detailed reply!

No problem :)

I was hoping to generate a large enough lorem ipsum stuff to reproduce the problem, but I might end up just sending you the files if nothing else would work

Please feel free to do so.

Did you try to reproduce with a minimal init.vim? That's actually more important than a minimal test.tex file. If not, can you try with something like this (obviously with a correct runtimepath!):

I just tested it just now. The problem is still there. …

Good! And thanks for the additional pointers. I'll await further updates from you, then.

Thank you for the patience!

No problem for me; and likewise from my side if I should not be able to respond fast.

Edit: in my field when people do collaborations they seem just to copy paste a bunch of macros with little structure.

I usually keep a relatively small preamble and avoid making a lot of extra customizations, and I don't mind copying whatever I need. But as long as it works there are no problems, right. :)

[griffinwxk]

A quick question as I'm trying to make a MWE: do you mind testing under the assumption of xelatax/lualatex and unicode-math?

Edit: actually, I found the culprit package: I use luacode package for some custom font kerning and removing the package makes the problem disappear. Therefore the problem is lualatex only. However, it's still dependent on the document itself. If the doc is only lorem ipsum pure text then there's no problem. So I'm trying to trim down my document to reproduce this right now. Maybe in the end just turning off certain feature for luacode package would do.

[griffinwxk]

I've sent the .sty and .tex files to your displayed email on github. Feel free to take your time with them. I don't need to write a lot in the near future. Thanks!

[lervag]

A quick question as I'm trying to make a MWE: do you mind testing under the assumption of xelatax/lualatex and unicode-math?

I don't mind testing under any assumption, as long as I can reproduce it on my end :)

I've sent the .sty and .tex files to your displayed email on github. Feel free to take your time with them. I don't need to write a lot in the near future. Thanks!

Sounds good!

[lervag]

Can you check if adding let g:vimtex_matchparen_enabled = 0 resolves the lagging on your end?

[griffinwxk]

Unfortunately it doesn't

[lervag]

That's strange, because that is the only cause of lags I'm observing on my end. Can you give a detailed explanation of how I would reproduce the problem from an clean directory with the two files you sent me?

[lervag]

So, to be clear, I'm trying to reproduce like this:

1. Copy the files you've shared into a test folder: test/

2. Create test.vim like this:

   set nocompatible
   set runtimepath^=~/.local/plugged/vimtex
   set runtimepath+=~/.local/plugged/vimtex/after
   filetype plugin indent on
   syntax enable
   
   let g:vimtex_matchparen_enabled = 0
   
   nnoremap q :call vimtex#profile#stop()<cr>:qall!<cr>
   
   silent edit mwe.tex
   
   call vimtex#profile#start()

3. Run nvim --clean -u test.vim, which will open mwe.tex and start profiling features of neovim. Position is at start of file.

4. Type Gkjk to go to the end and move up and down.

5. Type q to stop profiling and quit.

With g:vimtex_matchparen_enabled = 1, I notice a delay and observe that this delay is gone when I disable it. Since I'm using the profiling stuff, there is a file called prof.log that has some clues as to where things are slow.

[griffinwxk]

You need to compile first to see the real delay.

[lervag]

Ok. I've updated test.vim to this:

set nocompatible
set runtimepath^=~/.local/plugged/vimtex
set runtimepath+=~/.local/plugged/vimtex/after
filetype plugin indent on
syntax enable

let loaded_matchparen = 1
let g:vimtex_matchparen_enabled = 0
let g:vimtex_view_automatic = 0

nnoremap q :call vimtex#profile#stop()<cr>:qall!<cr>

silent edit mwe.tex

silent VimtexCompile

call vimtex#profile#start()

After step 3, compilation is started automatically. I move around, and everything looks fine. No delays.

[griffinwxk]

Sorry. A bit handful earlier. Hmm. Let me try your setup now.

[griffinwxk]

Can you try repeating k after G, and let cursor go all the way to the top and starts to scroll? That's when the lag appears for me in a minimal rc file.

[lervag]

Ok, cool, I can reproduce this now. I can also verify that turning off syntax removes the problem. But for me, compilation is not relevant. Thus, I can reproduce with the following minimal test.vim:

set nocompatible
set runtimepath^=~/.local/plugged/vimtex
set runtimepath+=~/.local/plugged/vimtex/after
filetype plugin indent on
syntax enable

let loaded_matchparen = 1
let g:vimtex_matchparen_enabled = 0

silent edit mwe.tex
normal! GH

1. Run nvim --clean -u test.vim.
2. Repeatedly typing k - lag is very noticeable.

Profiling did not reveal anything. There is also a syntax-specific profiling feature that I will look into now.

[griffinwxk]

Good to know! I also tried this profiling with luacode commented out and yeah it doesn't seem to reveal stuff.

I think compilation still matters. Did you delete all the compiled artifects? If not then the lag will persist no matter if you do another compilation or not. Can you confirm this?

[lervag]

The reason that VimtexCompile is necessary to reproduce this is because it has to do with a syntax package that is loaded "implicitly" by parsing the .fls file.

[lervag]

Thus, if you compile the project once to generate the .fls file, then compilation is no longer relevant.

[griffinwxk]

Yes. That makes sense to me.

[lervag]

Ok, we are getting closer! I've reduced the MWE to the file you've sent me except with this preamble:

\documentclass{minimal}
\usepackage{luacode}
\begin{document}

% REST OF ORIGINAL mwe.tex FOLLOWS ...

Obviously, this file won't compile. But it allows to reproduce the problem without having to compile anything.

[lervag]

Further, the issue comes from the syntax rules from autoload/vimtex/syntax/p/luacode.vim. I'll try to figure out exactly why this file is problematic.

[lervag]

Figured it out: The built-in syntax/lua.vim changes the syn sync minlines option to 1000, which is very large and leads to unnecessary processing time and delays. VimTeX sets it to 50, which is more sensible and should work well. I'll push a patch when I figure out how to properly wrap this for any included (nested) syntaxes.

[griffinwxk]

Wow that was quick! I just had my lunch and the problem already nearly solved!

[lervag]

There, I believe I've fixed the problem now. Can you update and test?

[griffinwxk]

Works like a charm! Thanks!

So it is really just the too-big number of lines (n)vim decides to look ahead for syntax that caused the problem?

[griffinwxk]

I will close the issue now btw. Thanks a lot!

[lervag]

Works like a charm! Thanks!

Glad to hear it! :)

Thanks for noticing this, reporting the issue, and providing input that helped me figure things out!

So it is really just the too-big number of lines (n)vim decides to look ahead for syntax that caused the problem?

Yes, exactly. the Lua syntax script specified a very large number of minimum lines to parse (1000), which means the syntax engine would always look behind and parse 1000 lines when you redraw the screen. This required O(100ms) and would be visible as a delay everytime the window changed content.

[griffinwxk]

I see. Wonder why the (n)vim upstream would do that. Maybe it's due to nested syntax? I'm no expert but would imagine pure lua would take considerably less time to parse than LaTeX so if it's not nested then maybe it's not a problem.

[griffinwxk]

Just a comment as I found the reason behind syn sync minlines stuff: from Neovim's history, apparently that number was changed by vim upstream from 100 to 1000 on September 2022. Neovim merged that patch a few days later. This explains why I suddenly suffered the lag during late last year.

[lervag]

I see. Wonder why the (n)vim upstream would do that. Maybe it's due to nested syntax? I'm no expert but would imagine pure lua would take considerably less time to parse than LaTeX so if it's not nested then maybe it's not a problem.

I have to admit I don't know. However, I do have the impression that a lot of people did not really learn how to write good syntax rules. Changing the syntax sync settings can be helpful to "fix" problems that could often be better solved by using better syntax rules.

Just a comment as I found the reason behind syn sync minlines stuff: from Neovim's history, apparently that number was changed by vim upstream from 100 to 1000 on September 2022. Neovim merged that patch a few days later. This explains why I suddenly suffered the lag during late last year.

Yes, I see that as well; unfortunately, there are no good commit messages that help explain the changes. Seems strange to me to change from 100 to 1000. :shrug: