Folding is incredibly slow

[kiryph]

I have added to my vimrc let g:markdown_enable_folding = 1 to enable folding. However, this makes vim not usable anymore. I had to turn it off.

I did some profiling according to http://stackoverflow.com/a/12216578/1057593.

❯ tail -23 profile.log
FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
 1024  27.219395  27.202780  markdown#FoldLevelOfLine()
 1832              0.016615  <SNR>112_SyntaxGroupOfLineIs()
    1   0.010944   0.010892  validator#utils#load_checkers()
    8  27.228979   0.008288  <SNR>20_WinDo()
    2   0.039826   0.007026  <SNR>9_LoadFTPlugin()
    7   0.012807   0.005226  <SNR>91_line()
    1  20.531948   0.003917  <SNR>102_openfile()
    2   0.039526   0.003830  <SNR>12_SynSet()
    1   0.003340   0.003296  lightline#highlight()
   15              0.002863  lightline#link()
   14   0.005898   0.002804  <SNR>91_expand()
    1              0.002633  <SNR>102_MapNorms()
    1   0.008210   0.002248  <SNR>102_BuildPrompt()
    1   0.013271   0.002091  <SNR>99_check()
    1   0.003495   0.001857  <SNR>102_opts()
   25              0.001683  <SNR>91_subseparator()
    1   0.002695   0.001580  <SNR>102_Open()
    1              0.001491  <SNR>102_MapSpecs()
   76   0.002408   0.001390  <SNR>91_convert()
   30   0.002152   0.001381  <SNR>102_formatline()

As expected the problematic function is the fold-function. The profiling result for FUNCTION markdown#FoldLevelOfLine() is in particular:

FUNCTION  markdown#FoldLevelOfLine()
Called 1024 times
Total time:  27.219395
 Self time:  27.202780

count  total (s)   self (s)
 1024              0.003851   let currentline = getline(a:lnum)
 1024              0.004317   let nextline = getline(a:lnum + 1)

                              " an empty line is not going to change the indentation level
 1024              0.006201   if match(currentline, '^\s*$') >= 0
  108              0.000147     return '='
                              endif

                              " folding lists
  916   0.013978   0.005125   if s:SyntaxGroupOfLineIs(a:lnum, '^markdownListItem')
                                if s:SyntaxGroupOfLineIs(a:lnum - 1, '^markdownListItem')
                                  return 'a1'
                                endif
                                if s:SyntaxGroupOfLineIs(a:lnum + 1, '^markdownListItem')
                                  return 's1'
                                endif
                                return '='
                              endif

                              " we are not going to fold things inside list items, too hairy
  916   0.012817   0.005055   let is_inside_a_list_item = s:SyntaxGroupOfLineIs(a:lnum, '^markdownListItem')
  916              0.001436   if is_inside_a_list_item
                                return '='
                              endif

                              " folding atx headers
  916              0.005400   if match(currentline, '^#\{1,6}\s') >= 0
   24              0.000353     let header_level = strlen(substitute(currentline, '^\(#\{1,6}\).*', '\1', ''))
   24              0.000080     return '>' . header_level
                              endif

                              " folding fenced code blocks
  892              5.871741   let next_line_syntax_group = synIDattr(synID(a:lnum + 1, 1, 1), 'name')
  892              0.007428   if match(currentline, '^\s*```') >= 0
                                if next_line_syntax_group ==# 'markdownFencedCodeBlock'
                                  return 'a1'
                                endif
                                return 's1'
                              endif

                              " folding code blocks
  892              9.018623   let current_line_syntax_group = synIDattr(synID(a:lnum, 1, 1), 'name')
  892             12.198250   let prev_line_syntax_group = synIDattr(synID(a:lnum - 1, 1, 1), 'name')
  892              0.007893   if match(currentline, '^\s\{4,}') >= 0
   24              0.000075     if current_line_syntax_group ==# 'markdownCodeBlock'
                                  if prev_line_syntax_group !=# 'markdownCodeBlock'
                                    return 'a1'
                                  endif
                                  if next_line_syntax_group !=# 'markdownCodeBlock'
                                    return 's1'
                                  endif
                                endif
   24              0.000025     return '='
                              endif

                              " folding setex headers
  868              0.009438   if (match(currentline, '^.*$') >= 0)
  868              0.004672     if (match(nextline, '^=\+$') >= 0)
                                  return '>1'
                                endif
  868              0.003954     if (match(nextline, '^-\+$') >= 0)
                                  return '>2'
                                endif
  868              0.000694   endif

  868              0.000945   return '='

The problematic lines are:

  892              5.871741   let next_line_syntax_group = synIDattr(synID(a:lnum + 1, 1, 1), 'name')
  892              9.018623   let current_line_syntax_group = synIDattr(synID(a:lnum, 1, 1), 'name')
  892             12.198250   let prev_line_syntax_group = synIDattr(synID(a:lnum - 1, 1, 1), 'name')

My vim version is

:ver
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Oct 29 2016 06:15:02)
MacOS X (unix) version
Included patches: 1-52
Compiled by travis@Traviss-Mac-592.local
Huge version with MacVim GUI. 

MWE-vimrc, using vim-plug installed to ~/.vim/autoload/plug.vim and the plugins to ~/.vim/plugged/

" invoke with
" $ vim -u ~/.vim/mwe-vimrc
set nocompatible
runtime autoload/plug.vim
call plug#begin('~/.vim/plugged')
Plug 'Konfekt/FastFold'
Plug 'gabrielelana/vim-markdown'
call plug#end()
let g:markdown_enable_folding = 1

I already use Konfekt/FastFold. However, folds still have to be updated when opening the file, opening and closing folds. There is no chance that I wait so long to open files, open or close folds.

Is there any chance to improve the performance?

[gabrielelana]

Nice work!

I tried to piggyback on the work done in the syntax detection (hence the synIDattr calls) but seems like it's not as fast as I thought...

The alternatives are:

• redo most of the work done in syntax detection (an hard and painful way)
• fold less elements (only headers?) and do it in a less accurate way (with regular expressions without considering the context)

This is the real problem with markdown, if you want to be "correct" as I try to be here, you need to consider a fair amount of context to decide what is what and that is slooow

So, the answer could be to ad an option g:markdown_enable_simple_folding and g:markdown_enable_smart_folding with g:markdown_enable_folding to enable g:markdown_enable_smart_folding to be backward compatible

Where g:markdown_enable_simple_folding is folding based on simple (and fast) regular expressions

Pull requests on this are welcomed 😄

[kiryph]

Thanks for your quick response and your thoughts!

I did some research about folding capabilities of vim plugins for markdown files:

• $VIMRUNTIME/ftplugin/markdown.vim offers header folding (maintainer Tim Pope, most recent version can be found here https://github.com/tpope/vim-markdown)
• https://github.com/plasticboy/vim-markdown offers also only header folding.
• https://github.com/vim-pandoc/vim-pandoc can fold more than only headers, but not arbitrary html tags. It supports fenced code blocks, dedicated fold comments such <!-- {{{ ->> or <!-- Custom fold fold-begin --> and selected div classes which end with </div foldend>.

I was hoping that your plugin can fold the html tag <details> in a markdown file:

  <details><summary>Abstract</summary>
  *The invention is directed to...*
  </details>

since none of the other plugins offer a possibility for this.

So I think, adding fast folding based on regular expressions is not necessary, just disable the folding of your plugin and use the one distributed with vim.

I have a question: Would it be possible to improve the speed by modifying the syntax file?

Right now, I switched to vim-pandoc using fold comments where, however, I miss support for fold texts (see https://github.com/vim-pandoc/vim-pandoc/issues/199). Having a line

##  <!-- {{{ --> / 12 lines / -----------

does not tell you anything about what is hidden in the fold.

[gabrielelana]

I have a question: Would it be possible to improve the speed by modifying the syntax file?

Good question, I don't know if synIDattr/synID could be faster with a simpler grammar, I don't think so because it's a fact that they are slow

But even so, we will go back to the same problem, simpler syntax can be achieved eliminating the context consideration (the amount of text to consider in the surrounding area) but then will be very difficult to maintain correctness 😞

In the end it's a tradeoff between "simple and fast" Vs "correct and slow", I chose the second, other plugins chose the first

[AdrienLemaire]

Hi @gabrielelana. Do you have some time to look at improving folding capabilities again?

I care about folding as well, but it's not very efficient at the moment. Since I always put a title with # at the top of my markdown files, vim-markdown always fold the entire file when I enter it.

I do not know what would be a good way to handling auto-folding, but maybe adding a setting to manually set the preferred depth for headers could make it less cumbersome.

let g:markdown_enable_folding = 1
let g:markdown_folding_level = 3

# Level 1 title

## Level 2 

+--- 9 lines: ### level 3 title··················
+--- 9 lines: ### another title··················

## Another Level 2 title

+--- 9 lines: ### level 3 title··················
+--- 9 lines: ### another title··················

[AdrienLemaire]

It is also strange that I cannot unfold 2 levels of headers with 2zo. Right now, 2zo and zo in

+--- 44 lines: # Level 1 title············

both result in

# Level 1 title

+--- 20 lines: ## level 2··················
+--- 20 lines: ## Another Level 2··········

I wish ## headers were considered under the cursor when the whole file is folded...