A new way to fold.

[GoogleCodeExporter]

Using foldmethod=expr is too slow. A new way to do the folding is manual 
folding. When open a vimwiki file, we can use a function to scan the file and 
create the folding. Also we can add a command to manual update the folding. 
This is what latex-suite do.

Also folding by syntax is good choice according to bug 247. It seems that 
vimwiki used syntax folding before. Why this was changed to foldexpr?

Original issue reported on code.google.com by Broken.zhou on 9 Dec 2011 at 4:16

[GoogleCodeExporter]

Could you supply some example files? If possible, make the file as simple as 
possible, but exhibiting the problem. I thought I had an example some time ago, 
but I cannot find it anymore. I had a theory that perhaps a long section 
(meaning many consecutive lines with no "header") might cause a problem, but I 
am not sure. I have plenty of files that have no problem at all (including 
files that take a lot more than half a minute to convert to HTML).

I suspect that using syntax for folding is also going to turn out to be too 
slow in some cases (that is probably the main reason why syntax files rarely 
define folds by default).

The general view is that "expr" method with a simple expression (that is the 
case of vimwiki) is probably the method least likely to exhibit performance 
problems (aside from the manual method), but it all depends on a file, and no 
one seems to understand these issues very well... 

Some examples for studying this issue are really needed!

Original comment by tpospi...@gmail.com on 13 Jan 2012 at 3:06

[GoogleCodeExporter]

Original comment by tpospi...@gmail.com on 3 Feb 2012 at 4:38

• Changed state: ExamplesNeeded
• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Original comment by tpospi...@gmail.com on 3 Feb 2012 at 4:42

• Changed state: ExampleNeeded

[GoogleCodeExporter]

With syntax folding it is almost impossible to fold list items.

If you want a function that makes folding for you - who can stop you from using 
it? Just turn off vimwiki's folding and use whatever you have come with.

Original comment by habamax on 5 Feb 2012 at 12:28

[GoogleCodeExporter]

Issue 247 has been merged into this issue.

Original comment by tpospi...@gmail.com on 27 Mar 2012 at 9:16

[GoogleCodeExporter]

First of all, I'd like to thank you for the amazing work with vimwiki. I'm 
using it for about two years, and its great!

I've just moved from vimwiki version 1.2 to 2.0.1.
Despite all the improvements, an outstanding change the increased time in 
folding. I've been using `g:vimwiki_folding = 1` with no noticeable delay, but 
the new version takes up to 15 seconds to open some long files.

I believe that it is a very good idea to have an option to select between the 
new fold and the one available at version 1.2 - as suggested on issue 247.
Having an optional fold method that works only for headers would also be very 
helpful.

I'm attaching a file that take some seconds to open when folds are enabled. 
This file opens instantly when `g:vimwiki_folding = 0`.

Original comment by marcmo...@gmail.com on 13 Sep 2012 at 4:56

Attachments:

• temp.wiki

[GoogleCodeExporter]

Any news on this? Is the example provided enough?

Original comment by marcmo...@gmail.com on 3 Dec 2012 at 3:29

[GoogleCodeExporter]

I have tested 2 methods on my large junk file with a lot of =headersN=

 * syntax: ~15 seconds to open
 * expr (only headers to fold): ~16 seconds to open

Now could anyone show me if syntax is that significantly fater than expr?

Original comment by habamax on 17 Feb 2013 at 11:06

[GoogleCodeExporter]

Changes are in repo.

Could anyone test it? May be my implementation of syntax folding is completly 
wrong and ineffective.

Original comment by habamax on 17 Feb 2013 at 11:11

[GoogleCodeExporter]

@marcmontu

This issue perhaps should have been removed a long time ago. It is perfectly 
understandable that you did not file a new issue, seeing this one in the list, 
but the extreme slowness of folding in vimwiki after 2.0 is bleeding obvious. 
That is definitely not what the original problem was.

It is not completely clear if the original reporters of #261 and #247 also had 
the list folding turned on at the same time, but they seem to be knowledgeable 
enough to be able to figure that out as the real reason for performance 
problems - they only mention headers in their reports. With only the 
section/header folding, there had not seem to be any simple way to demonstrate 
a serious slowness (with the old vimwiki, that is).

Another symptom of this new(ish) problem: trying to use "=" and "-" for 
adjusting header level also requires a lot of patience (although not as much as 
waiting for any file to open).

@Maxim
The original reporter actually suggested to use (semi)manual folding - that 
would indeed be very likely the fastest option for very large files. But also 
the least convenient for most people, since the folds would have to be 
explicitly updated once in a while (after creation of new headers).

We can hijack this old issue now for the new problem, I guess. The original 
reporters are unlikely to come forward with some useful information now anyway.

Original comment by tpospi...@gmail.com on 19 Feb 2013 at 5:17

• Changed state: Accepted
• Added labels: Priority-High, Type-Defect
• Removed labels: Priority-Low, Type-Enhancement

[GoogleCodeExporter]

Trivia: a bisection shows that folding apparently broke already in r540 (April 
2012), first due to a small refactoring omission, but later it went from 
slightly bad to much worse...

Now it should be back to what it used to be
 http://code.google.com/p/vimwiki/source/detail?r=ef34c8f8ba6e821d286a7b17dd40059824e14066
meaning Vimwiki should be normally useable with folding (only now one has to 
use let g:vimwiki_folding ='expr', whereas before it was g:vimwiki_folding = 1).

The list folding remains very slow, and having other issues, but it is possible 
that it was always like that (it is one of many, many options that I have never 
even tried to use...)

Original comment by tpospi...@gmail.com on 19 Feb 2013 at 6:31

• Changed state: Started

[GoogleCodeExporter]

Hi Tomas, it is very nice to have you back!

I have tested your commit -- folding with 'expr' is a way faster than it was, 
including current syntax folding -- 4 seconds instead of 16 on my test file.

I'll play with it more hoping there is no regression.

Original comment by habamax on 21 Feb 2013 at 4:02

[GoogleCodeExporter]

With your patch folding works as intended. 

But I have found that all versions incorrectly fold some list items.

I am not sure if list item folding should be fixed or removed -- I for myself 
don't use it and I don't know would I be able to fix it.

Original comment by habamax on 21 Feb 2013 at 2:56

[GoogleCodeExporter]

I believe I've fixed it.

Now with 'expr' that folds stuff faster than 'syntax' should we keep 'syntax'?

Original comment by habamax on 21 Feb 2013 at 3:46

• Changed state: FixedNeedMoreTest

[GoogleCodeExporter]

So the original issue is about manual folding, not the syntax or expr, I have 
misread it (time have passed since I touched vimwiki). Anyway foldexpr works 
faster now, and 'automated/bindedtokey' manual folding could be added to 
vimwiki by adding corresponding code to autocmd/nmap.

Original comment by habamax on 21 Feb 2013 at 4:02

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

There is an error when open markdown files,
because `g:vimwiki_rxH{i}_Start` and `g:vimwiki_rxH{i}_End` are not defined if 
`g:vimwiki_symH=0`.
And there is a patch for that. :)

Original comment by themacropodus@gmail.com on 22 Feb 2013 at 7:04

Attachments:

• folding_for_markdown.patch

[GoogleCodeExporter]

Thx!

Original comment by habamax on 23 Feb 2013 at 10:54

[GoogleCodeExporter]

I can confirm Maxim's observation that if I make a file large enough to see any 
difference, syntax folding is actually noticeably slower than expr-based 
folding.  Whether there could perhaps be a class of (useful) vimwiki files 
where syntax folding outperforms the expr folding, that I do not know. So far, 
the experience suggests that community is unlikely to contribute some useful 
analysis... But if the option to use syntax folding isn't there, the chances of 
that go pretty much to zero.

Some of Vim's runtime syntax files define folds by syntax, but syntax folding 
is never turned on by default - I suspect that the potential impact on 
performance is the main reason.

Syntax folding has one big advantage in general: it can better deal with nested 
block structures. But that is of no use in vimwiki, where only list items can 
nest, and those cannot be folded by syntax without further restrictions on 
their form and other complications... So the only added value syntax folding 
could provide easily is, to fold code blocks, for instance. But the same thing 
can be done easily with expr-based folding as well, and with a useful fold-text 
- that actually makes a very big difference.

Regarding list item folding: I have no idea how many people use it. Given that 
many people seem to be using vimwiki for maintaining various task lists (as 
opposed to creating documents), it is possible that some use the list folding, 
and their files are likely fairly small.

I have tried to see if I can make the folding work much faster by completely 
avoiding the expensive search of the closest header line. It does not seem 
possible in the most direct way, Vim's support for relative fold levels turns 
out to be very weak. For instance, Vim's foldlevel() function is rather useless 
when called inside the fold expression - it pretty much always returns -1, 
instead of giving access to presumably cached absolute fold-level values of the 
lines above.

This means one would have to implement their own header level caching and 
attempt some resource-light synchronization to keep up with the changes in 
buffer - sort of interesting project, but certainly not something one would 
undertake just to support a secondary feature that very few people actually use.

I am thinking of perhaps keeping optional list folding, but without folding 
sections at the same time - so that those who only create lists have some 
option to fold them. They have to sacrifice section folding, so that we do not 
have this quadratic-complexity algorithm in the performance-critical path.

Original comment by tpospi...@gmail.com on 24 Feb 2013 at 3:59

[GoogleCodeExporter]

List item folding is now (revision r741) separated into its own function, and 
does not attempt to fold sections. There are some lists on which it may work, 
but it is very easy to produce examples where it fails. The attached file shows 
a two-item list, and a one-item list (the first one may sometimes work for a 
while, but 'zx' fold refresh seems to definitely turn it into non-working). It 
assumes shiftwidth=2.

Of course, for more complex real-life lists, there is little chance it would 
actually work. But the function is left in the codebase, should anyone feel 
like investing time into improving list folding. 

Looking into the documentation, it turns out that list folding is advertised as 
the main folding feature! That is a much bigger problem, as it seems to be more 
of an experimental hack (and rather slow one).

Original comment by tpospi...@gmail.com on 24 Feb 2013 at 11:48

Attachments:

• Ftest2.wiki

[GoogleCodeExporter]

Ok. Let's have it separated. I hope there will be no complaints.

Original comment by habamax on 25 Feb 2013 at 3:59

[GoogleCodeExporter]

I found a bug when using `expr` folding.

Consider following markdown file:

    # header1

// this is a code block
#include <stdio.h>
// other stuffs
```

Since foldexpr has no state, it doesn't know the include statement is in the code block and consider it is a header.

In fact, this bug also occurs in vimwiki syntax:

= header1 =
```
// this is a code block
= foo bar =
```

I have a solution, but I am not sure if it is appropriate. Modify folling line in VimwikiFoldLevel if line =~ g:vimwiki_rxHeader to if synIDattr(synID(a:lnum, 1, 0), 'name') == 'VimwikiHeaderChar'

Since it already validate by vim's built-in parser, seems there is no need to check it again by ourselves?

There is another problem on markdown (and pandoc) code block. Since vimwiki_rxPreStart and vimwiki_rxPreEnd are exactly same, both synmbols will consider it is start and coused abnormal behaviors.

I hava a solution based-on the same idea, check synIDattr(synIDtrans(synID(a:lnum, 1, 1)),"name") == 'PreProc' and using following exprssion to distinguish between start and end synIDattr(synIDtrans(synID(a:lnum+1, 1, 1)),"name") == 'PreProc'

Original comment by `themacropodus@gmail.com` on 4 Mar 2013 at 12:34

[GoogleCodeExporter]

@themacropodus I have tested a similar modification before turning the block 
code folding on, and decided against it very quickly: the performance goes down 
quite noticeably - as soon as you make any call to syntax functions, Vim drops 
down to (or below) the speed of the 'syntax' fold method - so there would be no 
point of keeping the 'expr' method for its speed. Files must be opened fast. 
Even the original reported issues were really about that.

Yes, it fails in fringe cases like apparent headers inside code blocks, 
apparent unpaired end-of-code block markers anywhere etc. - all these are cases 
that one may never encounter in their life while using vimwiki (and if you ever 
do, on a certain unfortunate file, you can simply open all folds with "zR" and 
carry on with your editing).

Regarding formats that use symmetric block markers: it is probably best to not 
attempt to support folding for anything like that, and certainly not with 
'expr' method. Choosing symmetric markers with no other constraints (e.g. 
requiring empty lines before or after) that would differentiate which is which 
(opening or closing) is a very poor design decision, because you cannot 
effectively sync on anything. Some people think they can get away with it if 
the environment does not need to nest, but it is a poor choice for many 
practical reasons. Such a format is "read only", as any editing support is 
going to be ridiculously inefficient (parse all the way from the start to 
figure out what _this_ ``` means). Not to mention that people typing the text 
usually do not insert the whole block atomically.

Aside from that, the syntax engine of Vim is not going to save the day every 
time, it is not all-knowing and infallible: it has no syncing points to latch 
on, so you have a choice of either slow (always sync syntax from the start), or 
occasionally wrong (or often wrong, in case the code blocks are long). It might 
appear to work fine, but only with relatively short blocks. Of course, with 
fringe events like unpaired markers, you are back to square one...

Original comment by tpospi...@gmail.com on 12 Mar 2013 at 3:20

[GoogleCodeExporter]

@tpospich8: Ok, I got it. Thank you for your kindly explanation. :)

Original comment by themacropodus@gmail.com on 13 Mar 2013 at 6:37

[GoogleCodeExporter]

I've tested version 5db3125c4e3d 
(http://code.google.com/p/vimwiki/source/detail?r=5db3125c4e3d6ddd27f3df62b511b4
5db8d7a3bd#) and it works like a charm!
The long files that used to take 15 seconds to load now open in less than one 1 
second.

Thank you very much for providing and improving this great plugin!

Original comment by marcmo...@gmail.com on 18 Mar 2013 at 11:13

[GoogleCodeExporter]

Original comment by habamax on 14 Apr 2013 at 2:50

• Changed state: Fixed