syntax highlighting is unstable/needs improvement

[fmoralesc]

I've found, on testing the plugin with several files, that syntax highlighting is unstable, and it ceases to work correctly on several cases. Headers and lists are specially prone to break.

This was pointed out first by dsanson, on issue #6, but couldn't be reproduced.

Some related issues seem to be:

• emphasis on setext headers break them. See doc/pandoc.txt as a pandoc file for an example.
• non-compact lists break.

[dsanson]

Yep. I've continued to have this problem, but inconsistently. It often seems like the very same file will sometimes be properly highlighted and other times not. Not sure what the cause is. I do think it was introduced by the changes you made vis-a-vis implicit links. In fact, on doc/pandoc.txt (with ft=pandoc) I can get the highlighting of the setext headers to work by turning that feature off:

let g:pandoc_no_empty_implicits = 1
set ft=pandoc

Does that fix all the problems you are seeing too? If so, perhaps we turn this off by default, at least until it is fixed.

[fmoralesc]

I think the problem is due to the rules in the syntax file depending on the nextgroup= option (that makes me feel that a complete rewrite is needed :/ ), and also due to the sync method used, which doesn't handle well items that span more than two lines (title blocks, for example). The problems with the headers can be somehow alleviated by changing line 18 into:

syn sync linebreaks=4

and lines 48.52 to:

"   Underlined, using == or --
syn match  pandocHeader    /^.\+\n[=]\+$/ contains=@Spell,pandocLatex,pandocEmphasis,pandocStrong 
syn match  pandocHeader    /^.\+\n[-]\+$/ contains=@Spell,pandocLatex,pandocEmphasis,pandocStrong
"   Atx-style
syn match pandocHeader /^\s*#\{1,6}[^#]*$/ contains=@Spell,pandocLatex,pandocLinkArea,pandocEmphasis,pandocStrong

It should fix headers:

% a
% b
% c

# as lkjalkj sas *sss*

# *hello*

aaaaaaa *test*
==============

aaaaaaaaaaaaaa
--------------

*aaaaaa*
========

*aaaaaa*
--------

[fmoralesc]

Only setext-headers that begin with * fail. That leads to some problem in the emphasis definition. I tried making a separate rule for emphasis on headers, but that can get overruled by the other rule for emphasis. (And is a bit messy).

And no, disabling highlighting of implicit links doesn't fix the problems for me.

[dsanson]

I haven't noticed a connection to using emphasis in headers.

Note that there is a mode line at the end of pandoc.txt (and so also README.markdown, since it is just a symlink) that sets filetype to help. So by default, it will not be highlighted as a pandoc file. This is to enable helpfile style linking, using |tag| and tag. This is a bit confusing, because the help file syntax highlights the ====== and ------- just like the pandoc syntax file highlights headers.

[fmoralesc]

BTW: I'm working on a rewrite of the syntax file, in a local branch. So far, it's going quite well, and the new syntax file seems to be much more stable than the old one. In fact, i've managed to implement proper highlighting of definitions, and made other improvements.

I still have to implement highlighting of tables; i tried to reuse the old rules, but they didn't work. When that is done, I'm pushing upstream.

[dsanson]

neat!

[fmoralesc]

I'm stuck with two problems with the new syntax:

1. Tables are still unsupported.
2. Emphasis and bold styles don't work well together, and don't handle cases where multiple lines should be highlighted. I've tried many approaches, but it doesn't seem to work. For now, I'm simply using the old rules for them, so there is no regression.

Besides this, the new syntax seems to a mayor improvement over the old one; at least, highlighting doesn't stop mysteriously in the middle of a document, and most of the pandoc-extended markdown syntax seems to be handled correctly. I'm uploading the branch so other people can test it; if there are no mayor issues with the changes, I'll include them in the master branch.

[dsanson]

Just tried it out on a few files and didn't have any problems other than what you mention. My sense is that bold and emphasis are just a problem for any regex based implementation of markdown, so I don't think we should worry about handling all the cases just right. (If we were really clever, we'd figure out how to use the pandoc library to parse the markdown and feed the resulting information into vim's syntax highlighter ;-)

[fmoralesc]

Yeah, I thought of something similar too. We could so something like what we do with empty implicit links. (Actually, I got vim to highlight bold and emphasis correctly, handling linebreaks and all, but not together, which is the main thing).

So, should we include the new syntax into master as it is?

[fmoralesc]

Unless there is some objection, I'm checking the syntax changes into master (i'll do

git checkout new-syntax -- syntax/pandoc.vim

instead of merging). I'll open a new issue regarding tables highlighting afterwards.

[dsanson]

No objection here. I think the new syntax file is unquestionably better than what is there now.

[fmoralesc]

OK then. Closing this issue.