C++ syntax doesn't always terminate begin-end groups correctly and results in very deeply nested scoping

[FichteFoll]

From @wldcordeiro on March 11, 2015 20:21

If you open a large file the syntax highlighting will usually break somewhere in it. Here's an example of that. https://gist.github.com/wldcordeiro/390d30eb7c9ba65139c6 Just save this file and open it in Sublime Text 3 (build 3075 is the one I tried most recently) the highlighting broke around line 7,000. Obviously editing files this large is unwieldy (and this one in particular is not meant to be since it's generated) but reading them is even problematic.

Copied from original issue: SublimeTextIssues/Core#717

[FichteFoll]

This is an interesting one. It appears that the syntax definition creates many begin-end lookup matches that don't terminate properly, which leads to the scope becoming exceptionally long. At some point the scope is so long that ST doesn't attempt to highlight it anymore.

Here is a screenshot of how it looks when the highlighting slowly diminishes: As you can see in line 6563, string punctuation is still highlighted, but in 6571 it's not anymore. Then, starting at line 6626 the highlighting is gone completely.

The exact scope for the string where the caret resides in in the screenshot is as follows (yes, it's long):

source.c++ meta.extern-block.c++ meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.namespace-block.c++ meta.class-struct-block.c++ meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.block.c meta.block.c meta.block.c string.quoted.double.c

The exact scope for a non-highlighted string in line 6630 is as follows:

source.c++ meta.extern-block.c++ meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.namespace-block.c++ meta.class-struct-block.c++ meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.block.c meta.block.c meta.block.c string.quoted.double.c

And finally, here is the scope at the very end of the file, line 305823:

source.c++ meta.extern-block.c++ meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.namespace-block.c++ meta.class-struct-block.c++ meta.function.c meta.parens.c meta.extern-block.c++ meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c meta.function.c meta.parens.c

So, basically, this appears to be a bug with the default C++ syntax definition. It could as well be a bug that ST does not attempt to highlight long scopes, but since the above behavior is caused by a bug anyway, in which case you can not assume ST to handle it properly in all cases, I will say that we need a concrete use case or real world scenario where this behavior is problematic.

[wldcordeiro]

Ah, good to know it's the C++ syntax that's broken and not the syntax highlighting in general.

It may be worth looking at how the C syntax highlighting is done because that one doesn't seem to break (though obviously doesn't support all the C++ syntax.)

[TheClams]

The root cause is due to the use of macro inside function parameters: the macro call is interpreted wrongly as a function definition which left open a scope for function definition. I don't have a clean fix yet, but the test to do is simple in the syntax file:

long func(int x, void *MYMACRO(y) ) {} // Currently failing , to be fix 
/*                     ^ -entity.name.function */
/*                                       ^ -meta.parens */

[wbond]

The file https://gist.github.com/wldcordeiro/390d30eb7c9ba65139c6 seems to be highlighted properly using the latest syntax definitions from https://github.com/sublimehq/Packages.