Cpp lexer does not work properly

I tested the scintilla .net cpp lexer with a fairly complex C-source and the highlighter fails in a lot of cases - even the most simplest ones:

Preprocessor hightlighting simply fail in a lot of cases, which i cannot understand why. I set the style to maroon (grey) but only a portion of preprocessor lines are colored.
The keyword highlighting does not work either, as every word is basically threated as a keyword and colored blue in my case.
Comment line docs highlighting does fail a lot as well

The source i tested it with are: https://raw.githubusercontent.com/f1nalspace/final_game_tech/master/final_platform_layer.h

My code for setting up the editor is fairly simple:

private void SetupEditor(Scintilla editor)
            {
                editor.Dock = DockStyle.Fill;

                editor.WrapMode = ScintillaNET.WrapMode.None;
                editor.IndentationGuides = ScintillaNET.IndentView.LookBoth;
                editor.CaretLineVisible = true;
                editor.CaretLineBackColorAlpha = 50;
                editor.CaretLineBackColor = Color.CornflowerBlue;
                editor.TabWidth = editor.IndentWidth = 4;
                editor.Margins[0].Width = 16;
                editor.ViewWhitespace = ScintillaNET.WhitespaceMode.Invisible;
                editor.UseTabs = true;

                Font editorFont = new Font(FontFamily.GenericMonospace, 14.0f, FontStyle.Regular);
                editor.StyleResetDefault();
                editor.Styles[Style.Default].Font = editorFont.Name;
                editor.Styles[Style.Default].Size = (int)editorFont.SizeInPoints;
                editor.StyleClearAll();

                editor.Lexer = ScintillaNET.Lexer.Cpp;

                editor.Styles[Style.Cpp.Default].ForeColor = Color.Silver;
                editor.Styles[Style.Cpp.Comment].ForeColor = Color.Green;
                editor.Styles[Style.Cpp.CommentLine].ForeColor = Color.Green;
                editor.Styles[Style.Cpp.Number].ForeColor = Color.Olive;
                editor.Styles[Style.Cpp.Word].ForeColor = Color.Blue;
                editor.Styles[Style.Cpp.Word2].ForeColor = Color.Purple;
                editor.Styles[Style.Cpp.String].ForeColor = Color.FromArgb(163, 21, 21);
                editor.Styles[Style.Cpp.Character].ForeColor = Color.FromArgb(163, 21, 21);
                editor.Styles[Style.Cpp.Verbatim].ForeColor = Color.FromArgb(163, 21, 21);
                editor.Styles[Style.Cpp.Identifier].ForeColor = Color.Blue;
                editor.Styles[Style.Cpp.StringEol].BackColor = Color.Pink;
                editor.Styles[Style.Cpp.Operator].ForeColor = Color.Purple;
                editor.Styles[Style.Cpp.Preprocessor].ForeColor = Color.Maroon;

                editor.Styles[Style.Cpp.CommentDoc].ForeColor = Color.Purple;
                editor.Styles[Style.Cpp.CommentDocKeyword].ForeColor = Color.Red;
                editor.Styles[Style.Cpp.CommentDocKeywordError].ForeColor = Color.Red;
                editor.Styles[Style.Cpp.CommentLineDoc].ForeColor = Color.Purple;

                editor.TextChanged += (s, e) =>
                {
                    ScintillaNET.Scintilla thisEditor = (ScintillaNET.Scintilla)s;
                    TabEditorState tabState = (TabEditorState)thisEditor.Parent.Tag;

                    // Autofit left-margin to fit-in line count number
                    int maxLineNumberCharLength = thisEditor.Lines.Count.ToString().Length;
                    if (maxLineNumberCharLength != tabState.MaxLineNumberCharLength)
                    {
                        thisEditor.Margins[0].Width = thisEditor.TextWidth(Style.LineNumber, new string('9', maxLineNumberCharLength + 1));
                        tabState.MaxLineNumberCharLength = maxLineNumberCharLength;
                    }

                    tabState.IsChanged = true;
                    UpdateTab?.Invoke(this, new EventArgs());

                    tabState.TextChangedTimer.Stop();
                    tabState.TextChangedTimer.Start();
                };

#if false
                editor.StyleNeeded += (s, e) =>
                {
                    ScintillaNET.Scintilla thisEditor = (ScintillaNET.Scintilla)s;
                    TabEditorState tabState = (TabEditorState)thisEditor.Parent.Tag;
                    int startPos = thisEditor.GetEndStyled();
                    int endPos = e.Position;

                    var timer = Stopwatch.StartNew();
                    // @TODO(final): Custom styling
                    Debug.WriteLine($"Lex took: {timer.Elapsed.TotalMilliseconds} ms");
                };
#endif

                editor.KeyDown += (s, e) =>
                {
                    if (e.Control && (!e.Alt && !e.Shift))
                    {
                        e.SuppressKeyPress = true;
                        if (e.KeyCode == Keys.Home || e.KeyCode == Keys.Up)
                            GoToPosition(0);
                        else if (e.KeyCode == Keys.End || e.KeyCode == Keys.Down)
                            GoToPosition(Editor.TextLength);
                    }
                    else if (!e.Alt && !e.Shift)
                    {
                        if (e.KeyCode == Keys.F3)
                        {
                            //if (_searchReplaceViewModel.IsShown)
                            //    _searchReplaceViewModel.SearchExecutedCommand.Execute(SearchReplaceViewModel.SearchDirection.Next);
                        }
                        else if (e.KeyCode == Keys.Escape)
                        {
                            //_searchReplaceViewModel.HideCommand.Execute(null);
                        }
                    }
                };

                editor.InsertCheck += (s, e) =>
                {
                    if ((e.Text.EndsWith("\n")))
                    {
                        var curLine = Editor.LineFromPosition(e.Position);
                        var curLineText = Editor.Lines[curLine].Text;
                        StringBuilder addon = new StringBuilder();
                        for (int i = 0; i < curLineText.Length; ++i)
                        {
                            char c = curLineText[i];
                            if ((c != '\n') && char.IsWhiteSpace(c))
                                addon.Append(c);
                            else
                                break;
                        }
                        e.Text += addon.ToString();
                    }
                };
            }

I am not sure if the error is inside the wrapper or inside scintilla itself. My guess is, the lexer in scintilla itself is broken and i should roll out my own.

Seems that the C++ lexer has the property lexer.cpp.track.preprocessor enabled by default. This is what's causing the "fail". You have two options to solve this.

Disable the property via editor.SetProperty("lexer.cpp.track.preprocessor", "0");

Colourise styles from 64-90:

// Inactive whitespace
editor.Styles[64].ForeColor = ColorTranslator.FromHtml("#C0C0C0");
// Inactive block comment
editor.Styles[65].ForeColor = ColorTranslator.FromHtml("#90B090");
// Inactive line comment
editor.Styles[66].ForeColor = ColorTranslator.FromHtml("#90B090");
// Inactive Doc comment: block comments beginning with /** or /*!
editor.Styles[67].ForeColor = ColorTranslator.FromHtml("#D0D0D0");
// Inactive number
editor.Styles[68].ForeColor = ColorTranslator.FromHtml("#90B0B0");
// Inactive keyword
editor.Styles[69].ForeColor = ColorTranslator.FromHtml("#9090B0");
// Inactive double quoted string
editor.Styles[70].ForeColor = ColorTranslator.FromHtml("#B090B0");
// Inactive single quoted string
editor.Styles[71].ForeColor = ColorTranslator.FromHtml("#B090B0");
// Inactive UUIDs
editor.Styles[72].ForeColor = ColorTranslator.FromHtml("#C0C0C0");
// Inactive preprocessor
editor.Styles[73].ForeColor = ColorTranslator.FromHtml("#B0B090");
// Inactive operators
editor.Styles[74].ForeColor = ColorTranslator.FromHtml("#B0B0B0");
// Inactive indetifiers
editor.Styles[75].ForeColor = ColorTranslator.FromHtml("#B0B0B0");
// Inactive end of line where string is not closed
editor.Styles[76].ForeColor = ColorTranslator.FromHtml("#000000");
// Inactive verbatim strings for C#
editor.Styles[77].ForeColor = ColorTranslator.FromHtml("#90B090");
// Inactive regular expressions for JavaScript
editor.Styles[78].ForeColor = ColorTranslator.FromHtml("#7FAF7F");
// Inactive doc line comment: line comments beginning with /// or //!.
editor.Styles[79].ForeColor = ColorTranslator.FromHtml("#C0C0C0");
// Inactive keywords2
editor.Styles[80].ForeColor = ColorTranslator.FromHtml("#C0C0C0");
// Inactive comment keyword
editor.Styles[81].ForeColor = ColorTranslator.FromHtml("#C0C0C0");
// Inactive comment keyword error
editor.Styles[82].ForeColor = ColorTranslator.FromHtml("#C0C0C0");
// Inactive raw strings for C++0x
editor.Styles[84].ForeColor = ColorTranslator.FromHtml("#B090B0");
// Inactive triple-quoted strings for Vala
editor.Styles[85].ForeColor = ColorTranslator.FromHtml("#90B090");
// Inactive hash-quoted strings for Pike
editor.Styles[86].ForeColor = ColorTranslator.FromHtml("#90B090");
// Inactive preprocessor stream comment
editor.Styles[87].ForeColor = ColorTranslator.FromHtml("#A0C090");
// Inactive preprocessor stream doc comment
editor.Styles[88].ForeColor = ColorTranslator.FromHtml("#C0C0C0");
// Inactive user defined literals
editor.Styles[89].ForeColor = ColorTranslator.FromHtml("#D7A090");
// Inactive task marker
editor.Styles[90].ForeColor = ColorTranslator.FromHtml("#C3A1CF");

When the property lexer.cpp.track.preprocessor is enabled, it basically interprets #if/#else/#endif to grey out code that is not active, like how Visual Studio does it. So, if you don't colourise these inactive styles (see option 2), everything inside the preprocessor, depending on the conditional, will appear unstyled.

Imgur

Every word is appearing in blue color because your Style.Cpp.Identifier and Style.Cpp.Word are sharing the same color. Either change the color or remove the identifier styler.

Also, based on your code, you're not setting the lexer keywords. This will by default style them as Style.Cpp.Identifier. You can set them with:

scintilla.SetKeywords(0, "abstract as base break case catch checked continue default delegate do else event explicit extern false finally fixed for foreach goto if implicit in interface internal is lock namespace new null object operator out override params private protected public readonly ref return sealed sizeof stackalloc switch this throw true try typeof unchecked unsafe using virtual while");
scintilla.SetKeywords(1, "bool byte char class const decimal double enum float int long sbyte short static string struct uint ulong ushort void");

I'm unable to replicate a line comment doc failure. If you have an example, throw it in and I'll take a look.

Awesome, Thanks for that explanation! Now everything works as expected, expect for one thing: The comment doc blocks are not colored the way like i want it to be, so i want to add additional custom highlighting for that. How do i do that?

I know i can use StyleNeeded to colorize a range of text, but then i would lose the Cpp lexer :-( How can i call the Cpp lexer for a specific text range?

What i want is this:

Treat every source file as C/CPP and use the Cpp lexer to colorize everything
After that i want to recolorize all doxygen comment blocks by myself, but use the markdown/html lexer for initial highlighting
Also i want to use the Cpp lexer to colorize inline-code blocks inside doxygen comment blocks (@code or \code)

I already have all the code implemented to parse and colorize doxygen comment blocks, but i have no idea how to use a existing lexer. So what i basically want is to make highlighting for multiple languages.

@f1nalspace Its not possible to use multiple lexers at the same time.. However you can use a lexer and then colorize whatever else using the UpdateStyle event.. I'm not sure why you're saying you'd lose the lexer cause you won't.

First you set the styles on form load:

private enum NewStyles
        {
            Hotspot = 100,
            Functions,
            Publics,
            Stocks,
            Natives,
            Defines,
            Macros,
            Enums,
            PublicVars,
            Tags
        }

Editor.Styles[NewStyles.Functions].ForeColor = Color.Red;

Then in the event

private void Editor_UpdateStyle(object sender, UpdateUIEventArgs e)
{
    var startPos = Editor.Lines[Editor.FirstVisibleLine].Position;
    var endPos = Editor.Lines[Editor.FirstVisibleLine + Editor.LinesOnScreen].EndPosition;

    int startStyle; int length; int style;
    //TODO: some code to fill this vars.. gonna be a bit complex for blocks of code and you might want to go further then visible code..

    Editor.StartStyling(startStyle);
    Editor.SetStyling(length, style);
}

Thanks for the info, but i have now fully implemented everything myself: https://github.com/f1nalspace/doxygeneditor/tree/develop

jacobslusser / ScintillaNET

Cpp lexer does not work properly #429