Distinguish between lexical character sets and list literals

[ahmadsalim]

Currently all elements between two brackets [] are treated naïvely as lexical character set, which results in issues with highlighting list literals. Particularly, we must ensure that only things between brackets inside lexical or syntax scopes are treated as character sets, and those outside are treated as list literals.

Note that removing the support for lexical character sets ruins the highlighting of the rest of the programs since \" and "/", and other similar characters can appear freely in character set expressions.

Examples of mis-highlighted terms:

// comment 1

n = 1;  // comment 2
l = [1, // comment 3: non highlighted
     2, 
     3];

list[int] n

private MuExp translateStringLiteral(s: (StringLiteral) `<PreStringChars pre> <Expression expression> <StringTail tail>`) {
    preIndent = computeIndent(pre);
    return muBlock( [ muCallPrim3("template_open", translatePreChars(pre), pre@\loc),
                      *translateExpInStringLiteral(preIndent, expression),
                      *translateTail(preIndent, tail),
                      muCallPrim3("template_close", [], tail@\loc)
                    ]   );
}

[ahmadsalim]

OK, I have tried to fix the issues now:

First Example

Second Example

Third Example

Non-regression: Grammar

I apologise for it taking a bit of time, but I had used a lot of time debugging things because I misunderstood a part of how the syntax highlighting matched over newlines (apparently a single pattern using match is only run for a single line, begin and end work with newlines).

[DavyLandman]

You are making great progress!

Perhaps update the pull request on the linguist library? to point to this newer version?

[PaulKlint]

I have tried to highlight two dozen fragments of complex Rascal code and so far I have not yet spotted irregularities. Great work. This is rapidly becoming very useful.

[ahmadsalim]

Great, thanks for the comments.

@DavyLandman I think they use git submodules, so it should synchronise automatically I hope.

[DavyLandman]

git submodules point to a specific commit in the history, so you'll have to update which commit it points to (see here).

You'll have to go into the vendor/grammars/rascal-syntax-highlighting directory, perform a git pull, go back to the root of the linguist folder and commit the changes (git status will show the pointer to the commit has moved)

[ahmadsalim]

@DavyLandman OK, done.