Array access on a new line break tokenization

[alexr00]

Checklist

• [ ] This problem exists even with the setting "C_Cpp.enhancedColorization": "Disabled"
• [ ] This bug exists for C
• [x] This bug exists for C++
• [ ] This bug exists for Objective-C
• [ ] This bug exists for Objective-C++

The code with a problem is:

bool function1() {
    int mat[100][100];
    int x = 50;
    auto price = mat[x]
        [x];

    // Keywords in following statements are not highlighted
    auto tmp = 2;
    auto a = tmp;
    return true;

}

bool function2() {
    int mat[100];
    int x = 50;
    auto price = mat
        [x];        // Keyword in statements following are not highlighted
    auto tmp = 2;
    return true;
}

It looks like:

Dark+

It should look like:

This, but with the like breaks from the above sample

Originall from @wsrfad in https://github.com/microsoft/vscode/issues/107346

[a-stewart]

I've been looking into this a bit, and the issue I think is in this:

  "name": "meta.bracket.square.access",
  "begin": "([a-zA-Z_][a-zA-Z_0-9]*|(?<=[\\]\\)]))?(\\[)(?!\\])",

I was hoping that I could just add \s* before the (\\[) but that doesn't seem to be working for some reason.

The main issue is that the but that comes before the square brackets is being clobbered by the lambdas, which also look for [. It seems like when it is on the same line, the lambda doesn't match, but when it is given a new line, it prefers to associate it with lambda.

Removing both references to "include": "#lambdas" fixes this issue, as does placing "include": "#square_brackets" above "include": "#lambdas".

I need to investigate if the latter change has any other side effects that might cause issues.

[jeff-hykin]

I need to investigate if the latter change has any other side effects that might cause issues.

If you're able to get the codebase running with commands/start or if you manually install all the npm stuff

You can test side effects by

1. running this command

npx textmate-tester \
    --textmateExtension cpp \
    --syntax "./autogenerated/cpp.tmLanguage.json" \
    --examples "./language_examples/" \
    --supportSyntaxes '{ "source.cpp.embedded.macro": "./autogenerated/cpp.embedded.macro.tmLanguage.json" }' \
    generate-specs --all

2. Looking at all the changed yaml files.

Every yaml file has a corrisponding cpp file, open up the cpp file in the debugging window to see if it looks broken. There's probably a file specifically for lambda's so you could just go straight to that one.

I was hoping that I could

Yeah.. welcome to TextMate grammar editing; caveats abound. Look-behinds like (?<=) can't contain quantifiers, and it fails silently. Its something I forget on occasion too. Ironically we can brute force it to a certain number of characters. For example by saying one space, or two space, or three spaces, ... (?<=a| a| a| a| a| a).

But before we get into full textmate-hack-mode, I've got to say this one looks actually fixed on my end (scopes too not just the colors). What version of the extension are you using @a-stewart ?

[jeff-hykin]

I'm going to close this as it seems fixed for me. But if its an issue for anyone I'll open it back up

[a-stewart]

Hi,

Sorry for the slow reply on this. Both the existing functions seem to work, but I have anothe example of what seems to be a broken setup. (Using insiders.vscode.dev).

bool function1() {
    int mat[100][100];
    int x = 50;
    auto price = mat[x]
        [x];

    // Keywords in following statements are not highlighted
    auto tmp = 2;
    auto a = tmp;
    return true;

}

bool function2() {
    int mat[100];
    int x = 50;
    auto price = mat
        [x];        // Keyword in statements following are not highlighted
    auto tmp = 2;
    return true;
}

bool function3() {
    auto foo = foo
        [bar]
            .function("string", 7, "string", true);
    return foo;
    // We are still in a string.
}

[RedCMD]

lookarounds are allowed quantifiers inside of them you're just not allowed to put the quantifer directly on the lookaround however they are limited to 10,000 characters