Backslashes in DataTable messes with highlighting

[maave0]

👓 What did you see?

I made a .feature file that included a DataTable and Examples table. Some lines had the last several characters uncolored - The lines that had backslashes in them.

✅ What did you expect to see?

All the text inside the pipe formatting is colored.

📦 Which tool/library version are you using?

Cucumber extension v1.7.0 for VSCode installed on VS Code v1.74.0

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Install Cucumber extension v1.7 onto VS Code v.1.74.0
2. Create a file called 'whatever.feature'
3. Write a scenario that includes either DataTable or Examples table
4. Put numerous backslashes inside the table's text

Example:

@BugReport
Feature: BugReport

Scenario: DEID117 Success
    Given I submit a bug report

    And I put backslashes in a DataTable or Examples table
    | this line works fine |
    | C:\\Users\\this line won't highlight the last 2 characters |
    | I see it with escaped newlines and escaped backslashes. ( \n \n \n \n  ) will produce 4 uncolored characters|

    Then Cucumber team fixes the highlighting :)

[kieran-ryan]

Thanks for catching this issue! Capturing some notes in case someone is in a position to take a look.

What's happening is the gherkin is being read and parsed as strings; and as they are escaped when evaluating their length, they are being incorrectly calculated as shorter than their full unescaped length. Thus, the syntax highlighting is not fully applied.

https://github.com/cucumber/language-service/blob/aa2843dee311cfdb5427f893d9ae14483e502e34/src/service/getGherkinSemanticTokens.ts#L49-L63

There is also an issue where when parsing we may lose information e.g. \ and \\ may both evaluate to \\.

Text	Token	Length	Expected Length
valu\e	valu\\e	6	6
valu\\e	valu\\e	6	7