Automatic source detection file format fails when source includes tabstops when coboleditor.enable_tabstop=false && editor.useTabStops=true

[MCEmperor]

Describe the bug The syntax highlighting is messed up with certain program structures. In fixed format, in combination with a tab used as indentation to 'skip' the sequence number area, parts of keywords are colored grey (the comment color).

To reproduce

• Copy the following snippet to a file.

  ········program-id.·SOMETEST.
  
  ········author.·SomeGuy.
  
  ········date-written.·July·20,·2021.
  
  ·······procedure·division.
  
  →   →   display·"Hi·there"
  →   →   goback·0
  →   →   .

  For visibility reasons, I have marked the spaces with · and the tabs with →.

• Replace each occurrence of a Unicode right long arrow followed by exactly three spaces by a tab character.
• Replace each occurrence of a Unicode middle dot (·) by a space.
• Select language mode: COBOL

Screenshot

The weird thing is, if we remove a newline preceding the procedure division, then the issue disappears, and the code is highlighted correctly.

With the same code as above, if we type an asterisk in the indicator area on the line immediately preceding procedure division, the problem disappears:

        ...
        date-written. July 20, 2021.

      *
       procedure division.
       ...

However, if we start writing a comment text behind the asterisk, then the issue reappears:

        ...
        date-written. July 20, 2021.

      * Start typing this text here and the problem reappears
       procedure division.
       ...

This problem has been introduced since version 7.6.6 (in 7.6.5, the problem was not there). This means that the changes by commit b3d6c4dffc958d2 has broken something.

System My COBOL related settings are as follows:

{
    "[ACUCOBOL]": {
    "editor.insertSpaces": true,
    "editor.renderIndentGuides": true,
    "files.autoGuessEncoding": false,
    "editor.autoIndent": "full",
    "editor.formatOnType": true,
    "editor.rulers": [4, 120],
    "editor.useTabStops": true,
    "editor.detectIndentation": false,
    "editor.wordSeparators": "`~!#$%^&*()=+[{]}\\|;:'\",.<>/?",
    "editor.highlightActiveIndentGuide": true
  },
  "[COBOL]": {
    "files.autoGuessEncoding": false,
    "editor.renderIndentGuides": false,
    "editor.autoIndent": "full",
    "editor.insertSpaces": true,
    "editor.formatOnType": true,
    "editor.rulers": [6, 7, 72],
    "editor.detectIndentation": false,
    "editor.wordSeparators": "`~!@$%^&*()=+[{]}\\|;:'\",.<>/?"
  },
  "coboleditor.enable_tabstop": false,
  "coboleditor.linter": true
}

[spgennard]

Please answer the following questions:

• you have included two language configurations but have not said which one you are using

Meanwhile, this is expected as the extension has no support for embedded tab characters, so a tab character is treated as one character

[GitMensch]

Not sure if this is the case, but I think it should: is the following setting taken into account when considering the position (both for free/fixed-form reference-format recognition and for syntax highlighting)?

// The number of spaces a tab is equal to. This setting is overridden
// based on the file contents when `editor.detectIndentation` is true.
"editor.tabSize": 4,

[spgennard]

@GitMensch - thank you for the learned comments, I think I know what should or should not do, thank you

[spgennard]

@GitMensch - fyi, in the example, the source format detection is correct, the left-hand margin is displayed wrong due to the presence of the tabs, when the "tabbed" lines are moved into the first couple of lines, it identifies it as variable format and the margin is not shown, giving the perspective that the detection is correct but it is in fact not.

Making the source format understanding tabs will not affect this issue.

The fix, is to ensure no margin is shown if the line includes a tab character on a line-by-line basis.

[GitMensch]

I see - starting with a "real" tab makes it a variable-length line (for the internal syntax-highlighting margin). Thanks for the explanation.

[MCEmperor]

Yes, I can happily report that the problem has disappeared. Thanks for fixing.