Formatter cannot handle tabs and spaces properly for hidden terminal rules

[joluj]

Langium version: reproducible in 1.2.0 & 2.1.0 (current) Package name: langium

Steps To Reproduce

1. Open example/domainmodel in vscode with the domainmodel extension enabled
2. Add a new line with tabs and spaces before a comment (i.e. a hidden terminal rule)
3. Format code

Code example:

qualified-names.dmodel

This file contains tabs and spaces mixed. It works fine with non-hidden rules but breaks with hidden rules (e.g. comments) ``` entity E1 { name: String // Tabs are replaced in non-hidden nodes (expected) time: big.Int // Tabs and spaces mixed are converted to spaces (expected) // Tabs are not replaced in hidden nodes // Tabs and spaces mixed lead to total confusion } ```

The current behavior

• Tabs are not converted to spaces in hidden rules.
• Indent breaks if tabs and spaces are mixed in hidden rules.
• Both bugs don't occur with non-hidden rules.

  
  entity E1 {

• name: String // Tabs are replaced in non-hidden nodes (expected)
• time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
• // Tabs are not replaced in hidden nodes
• // Tabs and spaces mixed lead to total confusion
• name: String // Tabs are replaced in non-hidden nodes (expected)
• time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
• // Tabs are not replaced in hidden nodes
• // Tabs and spaces mixed lead to total confusion }

## The expected behavior
- Hidden and non-hidden rules have the same behavior
```diff
entity E1 {
-   name: String // Tabs are replaced in non-hidden nodes (expected)
-     time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
-   // Tabs are not replaced in hidden nodes
-     // Tabs and spaces mixed lead to total confusion
+    name: String // Tabs are replaced in non-hidden nodes (expected)
+    time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
+    // Tabs are not replaced in hidden nodes
+    // Tabs and spaces mixed lead to total confusion
}

[georg-schwarz]

Is there already any known workaround or solution for this?

[msujew]

@georg-schwarz Someone would probably need to take a look at how edits for hidden nodes are created. I don't think there's a fix for this yet. I have a general refactoring of the formatter in mind, which is why this issue is currently in the backlog.

[georg-schwarz]

Thanks for the pointer! We will share our solution here in case we decide to tackle it before you are done with the refactoring you mentioned ;-)