[RFC] Provide a generic way to highlight operator keywords

[jwortmann]

What happened?

This issue is supposed to track the status of the problem described in the rejected and now closed PR #2090, because finding and implementing a solution is still relevant to me. Probably most of the important aspects have already been discussed in the referenced PR, however, no final decision was taken.

tldr; it is currently impossible in Sublime Text color schemes to target operator keywords in a general way, and sometimes there doesn't even exist a workaround to highlight certain operator keywords, because they use the exact same scope as operator symbols.

There is also a rule for it in the scope naming guidelines, but it is not widely used in the default syntaxes, and the consensus seems to be that this rule should not be followed.

---

Finally, here are screenshots from a few other editors, with two code examples from Rust & PHP. Mostly (but not always) those editors are able to highlight operator keywords, but no operator symbols, and this is even the default style in their built-in themes (pay attention to in keyword in Rust, and and & or keywords in PHP):

Visual Studio Code

(Fails to highlight `and` & `or` keywords in PHP) 

Notepad++


JetBrains Fleet


Syntax highlighting on GitHub

```rust fn main() { for _ in 1..3 {} let n = 1 + 1; match n { 1 => println!("One"), 2 => println!("Two"), } } ``` (Fails to highlight `and` & `or` keywords in PHP) ```php ```

[frou]

I agree that this is a very conventional style and not some oddball request.

Ideally a ST Color Scheme author would be able to achieve this style (word operators coloured like keywords, symbol operators not coloured (or coloured separately)) with a nice declarative rule such as:

{
  "scope": "keyword - keyword.operator , keyword.operator.word",
  "foreground": "var(green)"
}

(The previous discussion suggested a tweak to use meta.operator.word keyword.operator instead of keyword.operator.word. EDIT: and deathaxe's post below has even better suggestions)

But as things stand, a Color Scheme rule like the above is not sufficient in two ways:

1. It needs continuous language-specific additions tacked on to catch various word operators that slip through the net, e.g. keyword.operator.logical.python to catch and, in, is, not, or in Python; keyword.operator.assignment.alias to catch as in SQL, and so on.

2. Given some Syntax Definitions, it's currently impossible to use a selector to target certain word operators distinctly from symbol operators. Some random examples:

   • not does not have distinct scope from && in Ruby.
   • rem does not have a distinct scope from + in Erlang.
   • as does not have a distinct scope from postfix ! in TypeScript.

[deathaxe]

ST's scope naming guidelines don't express current state of implemented scope naming schemes of operators.

1. The word scope was designed for syntaxes which primarily use symbolic operators with some minor exceptions.

   To provide consistent highlighting, the word scope would need to be applied to all syntaxes, even if they only use word-like operators. Those would loose semantic meaning when only applying keyword.operator.word to avoid stacking.

2. Operators' (and other) 3rd-level scopes are currently dedicated to semantic meaning:

   keyword
    operator
      assignment
      arithmetic
      bitwise
      comparison
      logical

   All of them may be expressed by symbols or words. Hence 3rd-level scope word doesn't fit into the current scheme (any more).

3. ST supports up to 64 scopes being stacked onto each other.

   Once hit, highlighting just stops. _This has been reported several times for at least JavaScript in the past. (see: https://github.com/sublimehq/sublime_text/issues/5283)_

   With syntax based folding requiring a meta scope for each block/group/... to be able to distinguish folding levels and syntaxes extending or being embedded into each other, it might easily become more likely to hit that limit.

   Hence stacking scopes should be used with care.

4. Proposed meta.operator.word keyword.operator is quite the opposite pattern of what was the outcome of a former discussion about numbers.

   Semantic meaning was moved to meta scope (meta.number.[integer|float|...]). Syntactic scopes constant.numeric.[base|value|suffix] were decided to stay in constant scope, dedicated for primary highlighting.

   If we ignore 2.) and don't want to mix up such patterns, operator scope naming schemes would need to be changed to something like:

   meta
    operator
      assignment
      arithmetic
      bitwise
      comparison
      logical
   
   keyword
    operator
      symbol
      word

   That's a breaking change (again) for probably various official or private/personal color schemes.

   Just stacking keyword.operator.arithmetic keyword.operator.word would avoid that, but ...

   • still having 2 scopes per operator
   • stacking same kind of scopes (e.g. keyword) turned out to have unwanted implications in the past
   • word may also be interpreted as semantic word-like operator which combines 2 dword operands and thus may cause confusion to new syntax authors in its current context. Hence having it in the same (level of) scope is not ideal.

---

I don't have a final proposal/solution, yet, but if a breaking change was not the end of the world, my personal favourite was Thom's proposal about moving semantic meaning to level 4.

   keyword
     operator
       symbolic
       word
         assignment
         arithmetic
         bitwise
         comparison
         logical
         other

Reasons:

• only one scope
• symbol vs. word is more likely to be relevant for highlighting than semantic meaning. The latter might rarely be addressed.
• majority of legacy color schemes addresses keyword.operator only, so maybe junger ones might be effected primarily (which are hopefully still maintained).

[deathaxe]

FWIW, here's an excerpt of TextMate's scope naming guideline:

• keyword — keywords (when these do not fall into the other groups).
  • ...
  • operator — operators can either be textual (e.g. or) or be characters.
  • ...