Annotation qualifier scope names

[wbond]

With new guidelines for annotations in #709, we didn't address the issue of scoping identifier qualifiers, i.e. "paths". Example identifiers with qualifiers:

Python

@ClassName.method(True)
def my_func():
    pass

Scalar

@scala.beans.BeanProperty
var status = ""

Java

@qualifier.ClassName
class Foo { }

My intention is to scope the final identifier as variable.annotation, with ., ::, etc as punctuation.accessor and the @ (or other symbol) as punctuation.definition.annotation.

The question raised in #735 is if the whole qualified identifier should get variable.annotation, or if classes/namespaces/modules should get scopes they would in other parts of the syntax. For instance something like support.class in some cases.

What do you think @djspiewak, @gwenzek, @FichteFoll, @keith-hall, @michaelblyons?

[keith-hall]

I personally think they should get the scopes they would in other parts of the syntax - if users/color scheme authors want, they can make the whole annotation the same color by utilizing the meta scope, while allowing others to have it presented in the same colors as they have come to expect from the main document.

[djspiewak]

Quoting myself from the other issue, so that the discussion is in one place:

That's fair, though I think syntactically it might look a little weird because the @ is to the far left (whereas function calls have nothing to the left). Additionally, in the original version of the proposal, the "annotation scope" (whatever it ended up being) was supposed to be applied to the @, overlapping the punctuation scope. Is that changed now?

If it's unchanged, it will end up with this rather odd situation where variable.annotation starts, then stops after one character and is replaced by "normal" scoping, and then starts again for a single identifier.

FWIW, nearly all editors which have any scoping at all for Java/Scala annotations tend to scope the entire token string, including quantifiers.

While it is certainly possible for color scheme authors to use the meta scope to color the entire construct if they so desire, I'm almost never in favor of scoping which pushes color schemes to use the meta scopes directly. And I think that, especially for modes which scope user constructs (like Scala), it would look remarkably weird. Even without user construct scoping. This might give you an idea:

That's with an edited Material color scheme. At present, it's just inheriting scoping from variable, but even adding specific scoping for variable.annotation and maybe punctuation.definition.annotation, that middle bit looks remarkably out of place.

[FichteFoll]

It would look as much out of place as with just scala.beans.BeanProperty in a normal statement, so I'm with @keith-hall. By the way, I will add a dimmed background highlight for my color scheme that targets meta.annotation and will include potential parameters, as I already have currently.

I posted an example of this in #704:

[djspiewak]

@FichteFoll I think it looks more out of place than the equivalent simply because the syntax is driven by the token on the far left. Highlighting the meta scope obviously does fix the problem, but again, is that really what we want to encourage color scheme authors to do?

[wbond]

That's with an edited Material color scheme. At present, it's just inheriting scoping from variable, but even adding specific scoping for variable.annotation and maybe punctuation.definition.annotation, that middle bit looks remarkably out of place.

No more out of place than a qualified function call, right? I personally think the leading punctuation will make the biggest difference. I also think that color schemes look better with accessors set to an opacity of something like 0.8.

Perhaps that leading @ should be a keyword.operator instead of punctuation.definition? That is what it is currently set to in Python for decorators, and I've generally liked it.

[FichteFoll]

@djspiewak what is "the equivalent"? Note that the above screenshot was taken before any of the standardization has been implemented (which I am currently working on).

[FichteFoll]

Perhaps that leading @ should be a keyword.operator instead of punctuation.definition? That is what it is currently set to in Python for decorators, and I've generally liked it.

That's wrong semantically, though. Ideally the color scheme should target punctuation.definition.annotation and highlight it however it wants to (keyword.other* in this case), but we all know that most color schemes will never implement it.

Might be interesting to know which color schemes are used the most, so we can ensure that the largest user base will benefit from these decisions by us patching them upstream.

[wbond]

That's very wrong, semantically though.

Well, I am sort of asking here if it really is semantically wrong. If I thought it was totally wrong I wouldn't have suggested it.

The @ in almost all of these languages is like a prefix operator that accepts a callable and applies it to the following construct, correct?

[djspiewak]

@wbond I can't speak to the semantics in other languages, but for everything on the JVM (Java and Scala especially), annotations semantically work in the following way.

The @ "operator" is basically like a special form of new, run at compile time. Unlike new, it can only be applied to interfaces (if you're not familiar with Java, these are basically just like special classes), it can only take certain types of data as constructor parameters, the target interface must itself have special metadata (denoted in Scala by its own annotation), and it does not produce a value as a result. Instead of a value, the result of the instantiation is placed in the compiled bytecode.

From that perspective, the following scoping would seem (mostly) right:

   @foo.bar.Baz
// ^ keyword.operator.annotation.scala
//  ^^^^^^^^^^^ support.class.scala
// ^^^^^^^^^^^^ meta.annotation.scala

But of course, no one thinks of it that way. Because the compiled results are placed in the bytecode, no one really thinks of @ as an instantiation operator. Instead, the declarative perspective dominates, and you usually read @ as a marker for declarative compile-time metadata which happens to be represented by an instance of an interface.

This is why scoping the whole string (foo.bar.Baz) as variable.annotation makes a lot of sense, since its purpose in an annotation is really entirely unlike its purpose in normal code. It's not normal code. It literally runs at compile time, and its value is baked into the binaries with a special encoding.

@FichteFoll Here's a redone screenshot with a couple variants, vertically aligned (note that I still haven't modified my color scheme to do anything special with any of the annotation scopes):

The parentheses have no impact. I just added them to demonstrate that the Scala mode doesn't use the variable.function scope.

[FichteFoll]

I mentioned this on IRC too, but I wonder about whether or not meta.annotation.identifier should be used for the entire path ("qualified name" in Python) since identifiers are, to me, always atomic. As such, I settled with just meta.annotation for the entire thing, , meta.annotation.function for a decorator function call and meta.annotation.arguments/parameters for potential arguments, while the last non-special identifier in the path gets variable.annotation or variable.annotation.function for function calls.

The following tests should make this more clear:

@ normal . decorator
# <- meta.annotation punctuation.definition.annotation
#^^^^^^^^^^^^^^^^^^^ meta.annotation
# ^^^^^^^^^^^^^^^^^^ meta.qualified-name
# ^^^^^^ meta.generic-name - variable.annotation
#          ^^^^^^^^^ variable.annotation
#        ^ punctuation.accessor.dot
class Class():

    @functools.wraps(method, 12)# comment
#^^^ - meta.annotation
#    ^^^^^^^^^^^^^^^^^^^^^^^^^^^ meta.annotation.function
#   ^ punctuation.definition.annotation
#    ^^^^^^^^^^^^^^^ meta.qualified-name
#    ^^^^^^^^^ meta.generic-name - variable.annotation
#             ^ punctuation.accessor.dot
#              ^^^^^ variable.annotation.function
#                   ^ punctuation.section.arguments.begin
#                              ^ punctuation.section.arguments.end
#                               ^^^^^^^^^ comment
    def wrapper(self):
        return self.__class__(method)

    @deco #comment
#^^^ - meta.annotation
#         ^^^^^^^^ comment

    @deco[4]
#        ^ invalid.illegal.character

    @deco \
      . rator
#     ^^^^^^^ meta.annotation
#     ^ punctuation.accessor.dot

    @ deco \
      . rator()
#     ^^^^^^^ meta.annotation.function
#       ^^^^^ meta.annotation.function variable.annotation.function

    @ deco \
#     ^^^^ meta.qualified-name meta.generic-name - variable.annotation
#          ^ punctuation.separator.continuation.line

[deathaxe]

From @FichteFoll:

... Ideally the color scheme should target punctuation.definition.annotation and highlight it however it wants to (keyword.other* in this case), but we all know that most color schemes will never implement it.

I think the first and most important step is to clearly define the set of scopes and then care about color schemes. The other way round authors always try to hack their color schemes to provide the "best" look and feel with the set of available scopes for certain languages, which were not very consistent between the different languages in the past.

Some color schemes implement special colors for single languages to provide best experience for JavaScript or HTML developement but looking ugly for others. This can be avoided by clearly defining and promoting language independed scopes, only.

IMHO: Users will always use color schemes which provide best experience to them and I am quite sure popular color schemes will be updated somehow. Give evolution a chance ;-)

[deathaxe]

As a conclusion from @djspiewak's description the whole @... stuff is just like a preprocessor definition in C or C++ und therefore should be treated like this. Means the different parts should get their own scopes but should not look like normal qualifiers. They should differ from normal code.

I vote for @ beeing scoped with punctuation.definitinon.annotation as it looks very like the introduction of a comment (//) or precompiler definition (see #) which are or should be scoped with punctuation, too.

@keith-hall: To quote from Sublime Text Documentation:

Meta scopes are used to scope larger sections of code or markup, generally containing multiple, more specific scopes. These are not intended to be styled by a color scheme, but used by preferences and plugins.

You should never ever encourage color scheme authors to create style definitions for meta-scopes. Even though I am not a friend of differen background colors for certain scopes, this might me the only valid exception for themed meta scopes.

[FichteFoll]

Maybe the exclusion of meta scopes for color highlighting is aging as well. Or at least for all meta scopes.

For string interpolation, we started highlghting the entire construct as meta.string but only the non-interpolated parts get string while the interpolated parts get ... something else (not at the top of my head). The latest example for this should be Python, where I added this in early December.

[deathaxe]

I don't want to hijack this issue but with a look on the python package I see meta.string.python always beeing used with a string.quoted next to it. This means to me meta.string simply declares the whole section as a selector for snippets/completions but the default color for the content is defined by string.quoted.

As a result the whole content has a meta-scope and can have different color-scopes with string.quoted as a default. This is exactly correct and doesn't break the rule not to highlight meta-scopes from my perspective.

This should keep the rule for meta in future I think. Clear rules are the base to automatically check color schemes and/or syntax definitions to ensure the least requirements are met.

The only way a meta scope should be allowed to be used for coloring is to create a different set of colors for existing keywords. So an integer value can have a different accent color when used in a string then it has in normal code. This should be encouraged so the user can clearly identify it as part of a string or part of normal code.

[keith-hall]

I personally think that either we give allowance for certain meta scopes to be targeted by color schemes (think JSON keys https://github.com/sublimehq/Packages/issues/421#issuecomment-221530223) or we rethink our scoping strategy to introduce non-meta scopes that color schemes can optionally target.

I do like your idea of automating color scheme conformance checks though, @deathaxe 👍

[djspiewak]

@keith-hall We basically have to give allowance for certain meta scopes to be targeted in some way, otherwise the proposed annotation scoping scheme simply doesn't work at all for color schemes that prefer to cover the entire thing. As an example:

The above is impossible to achieve without targeting meta.annotation. I find that very very annoying, but it's necessary to achieve the coloring that makes the most sense.

[FichteFoll]

@djspiewak the string.quoted scope gets removed here. It's a bit complex because format specifiers can be nested, but works nicely.

[deathaxe]

... otherwise the proposed annotation scoping scheme simply doesn't work at all ... @djspiewak: Why not? If the different elements can have different colors they can have the same, too.

A color definition could address variable.annotation for all normal elements.

The problems are with punctuations as they would need some kind of annotation specialization

Using meta.annotation punctuation. to color them would work without breaking the rule. This way the meta. scope is used as a selector to distinguish between normal punctuations and those within a meta scope but not as highlighted scope itself. This may be compared to embedded syntaxes somehow as metas allow a complete set of colors for certain sections of code.

You are right, if you say the result is the same as coloring meta.annotation directly, but it feels a bit like anarchy to me. "Meta" is some kind of hidden information to be used behind the scenes.

---

I vote for meta scopes being used for contextual reasons only as descriped in the current official Documentation. I just found the following good advice at the end of ST3's Documantation:

When styling scopes, resist the urge to directly style meta scopes. They are primarily intended to provide contextual information for preferences and plugins.

[djspiewak]

A color definition could address variable.annotation for all normal elements.

Nope.

  @scala.annotation.tailrec
// ^^^^^ - variable.annotation
//       ^^^^^^^^^^ - variable.annotation

[FichteFoll]

I made my color scheme highlight meta.annotation with a background color and nobody is gonna stop me from doing that!

[djspiewak]

@FichteFoll I think the objection I have is more that we're forced to target the meta scope if we want to color the whole thing.

[wbond]

@djspiewak I think the idea is that coloring the whole thing as one color is not the preferred way we are intending it to be used, instead we are intending for color schemes to color the individual parts. However, if someone wants to jam the square peg in the round hole, they certainly can.

In some syntaxes, we have a meta.path scope over a series of "labels" that include separators also. Again, I don't think that generally we want separators to be scoped the same color as "labels", but meta.path allows users to target a sequence of labels and separators, aka a "qualified identifier".

If we are going to make further changes, I would think perhaps adding meta.path to the qualified identifier in the annotation would make sense.

Either way, it is clear that there are two major ways that color scheme authors may want to style qualified identifiers in annotations, and baring a clear technically-superior choice, I think it'll come down to what I think we want the "default" to be.

[gwenzek]

Here is my usecase, I know it's not common, but I thought it would be interesting to keep in mind, when thinking about what we want the user to be able to do.

Since the new syntax definition has been added I customized my color scheme and the syntaxes I'm using the most to get something like:

The main original point is that: the punctuation is matching the color of the related identifier. So the parenthesis delimiting the parameters of get_title are in the same color that get_title (orange), as well as the punctuation delimiting the code of get_title (in python it's only the :, in other languages it would be {}).

The parenthesis of a tuple doesn't receive the same color than parenthesis of a function call. It's pretty useful to distinguish between a np.zeros(5, 8) (which fail at run time) and np.zeros((5, 8)) (which creates a (5, 8) matrix).

I totally understand that not going to be the "default" but currently it's really complex to implement. So I'd be happy if this use case was keep in mind when writing syntaxes. The main thing to do is to not over reuse context so different parenthesis/bracket can have a scope depending on who pushed them. This would probably be easier with small improvements to the "sublime-syntax" syntax (like being able to pass a scope name to a context when pushing it).

To illustrate the complexity of writing correct color schemes, below is a snippet of my SCss color scheme for the annotation rule. I recently wrote it for Python (version from build 3125), and as you can see it's not pretty.

// Annotations

storage.modifier.annotation,
meta.statement.decorator meta.function-call,
meta.statement.decorator meta.function-call variable.function,
keyword.other.decorator,
variable.annotation,
{
  foreground: $annotations;
  fontStyle: bold;

  meta.function-call.arguments,
  {
    foreground: $foreground;
    fontStyle: none;
  }
}

meta.statement.decorator meta.function-call meta.function-call.arguments variable.function
{
  foreground: $function_call;
  fontStyle: none;
}

punctuation.separator.annotation,
punctuation.section.annotation,
meta.statement.decorator meta.function-call punctuation.section.arguments,
meta.statement.decorator meta.function-call punctuation.separator.arguments,
{
  foreground: $annotations;
}

meta.statement.decorator meta.function-call meta.function-call.arguments punctuation.section.arguments,
meta.statement.decorator meta.function-call meta.function-call.arguments punctuation.separator.arguments,
{
  foreground: $function_call;
}

Note that this is a feature that is only possible in ST, I don't know any other editor where you have such control over the syntaxes, (tm-language isn't powerful enough most of the time) and that easy to hack. This will make me stick with Sublime for a long time (even i f I have to fork syntaxes for all languages I use :-) )

Anyway, thanks to the ST team for the good work and for taking the time to think about this design issue and to ask the community, Keep making ST awesome !

[FichteFoll]

This would probably be easier with small improvements to the "sublime-syntax" syntax (like being able to pass a scope name to a context when pushing it).

You can push two contexts with the lower context just providing a meta_scope and popping immediately. See also https://github.com/FichteForks/Packages/commit/fa29a796d2fe681ad21f77cbb0b0c8224cd8e3c5 and https://github.com/FichteForks/Packages/commit/3e9d9b146db49f4f418f6bd276e0a25e5ff21287.

[wbond]

So I don't think we were ever able to come up with a plan for these. I made a comment at https://github.com/sublimehq/Packages/issues/1842#issuecomment-464185639 that is relevant, but would apply to more than just annotation qualifiers.