Add token scopes to TextRunProperties

[NickRimmer]

Hey, I didn't find how to get token scopes in editor. So there is an idea (;

During text decorating with TextTransformation, we already have token information here. So, we only need to include details to element TextRunProperties which we can read later whenever we need.

I didn't add any special method to read scopes, I use existing element properties:

private void Caret_PositionChanged(object sender, EventArgs e)
{
    var scopes = _textMateInstallation.EditorModel.GetTokenScopes(
        _textEditor.TextArea.Caret.Line - 1,
        _textEditor.TextArea.Caret.Column - 1);

    _statusTextBlock.Text = string.Format("Line {0} Column {1}. Scopes: {2}",
        _textEditor.TextArea.Caret.Line,
        _textEditor.TextArea.Caret.Column,
        scopes.Count > 0 ? string.Join(", ", scopes) : "None");
}

[danipen]

I don't think it's a good idea to set the TextMate tokens scopes in the VisualLines since the TextMate integration is optional and also for duplicating the in memory.

Probably would be better to enable some API in the TextMate installation to get the tokens for a certain offset.

[danipen]

The line tokens are available here: https://github.com/AvaloniaUI/AvaloniaEdit/blob/adbe72b8023207270d30c16beb4fadcd6d598023/src/AvaloniaEdit.TextMate/TextMateColoringTransformer.cs#L24

[NickRimmer]

I don't think it's a good idea to set the TextMate tokens scopes in the VisualLines since the TextMate integration is optional.

Probably would be better to enable some API in the TextMate installation to get the tokens for a certain offset.

Hm... yeah, let me think a bit. We can have just some extra dictionary with whatever here. But I'm surprised that TextMate is optional, as all examples and purpose of editor in general is syntax highlighting (;

[NickRimmer]

The line tokens are available here:

https://github.com/AvaloniaUI/AvaloniaEdit/blob/adbe72b8023207270d30c16beb4fadcd6d598023/src/AvaloniaEdit.TextMate/TextMateColoringTransformer.cs#L24

The idea to reuse tokens already calculated for text we showing in editor. Otherwise I can just get line/column indexes, load grammar and just calculate it again

[danipen]

But I'm surprised that TextMate is optional, as all examples and purpose of editor in general is syntax highlighting (;

TextMate is implemented as a plugin so the installation is optional and the dependencies are inverted (AvaloniaEdit doesn't know about TextMate so it doesn't make sense storing textmate tokens in the VisualLine)

[NickRimmer]

But I'm surprised that TextMate is optional, as all examples and purpose of editor in general is syntax highlighting (;

TextMate is implemented as a plugin so the installation is optional and the dependencies are inverted (AvaloniaEdit doesn't know about TextMate so it doesn't make sense storing textmate tokens in the VisualLine)

Yep, I agree, absolutely make sense

[NickRimmer]

I'm opened for ideas (:

I still think this is not a bad to store information in visual line, as it part of visual properties. And it easier to manage it here. Bur probably it should be as extra attributes, for whatever values can be read by anyone, not specifically TextMate

[danipen]

I think that what you need is just exposing the TMModel here as a public property: https://github.com/AvaloniaUI/AvaloniaEdit/blob/adbe72b8023207270d30c16beb4fadcd6d598023/src/AvaloniaEdit.TextMate/TextMate.cs#L37

Then, when the caret changes just access the TextMate installation and then access the TMModel to get the current line tokens and get the scopes from there.

[NickRimmer]

I think that what you need is just exposing the TMModel here as a public property:

https://github.com/AvaloniaUI/AvaloniaEdit/blob/adbe72b8023207270d30c16beb4fadcd6d598023/src/AvaloniaEdit.TextMate/TextMate.cs#L37

Then, when the caret changes just access the TextMate installation and then access the TMModel to get the current line tokens and get the scopes from there.

Thank you so much for your help, very appreciate it. Well, what I found, we don't even need to change anything in code ^^"

This code return already scopes:

 var scopes = _textMateInstallation
    .EditorModel
    .Get(_textEditor.TextArea.Caret.Line - 1)
    .Tokens
    .OrderBy(x => x.StartIndex)
    .LastOrDefault(x => x.StartIndex <= _textEditor.TextArea.Caret.Column - 1)?
    .Scopes ?? new List<string>();

So, I pushed small changes to get it a bit easier from EditorModel, but this is not so necessary anymore. I can close this PR if you agree that we don't need to change anything.

[NickRimmer]

I put solution here https://github.com/AvaloniaUI/AvaloniaEdit/discussions/408#discussioncomment-8954405