Add support for C# 7.2 keyword contexts

[analogrelay]

C# 7.2 added a number of new "keywords" (mostly just new places existing keywords can be used). These don't show up properly in GitHub colorization (see below) and the path of submodules led me here

See https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-7-2 for reference.

Examples:

// 'readonly' should be colorized like 'struct'
public readonly struct Foo
{
}

// 'ref' should be colorized like 'struct'
public ref struct Foo
{
}

// 'ref' should be colorized
public static ref Bar GetBar()
{
}

// 'in' works
// 'ref readonly' works
// 'private protected' works

[damieng]

Hey Andrew.

We've actually switched language-csharp over to an official Microsoft supplied grammar that has support for these alas github.com has not been able to upgrade to it yet as it uses some extensions the github highlighter does not support (named groups).

We're looking at whether we can manually or automatically change the grammar here so it can still work for github.com

[worldbeater]

Hello, Damien!

Could you please provide more info with examples describing what an unsupported named group is and how it should be replaced in order to support GitHub highlighters? Perhaps, the community and I can do all the routine work or write a script that will parse and fix all that stuff for you!

We love C# and we love GitHub, but we are quite disappointed with how current highlighter works. So we'll be glad to provide you any help we can 😸

[damieng]

@worldbeater I'm also disappointed (C# is my fave language). Our options right now are:

1. Go back to the old grammar and try and improve it in parallel with Microsoft's efforts. This was tricky in the past as the textmate grammar syntax is an unwieldy beast anyway. We'd probably want to fork this repo from the old one so Atom can keep using the Microsoft effort and we'd switch linguist over.

2. Figure out some kind of conversion between this and PCRE. I've tried doing this by hand and got nowhere fast. I suspect the older version of PCRE that GitHub supports might be missing the features we need entirely given my attempts.

3. Switch GitHub's linguist grammar for C# over to a textmate grammar - if somebody is maintaining one still.

4. Hold out for changes to linguist to support different syntax engines such as Oniguruma or tree-sitter.

[worldbeater]

Waiting for changes in GitHub internals can take time, so we can start working on existing C# TextMate bundle improvements. Some fixes already are here and here, other ones may not be so difficult to implement.

[Alhadis]

Careful pattern order, intelligent structure, and a little judicious repetition can all achieve the same effect as a named-group or a subroutine call. I've never needed these features in my grammars, ever.

I recently started work on a replacement JavaScript grammar for GitHub, which will include support for JSX and static Flow typing. I've barely spent much time on it and it's already showing improvement:

• Before
• After

You'll see that what most users deem impossible without Oniguruma extensions are, in fact, possible without any extensions.

[Alhadis]

Figure out some kind of conversion between this and PCRE. I've tried doing this by hand and got nowhere fast. I suspect the older version of PCRE that GitHub supports might be missing the features we need entirely given my attempts.

@damieng Could you point to an example where these features were used? I don't know a single thing about C#, so I'm in the dark when it comes to how bad its highlighting is on GitHub...

[damieng]

I'm not saying that they're not possible without extensions - I'm saying that in order for us to switch to the official Microsoft-maintained C# grammar we'd need Oniguruma support or a reliable conversion.

I think if GitHub wants to continue to tap the grammars used in clients this is going to be more and more of a problem if they use those features.

Do you know what version of PCRE is available? The features being used appear to be named captures but my attempt at conversion was not successful when I tested it.

[worldbeater]

@damieng please check https://github.com/github/linguist/issues/3291#issuecomment-381115027 (thanks a lot to John Gardner for providing info!) Perhaps, numbered groups instead of named groups can be used if (?&name) captures are not working?

[damieng]

Yeah I converted the whole grammar file by hand using that approach and it made no difference at least on the limited testing tool I have available.

On Fri, Apr 13, 2018, 8:02 AM Artyom notifications@github.com wrote:

@damieng https://github.com/damieng please check github/linguist#3291 (comment) https://github.com/github/linguist/issues/3291#issuecomment-381115027 (thanks a lot to John Gardner for providing info!) Perhaps, numbered groups instead of named groups can be used if (?&name) groups are not working?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/atom/language-csharp/issues/112#issuecomment-381163929, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHQp_TIYt26TmtEL8UyHeeJ-Zvpio-Kks5toL3-gaJpZM4SViL6 .

[worldbeater]

@damieng we finally managed to get Microsoft's grammar play well with GitHub. Here is a pull request with a working demo on Github Lightshow Heroku App. https://github.com/atom/language-csharp/pull/116 Hope our assistance could help GitHub continue being the best service for hosting open-source projects! 😹

[damieng]

This should now work fine in Atom - it will be some time however before github.com picks up the latest. I'll see what we can do to get that expedited.