Distinction of end keywords

[graceful-potato]

• Added meta scopes for every structure which closes with end so now it is possible to distinguish end keyword between class, module, def, do, if, unless, while, until, for, begin and highlight these structures differently.

• Move rescue and ensure to separate rules so it possible to highlight them depending on whether they are inside the block or inside the method

• Removed equals sign from the scope string.unquoted.heredoc.ruby in front of heredoc.

  string = <<FOO
  this is my string
  FOO

[noniq]

This looks promising! However, it makes the already rather complex grammar for Ruby even more complex. I’m not really sure if it’s really worth the cost. Maybe @infininight or @sorbits can have a look?

[sorbits]

If it can actually work, it might add some value to recognize these structures, but with trailing if/while/unless’s etc. I’m rather skeptical.

So at least I would like to see a test file for this PR, also, the commits should be squashed, we do not want to pull commits that are later followed by fixups, optimizations, typo fixes, etc.

[graceful-potato]

@sorbits sorry, but what is this test supposed to look like? Is it some kind of text file with various examples of ruby valid code containing these structures? I would also like to know if there is a way to make benchmarks and see how these changes can affect the syntax highlighting performance.

[kiendang]

+1 for this. Would help a lot in syntax highlighting. Currently it is not possible to highlight def and do differently with them also matching their end.

[graceful-potato]

@noniq, @sorbits Hi, i think i resolved issues, and now it looks like if/unless/while/until works nice even after regex and division. Also i fixed while and for which could use optional do keyword. I'm not sure what did you mean by test file but i created a file where i write down some code with all these structures which can help test all scopes. Anyway, i'd be happy if you take a look and give some feedback.

[graceful-potato]

@kiendang yes, it's not possible and thats why i started this PR. Unfortunately, it turned out to be harder than i thought. At this point i think i fixed all bugs which i knew. And it will be very helpful if you or anybody else could start using ruby bundle from this branch in order to find bugs

[graceful-potato]

Also I created a theme for textmate where module, class and def structures highlights with another color. It could help in testing. If one of these structures closing with red colored end that means something wrong with grammar file.

[graceful-potato]

Commits has been squashed

[graceful-potato]

Split if-unless and while-until rules into if, unless, while and until. I was a bit concern about performance of these regexes and i didn't find any way to benchmark it in textmate, but i found how can i benchmark it using vscode-texmate library. I know that it is very different thing, but I didn't find anything better. So i tested 3 grammar files with this library and got strange results. Splitted version is faster then version with combined rules. Here is the results:

Original ruby grammar file
Tokenizing 595 lines
Took 49.9 milliseconds
---------------------------------------------------------------------
Ruby grammar file with combined if/unless and while/until rules
Tokenizing 595 lines
Took 118.7 milliseconds
---------------------------------------------------------------------
Ruby grammar file with splited if/unless and while/until rules
Tokenizing 595 lines
Took 89.1 milliseconds
---------------------------------------------------------------------

You can check out test here - https://github.com/graceful-potato/test_ruby_grammars

[graceful-potato]

According to results of this benchmark i optimized complex regexes and get a pretty good improvement. I am pretty sure this benchmark is not ideal and for some reason first test always show slower result, but anyway it is better than nothing.

Ruby grammar file with combined if/unless and while/until rules
Tokenizing 611 lines
Took 169 milliseconds
---------------------------------------------------------------------
Original ruby grammar file
Tokenizing 611 lines
Took 72 milliseconds
---------------------------------------------------------------------
Ruby grammar file with splited if/unless and while/until rules
Tokenizing 611 lines
Took 120 milliseconds
---------------------------------------------------------------------
Optimized Ruby grammar file
Tokenizing 611 lines
Took 85 milliseconds
---------------------------------------------------------------------

Update I broke some rules with this optimization. Will fix it soon.

[graceful-potato]

Fixed optimization. It slowed down performance a bit, but anyway it is faster then it was before

Ruby grammar file with combined if/unless and while/until rules
Tokenizing 611 lines
Took 168 milliseconds
---------------------------------------------------------------------
Original ruby grammar file
Tokenizing 611 lines
Took 78 milliseconds
---------------------------------------------------------------------
Ruby grammar file with splited if/unless and while/until rules
Tokenizing 611 lines
Took 117 milliseconds
---------------------------------------------------------------------
Optimized Ruby grammar file
Tokenizing 611 lines
Took 93 milliseconds
---------------------------------------------------------------------

[graceful-potato]

I've created a theme for vscode that uses this version of the grammar file. I hope it will help in testing. https://marketplace.visualstudio.com/items?itemName=GracefulPotato.gruvbox-ish