[Proposition] Create a separate repo for the grammar

[vors]

Dear friends, there were no releases and no active development of SublimeText/PowerShell since September 2015.

I would like to propose a "separate repo for the grammar" and explain, why I think it's a right step now.

In the meantime

PowerShell team is doing more and more development on GitHub in https://github.com/PowerShell.

Microsoft as a whole org doing more and more open-source development in https://github.com/Microsoft

New code editor VSCode emerged. VSCode is based on Atom. It has a great PowerShell support, driven by @daviwil (PS team member).

SublimeText/PowerShell Current state

SublimeText/PowerShell serves two functions:

• it's original purpose: a SublimeText specific plugin
• a place for the shared tmLanguage grammar

The second role is important: it allows us to converge and work together on the syntax highlighting, instead of reinventing the wheel in different places. This tmLanguage grammar is used in

• SublimeText
• GitHub
• Atom
• VSCode

It's dual-role start hurts:

• I was stopping @guillermooo from switching to a new Sublime grammar engine, because then we would lose the point to converge.
• Changes need to be made more carefully, because of the differences in tmLanguage engines in different editors.
• My unit test system became an over-complicated monster that discourage people from contributions. Tests are failing and it's impossible to set-up a local env to troubleshoot them.

  Proposition

I would like to create a dedicated repo, i.e. https://github.com/PowerShell/grammars, and move tmLanguage development there.

More details:

• I would like to include all .tmLanguge, .cson / .json, .yml formats of grammars and add a script that keep them in sync (and CI test that verifies it), so people would not need to mess around with converters.
• For testing I would like to use something editor-agnostic, like https://github-lightshow.herokuapp.com/
• I will use git filter-branch to keep git history on the .tmLanguage file.
• Every significant "release", maintainers will notify all interested parties that update is available (probably in the form of PR).
• I will remove burndown of powershell-based tests from this repo.

  Questions

• What do you think about this plan?
• Are you ok with hosting it in PowerShell github org?

@guillermooo @Jaykul @bcrotty

[guillermooo]

Fine by me! I can't devote any time to this at the moment anyway... :-/

[daviwil]

:+1: This would be a great thing for future evolution and maintenance of the grammar.

[vors]

cc @jugglingnutcase

[Jaykul]

Makes sense to me.

[jrsconfitto]

I'm on board too, sounds like a great idea to me.

FWIW: Because I'm lazy, I often avoided messing too much with this repo because I didn't want to have to install all the SublimeText artifacts to make sure my tests pass, etc. On Sun, Feb 21, 2016 at 1:43 AM Joel Bennett notifications@github.com wrote:

Makes sense to me.

— Reply to this email directly or view it on GitHub https://github.com/SublimeText/PowerShell/issues/148#issuecomment-186761027 .

[bcrotty]

That sounds good to me. It would be great to combine manpower and resources with other grammar developers. Have we heard from any of them? Do we have plans for merging projects from other platforms?

Is there any reason to look at using the new Sublime grammar engine or making some sort of an automatic converter? Or would it be best to just stick to tmLanguage?

[vors]

To the best of my knowledge, all people who did any recent development of tmLanguage-compatible PowerShell grammar are on this thread.

Specifically for ST, I would prefer to stick with tmLanguage. But if somebody will do the work and result would be superior, we should consider switch imo.

[sgtoj]

Is there any updates on the status of the change? I would like to contribute. However, there is very little documentation.

[vors]

@sgtoj glad to hear it

I'm didn't have a chance to execute this plan yet. Stay tuned.

[vors]

UPD: @daviwil start putting things together in https://github.com/PowerShell/EditorSyntax

[daviwil]

Just got the repo kicked off! We'll gradually start moving all known issues with the existing grammar over to the PowerShell/EditorSyntax repo. Tracking this effort in issue #1 there

[keith-hall]

It'd be great to know what the current state of this ST package is - the EditorSyntax repo referenced above mentions that this SublimeText/PowerShell repo uses the syntax definition from that package, which is blatantly untrue - perhaps this package should be updated to do so? Also, if it were, what would this package provide that the EditorSyntax doesn't? i.e. what is in this package that makes it worth using over just directly using EditorSyntax? From what I can see from a quick glance, EditorSyntax is inexplicably missing .tmPreferences files to handle indentation, commenting and goto definition etc. but as these are tied to the syntax def, surely they should be in EditorSyntax instead of this package, as they could get out of sync? Apart from that, this package just supplies a color scheme, no?

[vors]

The EditorSyntax currently has too many regressions compare to this package and doesn't suite as a replacement just yet. Ideally we would fine a good way to prevent regressions https://github.com/PowerShell/EditorSyntax/issues/2 with tests.

I think grammar is the biggest value at the moment, there are few other additions like color scheme, snippets, preferences, but I'm not sure how much are they used.

If you are interested in more details join the vscode-contributors channel on community slack http://slack.poshcode.org/

It would be very good to move all this project to the convergence point, I don't think it's on somebody's P0 list unfortunately.

[vors]

I just realized I didn't answer your first questison

what the current state of this ST package is

It's not going to be directly developed, instead the effort should be on the bringing EditorSyntax to the acceptable state and replacing the grammar here.

In the year, when I was actively developing and maintaining sublime package, I got pretty frustrated by the fact that there is no good way to guarantee that changes don't make regressions. That makes small fixes very labor intensive and too much context needed. So I don't think we should continue this route, even if there are some obvious problems with the current grammar. Everything is too fragile without a way to test it.

[vors]

Update: Thanks to @omniomi and others for recent kick-ass work on https://github.com/PowerShell/EditorSyntax at this point we are going to be provably in a much better shape if we switch to common grammar. Here is the proof: https://github.com/SublimeText/PowerShell/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3A%22Fixed+in+EditorSyntax%22+ a whole bunch of issues just going to be resolved.

I also spot checked our main test.ps1 file and a minor issues on the EditorSyntax repo: https://github.com/PowerShell/EditorSyntax/issues/103

EditorSyntax also already has unit tests in the form of common spec files, which is an amazing win.

Super-excited for the future! 🎉 🎉 🎉