IN PROGRESS: Add support for PowerShell Editor Services

[daviwil]

This change enables basic integration of PowerShell Editor Services with the PowerShell plugin to provide rich editing functionality for PowerShell scripts. This change adds code completions (IntelliSense), syntax and semantic markers, basic script execution, and some level of PowerShell-based extensibility to Sublime Text.

I'm sending this PR out before it's fully done so that we can talk about the implementation details. The code is not "production quality", there are a lot of TODOs around and some code is commented out. I'll clean all of that up before merging. The most important thing is to take a look at the code design and let me know if you see any big problems.

I would love it if someone could try this out and see if it works for you. I've included installation instructions in the README.md file.

Let me know what you think!

⚠️ Issues to be fixed before merge ⚠️

Here's a list of known issues that I still need to fix before this PR is merged:

• [ ] IntelliSense sometimes inserts the incorrect substring. This will be corrected soon.
• [ ] There are no keybindings for the commands that come with the plugin. We are investigating the issue.
• [ ] Can't hover over syntax/semantic markers in the script and see the message. This is a limitation in Sublime Text's APIs and should be resolved in an upcoming Sublime update.
• [ ] PowerShell process do not get closed when Sublime exits. We need to wire up a shutdown handler to properly kill the PowerShell subprocess upon shutdown.
• [ ] Clean up the code (remove or file bugs for TODOs, get rid of commented out sections)

[nosami]

I got the following message on startup in OSX

2016-08-28 22:06:21,578: 140735206535168: INFO: PowerShell plugin starting...
2016-08-28 22:06:21,581: 140735206535168: DEBUG: PowerShell Editor Services log path: /var/folders/h4/hwtcfdnd7_718x5hzvl12y180000gn/T/1472418381-EditorServices.log
2016-08-28 22:06:21,582: 140735206535168: DEBUG: Using bundled modules path: /Users/jason/Library/Application Support/Sublime Text 3/Packages/PowerShell/editorservices/modules
2016-08-28 22:06:21,582: 140735206535168: DEBUG: Command parameter arg: & "/Users/jason/Library/Application Support/Sublime Text 3/Packages/PowerShell/editorservices/Start-EditorServices.ps1" -EditorServicesVersion "0.7.1" -HostName "Sublime Text Host" -HostProfileId "SublimeText" -HostVersion "3.0.3114" -LogLevel "Verbose" -LogPath "/var/folders/h4/hwtcfdnd7_718x5hzvl12y180000gn/T/1472418381-EditorServices.log" -BundledModulesPath "/Users/jason/Library/Application Support/Sublime Text 3/Packages/PowerShell/editorservices/modules"
2016-08-28 22:06:22,764: 140735206535168: DEBUG: Launch response: b'needs_install\n'
Traceback (most recent call last):
  File "./python3.3/json/decoder.py", line 367, in raw_decode

But I did install the modules

✔ ~/Library/Application Support/Sublime Package 3/Packages/PowerShell/editorservices/modules [editor-services|…25]
22:41 $ tree
.
├── PSScriptAnalyzer
│   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.dll
│   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll
│   ├── PSScriptAnalyzer.psd1
│   ├── PSScriptAnalyzer.psm1
│   ├── PSv3
│   │   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.dll
│   │   └── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll
│   ├── ScriptAnalyzer.format.ps1xml
│   ├── ScriptAnalyzer.types.ps1xml
│   ├── Settings
│   │   ├── CmdletDesign.psd1
│   │   ├── DSC.psd1
│   │   ├── PSGallery.psd1
│   │   ├── ScriptFunctions.psd1
│   │   ├── ScriptSecurity.psd1
│   │   └── ScriptingStyle.psd1
│   ├── coreclr
│   │   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.dll
│   │   └── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll
│   └── en-US
│       ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll-Help.xml
│       └── about_PSScriptAnalyzer.help.txt
└── PowerShellEditorServices
    ├── PowerShellEditorServices.psd1
    ├── PowerShellEditorServices.psm1
    └── bin
        ├── Desktop
        │   ├── Microsoft.PowerShell.EditorServices.Host.dll
        │   ├── Microsoft.PowerShell.EditorServices.Protocol.dll
        │   ├── Microsoft.PowerShell.EditorServices.dll
        │   └── Newtonsoft.Json.dll
        └── Nano
            └── Microsoft.PowerShell.EditorServices.Nano.dll

I ended up copying the Packages folder to another folder that didn't contain a space, and hacked your code to point to the new location, then it worked great!

[daviwil]

Damn, thanks for catching that! After having been bitten by this a couple times already I really should know better ;)

[daviwil]

Oh, and thanks a lot for giving it a try!

[nosami]

Completion seems a little iffy though... is that a problem with the Sublime code or the backend server?

[nosami]

e.g.

[nosami]

Guess this has to be a sublime issue - if the completions are returned correctly, the server did it's job.

[daviwil]

Ahhh yeah, looks like this particular case may be a bug in the logging I'm using there. Will get that fixed :)

In general though, the completions can be a little wrong at times because of the way I have to translate the completions into buffer edits. Sublime's API assumes that the completion will literally just be the rest of the symbol name after the cursor but PowerShell often returns completions that actually replace some of the existing text. I'm having to do some extra work there to manipulate the buffer after the completion is committed so it's not 100% reliable yet.

Did you have to do a similar thing for OmniSharp? I know the TypeScript package for Sublime has to do it this way, I've been learning a lot from their implementation.

[nosami]

I didn't work on the OmniSharp sublime plugin (apart from some guidance), so not sure. That API sounds a bit sucky though :) I did work on emacs, vim and Atom intellisense though, and none of those had that issue - they just did the right thing.

[daviwil]

Sublime's API is the most inflexible that I've seen, hoping that the other editors will be less challenging ;)

[nosami]

Just remembered that with all those editors, you need to tell the editor what the 'prefix' is - the part of the line that you are completing on / part to be replaced

[daviwil]

Hmmm, is that something I have to know in advance before resolving completions? Am I supposed to use the editor's tokenization of the source to pre-determine the prefix string, or can I give it the prefix after the user has selected the completion they want?

[nosami]

Vim prefix lookup code just uses a regex - https://github.com/OmniSharp/omnisharp-vim/blob/master/autoload/OmniSharp.vim#L17-L28

[dmoore44]

Not sure if I'm missing something, but I can't seem to get the Editor Services up and running...

I've got the module installed: PS /Users/dmoore44/Documents> ./Setup-EditorServices.ps1 /Users/dmoore44/Documents/Setup-EditorServices.ps1 : The PowerShell package already exists at the following path: /Users/dmoore44/Library/Application Support/Sublime Package 3/Packages/PowerShell. We do not automatically delete this path in case you need to back up your current installation. Please move or delete this path before continuing. At line:1 char:1

• ./Setup-EditorServices.ps1

• 
  - CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
  - FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Setup-EditorServices.ps1

Or: PS /Users/dmoore44/Documents> tree "/Users/dmoore44/Library/Application Support/Sublime Package 3/Packages" /Users/dmoore44/Library/Application\ Support/Sublime\ Package\ 3/Packages └── PowerShell ├── CONTRIBUTING.md ├── ISE\ Nostalgia.tmTheme ├── LICENSE.txt ├── PowerShell.sublime-project ├── README.md ├── Snippets │   ├── (beg)\ scriptblock\ begin\ clause.sublime-snippet │   ├── (catch)\ catch\ {...}.sublime-snippet │   ├── (comm)\ comment\ block.sublime-snippet │   ├── (elif)\ elseif\ {...}.sublime-snippet │   ├── (elifi)\ elseif\ in-line.sublime-snippet │   ├── (else)\ else\ {...}.sublime-snippet │   ├── (elsei)\ else\ in-line.sublime-snippet │   ├── (end)\ scriptblock\ end\ clause.sublime-snippet │   ├── (fil)\ new\ filter.sublime-snippet │   ├── (fin)\ finally\ {...}.sublime-snippet │   ├── (fun)\ define\ function.sublime-snippet │   ├── (here)\ heredoc\ string\ (single\ quoted).sublime-snippet │   ├── (hhere)\ heredoc\ string\ (double\ quoted).sublime-snippet │   ├── (if)\ if\ {...}.sublime-snippet │   ├── (ifelse)\ if\ {...}\ else\ {...}.sublime-snippet │   ├── (ifi)\ if\ in-line.sublime-snippet │   ├── (mydocs)\ path\ to\ My\ Documents.sublime-snippet │   ├── (proc)\ scriptblock\ process\ clause.sublime-snippet │   ├── (sd)\ enable\ script\ debug\ stepping.sublime-snippet │   ├── (sub)\ $(...)\ complex\ subexpression.sublime-snippet │   ├── (thisdir)\ path\ to\ current\ script.sublime-snippet │   ├── (tmp)\ create\ temp\ file.sublime-snippet │   ├── (try)\ try\ {...}.sublime-snippet │   └── Powershell.sublime-completions ├── Support │   ├── Comments.tmPreferences │   ├── Default.sublime-keymap │   ├── PowerShell.sublime-commands │   ├── Powershell.sublime-build │   ├── PowershellSyntax.sublime-settings │   ├── PowershellSyntax.tmLanguage │   └── Preferences.sublime-settings ├── Symbol\ List\ Functions.tmPreferences ├── appveyor.yml ├── bin │   ├── BuildFolder.ps1 │   ├── Config.ps1 │   ├── Generate-BuiltInVariablePattern.ps1 │   ├── Generate-BuiltinCommandletPattern.ps1 │   ├── Publish.ps1 │   ├── builder.py │   ├── check.py │   ├── make_version.py │   ├── release.tmpl │   └── toplist.py ├── editorservices │   ├── Setup-EditorServices.ps1 │   ├── Start-EditorServices.ps1 │   ├── init.py │   ├── client.py │   ├── commands.py │   ├── editor.py │   ├── editor_context.py │   ├── file.py │   ├── listeners.py │   ├── logger.py │   ├── messages.py │   ├── model.py │   ├── modules │   │   ├── PSScriptAnalyzer │   │   │   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.dll │   │   │   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll │   │   │   ├── PSScriptAnalyzer.psd1 │   │   │   ├── PSScriptAnalyzer.psm1 │   │   │   ├── PSv3 │   │   │   │   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.dll │   │   │   │   └── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll │   │   │   ├── ScriptAnalyzer.format.ps1xml │   │   │   ├── ScriptAnalyzer.types.ps1xml │   │   │   ├── Settings │   │   │   │   ├── CmdletDesign.psd1 │   │   │   │   ├── DSC.psd1 │   │   │   │   ├── PSGallery.psd1 │   │   │   │   ├── ScriptFunctions.psd1 │   │   │   │   ├── ScriptSecurity.psd1 │   │   │   │   └── ScriptingStyle.psd1 │   │   │   ├── coreclr │   │   │   │   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.dll │   │   │   │   └── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll │   │   │   └── en-US │   │   │   ├── Microsoft.Windows.PowerShell.ScriptAnalyzer.dll-Help.xml │   │   │   └── about_PSScriptAnalyzer.help.txt │   │   └── PowerShellEditorServices │   │   ├── PowerShellEditorServices.psd1 │   │   ├── PowerShellEditorServices.psm1 │   │   └── bin │   │   ├── Desktop │   │   │   ├── Microsoft.PowerShell.EditorServices.Host.dll │   │   │   ├── Microsoft.PowerShell.EditorServices.Protocol.dll │   │   │   ├── Microsoft.PowerShell.EditorServices.dll │   │   │   └── Newtonsoft.Json.dll │   │   └── Nano │   │   └── Microsoft.PowerShell.EditorServices.Nano.dll │   ├── protocol.py │   ├── settings.py │   └── utils.py ├── main.py ├── manifest.json ├── messages │   └── 2.x.txt ├── messages.json └── tests ├── pester │   ├── Syntax.Tests.ps1 │   └── SyntaxHelper.psm1 ├── py │   ├── init.py │   └── syntax_def │   ├── init.py │   ├── test_syntax.py │   └── test_token_gen.py └── samples ├── localhost_01.mof └── test-file.ps1

21 directories, 100 files

But I don't see any refs to the module being loaded in the Sublime console logs: startup, version: 3114 osx x64 channel: stable executable: /Applications/Sublime Text.app/Contents/MacOS/Sublime Text working dir: / packages path: /Users/dmoore44/Library/Application Support/Sublime Text 3/Packages state path: /Users/dmoore44/Library/Application Support/Sublime Text 3/Local zip path: /Applications/Sublime Text.app/Contents/MacOS/Packages zip path: /Users/dmoore44/Library/Application Support/Sublime Text 3/Installed Packages ignored_packages: ["Vintage"] pre session restore time: 0.205306 using gpu buffer for window startup time: 0.281167 first paint time: 0.309568 reloading plugin Default.auto_indent_tag reloading plugin Default.block reloading plugin Default.comment reloading plugin Default.convert_syntax reloading plugin Default.copy_path reloading plugin Default.delete_word reloading plugin Default.detect_indentation reloading plugin Default.duplicate_line reloading plugin Default.echo reloading plugin Default.exec reloading plugin Default.fold reloading plugin Default.font reloading plugin Default.goto_line reloading plugin Default.history_list reloading plugin Default.indentation reloading plugin Default.kill_ring reloading plugin Default.mark reloading plugin Default.new_templates reloading plugin Default.open_context_url reloading plugin Default.open_file_settings reloading plugin Default.open_in_browser reloading plugin Default.pane reloading plugin Default.paragraph reloading plugin Default.paste_from_history reloading plugin Default.profile reloading plugin Default.quick_panel reloading plugin Default.run_syntax_tests reloading plugin Default.save_on_focus_lost reloading plugin Default.scroll reloading plugin Default.set_unsaved_view_name reloading plugin Default.show_scope_name reloading plugin Default.side_bar reloading plugin Default.sort reloading plugin Default.swap_line reloading plugin Default.switch_file reloading plugin Default.symbol reloading plugin Default.transform reloading plugin Default.transpose reloading plugin Default.trim_trailing_white_space reloading plugin CSS.css_completions reloading plugin Diff.diff reloading plugin HTML.encode_html_entities reloading plugin HTML.html_completions reloading plugin 0_package_control_loader.00-package_control reloading plugin 0_package_control_loader.02-bz2 reloading plugin Package Control.1_reloader reloading plugin Package Control.2_bootstrap reloading plugin Package Control.Package Control plugins loaded Package Control: Skipping automatic upgrade, last run at 2016-08-31 11:27:41, next run at 2016-08-31 12:27:41 or after

[nosami]

Did you add "powershell_language_services_enabled": true to your user preferences?

[daviwil]

@dmoore44 damn it, I think I screwed up the script for OS X, looks like the package got installed here:

/Users/dmoore44/Library/Application Support/Sublime Package 3/Packages

Path should be "Sublime Text 3". If you move Sublime Package 3/Packages/PowerShell to /Users/dmoore44/Library/Application Support/Sublime Text 3/Packages does it work?

[dmoore44]

@daviwil You're correct - changing the path to ../Sublime Text 3/Packages works! I was wondering if that would be the solution, but had to step away for a bit.

No worries on the install script - I have to say that I'm quite impressed with the level of functionality on an "Alpha" project.

[daviwil]

Phew, glad to hear it :) I'll fix that script so that the next person doesn't hit the same issue.