Progress and road-map discussion

[gundermanc]

Roadmap

1. Light up colorization via VS TextMate (Easy) or custom parser (Medium).
2. Evaulate pros and cons of lighting up either Gopls language server or various open source Go utils for powering editor (Medium).
3. Implement selected editor integration (Medium to Hard depending on strategy).
4. Integrate debugger (Hard).
5. Integrate testing tools (Medium).
6. Integrate linting, code cleanup, advanced functionality (Medium).

Progress

• I’ve created an empty VSIX project under source control here: https://github.com/gundermanc/vs-go. Visual Studio already provides primitive colorization and outlining for Go via TextMate. We can gradually replace these features with our own as we make progress.
• I’ve installed Go SDK, downloaded Google’s official Gopls language server, and attempted to host it in VS. Gopls is very unstable and seems to give error messages on all LSP requests.
• I've started prototyping a from-scratch language service as well. Basic colorization is working via a quick and dirty lexer I wrote.
• Next steps are to do one or more of the following: o Debug into Gopls and determine the cause of the errors and submit a patch if needed. OR o Evaluate an alternative source of language smarts such as SourceGraph's Go language server, the OSS Go tools used by VS Code, or a from-scratch parser for the Go language service.

Strategies

Strategies for making progress in this project.

Host Gopls language server

Pros:

• Officially supported by Google
• Can light up a lot of features with a little bit of code via the Language Server Client for Visual Studio.
• I believe this offers support for the debugger for free.

Cons

• Gopls is currently very very alpha and unstable.
• Attempts to light it up thus far have been met with hard to decipher errors.
• UX fidelity is limited by what LSP and Gopls offer us. Gopls is unlikely to accept patches for providing VS specific additional data.
• Language Server client doesn't officially support VS Mac, so path to supporting VS Mac is uncertain. Using the xplat editor API, however, would be portable with no code changes.
• Harder to delegate to small contributions for editor layer (test, project, etc. can still be delegated).
• Work done here can't be released as an extension on the VS Gallery until Gopls stabilizes.

Host OSS Go tools used by VS Code

Pros

• Stable and battle tested via VS Code and other text editors.
• Composed of many parts, enabling delegation of feature-work and piecemeal light-up.
• Can be integrated with xplat editor API for day one support for VS Mac.
• More flexibility to customize and define the experience.
• Work done here can be immediately realized into an extension that can go on the VS Gallery.

Cons

• More third party components to verify and light up.
• No official Google support for tools.
• Community supporting tools may dry up once Gopls is stable and more widely adopted, rendering any work we do for the editor layer obsolete. Any testing, profiling, and project support may be able to live on unmodified since this comes from Go SDK tools.

From-scratch parser for Go + SDK Tools (godoc etc).

Pros

• Opportunity to build the first from-scratch language service that dual targets VS Windows and VS Mac.
• Opportunity to write a language parser from scratch (fun IMO 💯 )
• Much closer interaction with the language... literally reading and interpreting the spec.
• Cheap and fast light up of basic syntax driven editor features like colorization, outlining, structure guides, etc.
• Allows most granular customization of experience and more opportunities for distinguishing features relative to other editors.

Cons

• Playing catch up with the community tools that already solved a number of these problems.
• Requires much more maintenance.
• Harder to grow and evolve with the language.

Please feel free to add to the discussion 👍

[benhu-msft]

Hi Christian. Thank you for initiating this project. I'm very new to compiler/parser/lex and did spend some time understanding this article. I think Gopls option is very limited and blocked by Google's progress. I'm also torn between option 2 and 3. Option 2 does give us a good head start. But option 3 sounds mode fun! Practically speaking I prefer option 2, as we have limited time and resources, and given how popular VS Code extension is, I don't think the OSS tools will die anytime soon.

[gundermanc]

@benhu-msft sounds good! That was my thought as well. How does the following sound:

1) M1: Repackage the VS Code tools:

• Wire the same set of tools used in VS Code up to VS APIs for Windows and Mac.
  • Reqs: Build, debugger, colorizer, doc tooltips, Go to Definition, Find all References, and Symbol Search.
• Use the lexer based colorizer I put together.
• Demo and release on gallery.

2) M2: grow into full LS post hackathon.

• Anyone that's interested in continuing to contribute can help with this.
• I'd like to build up the lexer into parser + type checker that can power a full language service and move to using this for quick info, completion, and other similar features. Doc comments, debugger, and testing may continue to be supported by the OSS Go tools.

[gundermanc]

Btw: is anyone interested in taking point on VS Mac? I can support this effort but I don't own a Mac machine.

Alternatively, the debugger and testing integrations are both large pieces that will need dedicated effort as well.

[gundermanc]

Over the past 2 weekends, I've pushed an initial round of prototypes that 1) provide documentation tooltips, error squiggles for syntactic errors, basic colorization, structure guides, and outlining.

Now that the easy, tool-driven stuff is hackily working, I want to focus on defining and lighting up the critical parts of the experience and stabilizing the hacks into actual solutions.

Prioritized roadmap is:

1) Can build in the IDE and see errors in the error list 2) Basic parser with some completion support 3) Complete parser with full completion support 4) Debugger support 5) Code/symbol search support.

[benhxy]

Thanks Christian. I took some time to read about VSIX and its concepts. Now I have a general picture of what we might need to do. Do you mind creating a documentation on the design/ architecture of this project? It will be helpful for extension beginners like me. Cheers.

[gundermanc]

Thanks Christian. I took some time to read about VSIX and its concepts. Now I have a general picture of what we might need to do. Do you mind creating a documentation on the design/ architecture of this project? It will be helpful for extension beginners like me. Cheers.

Absolutely! I'll start here and we can migrate it into the repo as we get stuff done.

Planned Repo architecture:

• Go.CodeAnalysis - non-VS specific from-scratch language service components. For now this just has the beginnings of a lexer, but this is where the parser, type checker, and various non-VS specific abstractions for language service features would live. Ideally this same binary can be rehosted in other editors if desired. Note that despite the name, I am not currently planning on taking any dependencies on Roslyn.
• Go.Editor - Cross-plat feature DLL for VS Windows and VS Mac Editor. This DLL implements the integration with actual IDE features that are supported on both Mac and Windows. For now, it is mostly lighting up editor features by calling into open source Go tools from the community, but over time, I want to migrate it to using bits primarily from Go.CodeAnalysis. Most features should be implemented here.
• Go.Windows - VS Windows specific DLL. This is the only DLL allowed to contain VS-Windows and Windows specific references such as WPF, VS Shell, COM interops, etc. Most features should be implemented in Go.Editor, but any Windows specific functionality required will have to be implemented in this layer and injected into Go.Editor via MEF.
• Go.Mac - VS Mac specific DLL. This is the only DLL allowed to contain VS-Mac and Mac specific references such as Xamarin.Mac, MonoDevelope Shell, etc. Most features should be implemented in Go.Editor, but any Mac specific functionality required will have to be implemented in this layer and injected into Go.Editor via MEF. Ideally this should be exactly symmetrical to what is in Go.Windows to maintain feature parity.

Detailed roadmap:

1) Can build in the IDE and see errors in the error list: For this, we need to define a MEF part in the Go.Editor layer that will be responsible for managing builds and running go build on the workspace. We then wire it up to build on VS Windows and VS Mac in Go.Windows and Go.Mac respectively. 2) Basic parser with some completion support: Current colorizer is provided via an initial draft of a lexer. Next steps are to build a parser for this lexer which is capable of producing an AST for a file. With this, we'll be able to walk the AST and generate a store of symbols appearing in the document. 3) Complete parser with full completion support: once the basic completion is implemented, we should be able to extend this to work across the entire workspace and between files. This same capability could be used to power symbolic search. 4) Debugger support: Will be provided by the community dlv tool. This will need to be integrated with the VS debugger UX on the respective platforms. Note that I have the least experience with this, so this will require the most help. 5) Code/symbol search support: Once we have everything in place for completion, it should be relatively simple to use the symbol discovery mechanism to search symbols via an INavigateToItemProvider