x/tools/gopls: support for an IDE documentation viewer

firelizzard18 commented 1 year ago

I propose adding a documentation request/response to gopls. As a user, I want to easily view documentation for the code I'm working on, and ideally for private modules I am using. As a developer, I am interested in contributing a documentation viewer to the VSCode extension.

I've considered a number of possibilities for adding a documentation viewer to the extension. Fundamentally, any such solution must involve either parsing Go or go doc comments in JavaScript/TypeScript, or calling a separate binary. Not wanting to build a parser, I explored using godoc or gddo. Using a server model is annoying because the extension would need code to manage the lifetime of that server. On the other hand, calling a binary to parse a given file is less code but more overhead and either of the aforementioned tools are particularly suited to that - I spent a couple days exploring that.

I think an elegant solution would be to add a method to gopls that requests documentation for a given symbol or package:

gopls already has tooling for reading and parsing source
go/doc can be used to parse doc comments
When gopls is built with Go 1.19+, go/doc/comment can be used to provide richer output

I am interested in implementing this. Hopefully parsing doc comments for workspace files will be relatively simple. For other packages, I can leverage the fact that gopls already knows how to determine which version of a package is in use and knows how to locate that file in the module cache.

firelizzard18 commented 1 year ago

@gopherbot, please remove label Documentation

This relates to documentation, but it is a task to write code, not to write documentation.

firelizzard18 commented 1 year ago

@gopherbot, please add label FeatureRequest

hyangah commented 1 year ago

Thanks for filing the feature request.

We discussed this feature during our triage meeting.

I personally dreamed of this feature. But for long-term maintainability, we will need careful design, discussion, and code review. Unfortunately, the gopls team does not have bandwidth for the needed support right now.

It looks like you were thinking to implement this as a custom gopls command. Gopls custom command is not too different from running a separate command except that the command can have access to the parsed go files already. However, that information should be also available with golang.org/x/tools/go/packages and go list too.

How about making this feature as a separate command & extension first? Prototype & demo may help people see the value of this feature more clearly.

Another aspect to consider: if it is implemented using gopls custom commands, it's likely that only certain editors (e.g vscode) that can implement UI frontend will benefit from this feature. If it's implemented as a web server (maybe running on gopls?), other editor users may utilize the feature through web browsers.

firelizzard18 commented 1 year ago

Gopls custom command is not too different from running a separate command except that the command can have access to the parsed go files already.

In my view the main differences are that using gopls requires less additional code and gets the advantage of a long-running process that can cache work without the complexity of managing an additional process.

How about making this feature as a separate command & extension first? Prototype & demo may help people see the value of this feature more clearly.

👍

Another aspect to consider: if it is implemented using gopls custom commands, it's likely that only certain editors (e.g vscode) that can implement UI frontend will benefit from this feature. If it's implemented as a web server (maybe running on gopls?), other editor users may utilize the feature through web browsers.

Taking HTML output from a tool and embedding it in a webview puts significant restrictions on integration with the IDE. I experimented with taking HTML output from gddo (or was it godoc?) and putting it in a webview. IMO that experience is not good. I feel the same about the pprof webviewer, but I am not going to write a custom profile viewer so it's good enough.

For documentation, I want the command to provide structured data about the package and I want it to be the extension's responsibility to convert that to a UI.

Southclaws commented 1 year ago

I've recently run into this as I wanted to provide a ton of documentation and examples with properly formatted text on pkg.go.dev but the existing godoc tool doesn't support all this formatting so it comes down to either trial and error pushing and waiting for go.dev to pick up the changes or try to run pkgsite locally (which isn't well documented and requires a ton of external dependencies like a database and message queue from what I can tell)

So I'd like to help on this, I'd love to see something like this land in the official vscode extension or some feature in gopls. Perhaps we could collaborate on a proof of concept?

aarzilli commented 1 year ago

the existing godoc tool doesn't support all this formatting

Support for the new formatting (links, lists and headings) was added to godoc in 79f3242e4b2ee6f1bd987fdd0538e16451f7523e

Southclaws commented 1 year ago

Oh I didn't know about that! I'm on quite an old version, thanks!

adonovan commented 1 year ago

FWIW, the way I would like to address this feature is by adding a gopls command that would cause it to start a pkgsite instance on a local loopback port, and send a showDocument(http://localhost:12354/pkg/foo) request to the client (e.g. VSCode) which would then open a browser tab. Pkgsite would render the documentation for the workspace packages, including local edits (and eventually perhaps including unsaved local edits). The pkgsite links to the source code would link to endpoints within gopls that cause it to send showDocument(file:...) requests back to the client editor, so the net effect would be that you can seamlessly navigate between editor and pkgsite and back again just by clicking.

firelizzard18 commented 1 year ago

@adonovan Is this something you're planning to work on or suggesting the core team do this, or are you speaking in a hypothetical sense? I was thinking of working on an extension as @hyangah suggested when I have time, but if the core team will be working on it I don't want to duplicate your efforts.

Running a pkgsite instance and opening a browser tab (within VSCode I assume?) would certainly be the simplest solution and I believe it could be done without any custom LSP methods. However I personally would like something more seamless as I discussed above. I'd like to have a side bar view for browsing packages/types/etc and I'd prefer the actual documentation display/tab to feel like it's a native part of VSCode as opposed to an embedded website. Thus my suggestion that gopls return a structured response that the extension can do what it wants with.

adonovan commented 1 year ago

I plan on working on the feature I described, since it allows users to see the documentation exactly as it will appear in the production pkgsite service. I think it should be straightforward to implement.

Running a pkgsite instance and opening a browser tab (within VSCode I assume?)

When an editor (e.g. VSCode, Emacs) gets a showDocument request, it typically opens file: URLs as editor buffers, and http: URLs in the system browser, via open(1) or xdg(1). I think there's a way to request an embedded browser tab within the editor, but in my limited testing VSCode didn't seem to honor it.

I personally would like something more seamless as I discussed above. I'd like to have a side bar view for browsing packages/types/etc and I'd prefer the actual documentation display/tab to feel like it's a native part of VSCode as opposed to an embedded website. Thus my suggestion that gopls return a structured response that the extension can do what it wants with.

That sounds like it would require a Go- and VSCode-specific extension, which limits its usefulness to users of a single client editor. It also sounds like it might have significant functionality overlap with the existing Hover and Outline functionality. It may be a reasonable thing to do, but we should take care to avoid feature redundancy or significant extra complexity. It might be worth a quick prototype before you spend too long on it.

@hyangah may be in a better position to advise w.r.t. VSCode-specific features.

firelizzard18 commented 1 year ago

Ultimately I have two more or less independent desires:

Preview what my documentation will look like on pkg.go.dev
View documentation within my development environment

(1) is clearly served best by your solution, since it would literally be doing exactly that (rendering it exactly as pkg.go.dev will). My goal for (2) is to present similar or the same information as what I see on pkg.go.dev but in a way that is more seamless and interrupts my development flow less than switching to a browser. Hover and outline functionality are very useful, but often I want a more holistic view where I can browse through all the declarations made in a package. Ultimately pkg.go.dev provides all of that, but I'd prefer to have it right within my IDE. If and when I start working on this, I think I'll do it as an extension that calls out to an executable each time. That will be a lot less efficient than a long-running process but it will be a lot easier to implement.

adonovan commented 3 weeks ago

(1) is clearly served best by your solution, since it would literally be doing exactly that (rendering it exactly as pkg.go.dev will).

The doc viewer is now implemented in the goplsv/0.16 pre-release; please try it out.

My goal for (2) is to present similar or the same information as what I see on pkg.go.dev but in a way that is more seamless and interrupts my development flow less than switching to a browser.

I wonder whether there is a way for the VS Code Go extension to inject a little bit of policy into the showDocument handler to make certain URLs open an internal browser frame. Worth investigating; it might be a small change overall.

firelizzard18 commented 2 weeks ago

How do I test this? "Locate configured Go tools" shows that I'm running gopls v0.16.0-pre.1 but I don't see a code lens and Ctrl+. just tells me "no code actions".

adonovan commented 2 weeks ago

It's a Code Action, and it should be accessible anywhere in the file. For example, this command:

$ gopls fix -a ./gopls/main.go:#1 source.doc

will open a browser window with the correct URL (though gopls terminates immediately so it won't serve the page--but it illustrates the mechanics). In VS Code, use the "Source action..." menu. In Emacs+eglot, use M-x eglot-code-actions.

firelizzard18 commented 2 weeks ago

The doc viewer is now implemented in the goplsv/0.16 pre-release; please try it out.

:+1: LGTM. This is perfect for the purpose of working on documentation for my packages. Changes I make to doc comments are immediately reflected in the rendered page (once I refresh) which is nice.

As far as I can recall I've never used source actions before, which is probably why I couldn't find it at first (or at least that's what I'm telling myself). It may be worth documenting that with screenshots. I also had no clue the "Browse assembly for function" feature existed.

I wonder whether there is a way for the VS Code Go extension to inject a little bit of policy into the showDocument handler to make certain URLs open an internal browser frame. Worth investigating; it might be a small change overall.

I'm threw together a patch that uses the language client's middleware config to intercept the showDocument request and opens a webview within vscode instead. The UX isn't wonderful but it works.

With respect to my other desire - integrating the documentation viewer into vscode - @adonovan @hyangah what would you think of adding a hook that allows a 3rd party extension to override the webview renderer? It would be a bit tricky to design well, but I could add a method to the extension API in extension/src/extensionAPI.ts that registered a handler, then check for and call that handler in openGoplsInWebview. That would allow me to experiment with a custom documentation viewer in a separate extension. The HTML coming from gopls is very clean. It shouldn't be too hard to apply styling to make it fit with the user's theme, and I could remove the navigation dropdown and create a side panel to replace it.

showDocument middlware

```typescript // Use this as middleware.window in the call to new GoLanguageClient in extension/src/language/goLanguageServer.ts const windowMiddleware = { async showDocument(params, next) { // TODO: The typing for next is a lie - it doesn't actually expect us to // pass a cancellation token. Also, it doesn't pass the token to us so // there's not much we can do. The latest (unreleased) version of // vscode-languageclient *does* pass us the token, so once that's // released and we've updated, we can remove this hack. const showDocument = next as (params: ShowDocumentParams) => Promise; try { if (/^http:\/\/127\.0\.0\.1:\d+\/gopls/.test(params.uri)) { await openGoplsInWebview(params); return { success: true }; } } catch (_) { // If this fails, try opening the normal way } return showDocument(params); } } async function openGoplsInWebview(params: ShowDocumentParams) { const externalUri = await vscode.env.asExternalUri(vscode.Uri.parse(params.uri)); const panel = vscode.window.createWebviewPanel('gopls', 'gopls', vscode.ViewColumn.Active); panel.webview.options = { enableScripts: true }; panel.webview.html = ` `; } ```

adonovan commented 2 weeks ago

The doc viewer is now implemented in the goplsv/0.16 pre-release; please try it out.

👍 LGTM. This is perfect for the purpose of working on documentation for my packages. Changes I make to doc comments are immediately reflected in the rendered page (once I refresh) which is nice.

Thanks, glad to hear it.

As far as I can recall I've never used source actions before, which is probably why I couldn't find it at first (or at least that's what I'm telling myself). It may be worth documenting that with screenshots. I also had no clue the "Browse assembly for function" feature existed.

Screenshots like this? ;-)

(It's ok, no-one reads release notes. We plan to add a "Browse gopls documentation" command to the VS Code main menu.)

The assembly feature is equally new.

I threw together a patch that uses the language client's middleware config to intercept the showDocument request and opens a webview within vscode instead. The UX isn't wonderful but it works.

Oh, very nice. I'll have a play with it; this might be something we should add to the VS Code Go extension.

With respect to my other desire - integrating the documentation viewer into vscode - @adonovan @hyangah what would you think of adding a hook that allows a 3rd party extension to override the webview renderer? It would be a bit tricky to design well, but I could add a method to the extension API in extension/src/extensionAPI.ts that registered a handler, then check for and call that handler in openGoplsInWebview. That would allow me to experiment with a custom documentation viewer in a separate extension. The HTML coming from gopls is very clean. It shouldn't be too hard to apply styling to make it fit with the user's theme, and I could remove the navigation dropdown and create a side panel to replace it.

This is a question for @hyangah more than me. What kind of styling would you like to apply to the HTML? I would be happy to accept contributions of unequivocal style improvements. Customization and optional features we should discuss, but perhaps those too.

firelizzard18 commented 2 weeks ago

Screenshots like this? ;-)

(It's ok, no-one reads release notes. We plan to add a "Browse gopls documentation" command to the VS Code main menu.)

Yeah, I wish I had found that :laughing: :facepalm:

I think I need to start reading gopls release notes. I see the extension release notes (but don't necessarily read them) because vscode opens them whenever the extension updates, but it didn't occur to me to check the gopls release notes.

What kind of styling would you like to apply to the HTML?

I want to apply the colors of the active VSCode theme to it, so not something that would be generally applicable. Specifically, VSCode injects a few hundred CSS variables and I want to inject CSS rules like these:

body {
    background-color: var(--vscode-editor-background);
    color: var(--vscode-editor-foreground);
}

pre {
    background-color: var(--vscode-textCodeBlock-background);
    border: 1px solid var(--vscode-widget-border);
}

This makes the webview feel less intrusive:

firelizzard18 commented 2 weeks ago

I added a script to openGoplsInWebview to inject stuff into the frame but I'm getting an error:

Uncaught DOMException: Failed to read a named property 'document' from 'Window': Blocked a frame with origin "vscode-webview://0naufkftkddsndfb3hpmhffqere574epss1bbs9h6ojfru0s5a4p" from accessing a cross-origin frame

I modified gopls to set response headers but that didn't help and it's looking like there's simply not a way to allow cross-origin frame access short of changing electron's security settings. So I'll either need to find a way to change the origin of the frame, or make the HTTP request in the extension and dump the response into the webview (aka stop using an iframe).

firelizzard18 commented 2 weeks ago

I gave up on injecting anything into the <iframe>. Unless there's some way to disable same-origin restrictions, I don't think that's a feasible path. Instead I wrote a module to fetch the document from gopls, parse it, fiddle with it, and plop it into the webview. It's kind of a hack but it works and the end result is promising. It needs some polish - navigation and reload buttons, jumping to the right location when loading with a #fragment - but it's a promising start.

browser.ts

```typescript import vscode from 'vscode'; import axios from 'axios'; import { HTMLElement, parse } from 'node-html-parser'; // eslint-disable-next-line @typescript-eslint/no-explicit-any type Tail = T extends [any, ...infer Tail] ? Tail : never; export class GoplsBrowser { readonly panel: vscode.WebviewPanel; readonly #history: string[] = []; constructor(...options: Tail>) { this.panel = vscode.window.createWebviewPanel('gopls', ...options); this.panel.webview.onDidReceiveMessage(async (e) => { if (typeof e !== 'object' || !e) return; try { switch (e.command) { case 'back': if (this.#history.length < 2) { return; } this.#history.pop(); await this.navigateTo(this.#history.pop()!); break; case 'navigate': await this.navigateTo(e.url); break; } } catch (error) { vscode.window.showWarningMessage(`Navigation failed: ${error}`); } }); } async navigateTo(url: string) { const page = vscode.Uri.parse(url).with({ fragment: '' }); const pageStr = page.toString(true); const base = (await vscode.env.asExternalUri(page.with({ path: '', query: '' }))) .toString(true) .replace(/\/$/, ''); // Fetch data const { data } = await axios.get(url); // If the response is empty, assume it was opening a source file and // ignore it if (!data) return; // Track history this.#history.push(url); // Process the response const document = parse(data); const head = document.querySelector('head')!; // Note, gopls's response does not include , all content is a // direct child of // Add the base URL to head children and the logo const baseStr = base; const addBase = (s: string) => (s.startsWith('/') ? `${baseStr}${s}` : s); fixLinks(head, addBase); fixLinks(document.getElementById('pkgsite'), addBase); document.querySelectorAll('a').forEach((a) => { const { href } = a.attributes; if (!a.hasAttribute('href')) { return; } // If the link is to an anchor on this page, trim it to just the #anchor if (href.startsWith(`${pageStr}#`)) { a.setAttribute('href', href.replace(pageStr, '')); return; } // If the link is to a different page from gopls, hijack it if (href.startsWith(baseStr)) { a.removeAttribute('href'); a.setAttribute('onclick', `navigate('${href}')`); a.classList.add('clicky'); } }); // Add to fix queries head.appendChild(parse(``)); // Add `) ); // Add `) ); // Update the webview this.panel.webview.html = document.toString(); } } function fixLinks(elem: HTMLElement | null, fix: (url: string) => string) { if (!elem) return; if (elem.attrs.href) { elem.setAttribute('href', fix(elem.attrs.href)); } if (elem.attrs.src) { elem.setAttribute('src', fix(elem.attrs.src)); } for (const node of elem.childNodes) { if (node instanceof HTMLElement) { fixLinks(node, fix); } } } ```

firelizzard18 commented 1 week ago

I've implemented 80-90% of what I was looking for. There's still room for polish (e.g. syncing the tree to the scroll position) but I'm happy with what I have. There's a glitch at the end where it opens the link in the browser. I fixed that but didn't re-record.

https://github.com/golang/go/assets/879055/b3bc0535-219f-41cd-b55a-f1114d9d08d9

golang / go

x/tools/gopls: support for an IDE documentation viewer #55861