proposal: x/pkgsite: package tours (additional documentation type for packages)

[matttproud]

Proposal Details

Background

Go packages can have two distinct forms of documentation:

1. Comments anchored to identifiers as described by Godoc: documenting Go code and Go Doc Comments.

2. Runnable (API) examples as described by Testable Examples in Go

Both of these are rendered in godoc and pkgsite.

These two forms of documentation serve somewhat overlapping/disjoint purposes:

• Go documentation comments are mostly in the vein of reference manual.
• Runnable (API) examples are minimal viable demonstrations of an API at the very least to sometimes complete end-to-end examples (which it is is predicated by how complicate the API is to work with).

Problem

In this public discussion, it became clear that a type of documentation is missing for users: step-by-step tutorials or material that focuses on an end-to-end developer journey. This led to a thought about how we could fix this gap in a nice Go-like way.

Proposal

What if we expanded the documentation servers to enable users to create their own API tours à la A Tour of Go? A package could have multiple tours to demonstrate the journeys.

Imagine a hypothetical Go module at github.com/matttproud/rot13 that follows a directory structure like this:

$ tree rot13
rot13
├── cmd
│   └── rot13
│       ├── rot13.go
│       ├── rot13_test.go
│       └── rot13_x_test.go
├── endtoend
│   ├── endtoend_test.go
│   └── testdata
│       ├── input.txt
│       └── output.txt
├── go.mod
├── rot13.go
├── rot13_test.go
└── rot13_x_test.go

We could have a directory structure like this:

rot13
│   … earlier elements elided …
├── tours
│   ├── basic.article
│   └── advanced.article
│   … later elements elided …

Then the respective documentation servers would have some sort of internal support to run tours (e.g., use what binary tour uses). When viewing the respective package in the viewer, the table of contents for the package would also include a heading that lists sub-elements of available tours (tracer shot):

This has the advantage of keeping the documentation for extended workflows adjacent to the code so that it can be always fresh.

Perhaps this could even be extended to support the present tool, too.

Rejected Alternatives

There is technically a third form available: the blog post (e.g., Go Concurrency Patterns: Context), but …

1. these blog posts are seldom back-linked to the Go package documentation.
2. findability is not great either with the blog posts (especially older ones).
3. sometimes the blog posts are superseded with new information/methodologies.

[fzipp]

Packages can already have a long top-level doc comment (rendered as "Overview") with detailed usage introduction and examples, see the testing package as an example.

[Merovius]

@fzipp That's true and arguably omitted from the top-comment. But I think a guided tour is still a qualitatively different and more helpful form of documentation, especially when considering that there might be multiple scenarios.

[fzipp]

My assumption is that if package maintainers aren't fully utilizing the potential of the available space already, they won't use additional space either. The real issue is that many developers prefer programming over writing documentation.

[matttproud]

There are several cross-cutting considerations here:

1. The community has not done a particularly good job socializing what good documentation is. A poignant example to me is that for over a decade, there was a set of undocumented syntax rules before the article Go Doc Comments was published. I don't fault the Go Team for that.
2. Documentation for user journeys is somewhat similar to the concept of a codewalk here. The thing is, due to the implicit need for pagination or sectioning of the documentation (e.g., Step 1. Open drawer, Step 2. Pull out carrot peeler, etc) with the amount of structuring required, potential length, and the number of different journeys a package could have, the top-level documentation block for a package could get very unwieldy. Some of existing standard library packages do have long ones, but I can think of a couple of foundational business logic libraries in my workplace that are not too complicated but that would easily add multiple hundreds of lines to a hypothetical package's doc.go file. The package's central documentation is fine for some things, but not others. Nevertheless, I fully agree that it is underutilized.
3. It seems like there isn't consensus among the Go Team on what is appropriate package documentation versus external documentation (e.g., 1 and 2: these are all points/misunderstandings that have caused users to trip up that I wanted to clarify). This is to say that if I wanted to document more user journeys of standard library APIs, I suspect these contributions would not be easily accepted in ordinary Go documentation comments or even as examples.

We need a multi-pronged solution to documentation. Developers who want do the right thing (there are many of us who do) simply do not have the means.

[earthboundkid]

I use the example docs extensively in my open source projects. If I could build a programmatically tested tour with the Go tool, I would probably do that instead of using a README because the README can get out of sync pretty easily.