go.dev: consider an 'api-documentation-license' file, or similar

[thepudds]

https://pkg.go.dev/license-policy currently lists approximately 13 recognized open source licenses that pkg.go.dev currently detects and recognizes. If a repository has a license outside of those 13 licenses, the repository currently does not have its documentation displayed on pkg.go.dev, and might otherwise be at a disadvantage in terms of search and so on.

In the now-closed #36840, @eliasnaur offered a couple of different suggestions, including possibly a sentinel file such as something like <reporoot>/.allow-sourceexplorers as a way to indicate that a repository owner is granting permission to pkg.go.dev to display the API documentation.

IANAL, but I could imagine a legal team not being satisfied with that as a clear enough indication of intent by the repository owner.

Perhaps instead a variation could be a file called something like api-documentation-license or similar, with the contents of the file being the actual open source license that the repository owner is stating covers the API documentation.

This could enable someone to publish their code under a less common license that is not currently recognized by pkg.go.dev (such as the Unlicense), but pick a more common license that is recognized pkg.go.dev (such as MIT or BSD) to cover their API documentation so that pkg.go.dev could programmatically determine that it is allowed to index and display the package documentation.

Perhaps the api-documentation-license file would just be the license itself, or perhaps it would be required to start with some type of canonical line, such as:

The following license applies to the API documentation:

<text of actual open source license for the API doc>

... or whatever verbiage might be acceptable to the legal team to make the intent and coverage of the license clear.

CC @kortschak @rogpeppe

[kortschak]

This would work for code that is maintained, but still has the problem of darkening a probably not insignificant set of packages that are no longer contributed to. Not wanting the perfect to be the enemy of the good (or even moderately reasonable), I'd vote for this. However there are deeper issues.

[komuw]

I'm still yet to hear a concrete reason as to why https://pkg.go.dev/ seems to be affected by licensing issues while other module/source explorers such as https://docs.rs/ aren't.

The best I've gathered from here and there is that the google legal team has advised as such; or something to that effect.

If this is all about the google legal team wanting to cover google Inc from supposed lawsuits that my non-legal mind can't fathom; then maybe https://pkg.go.dev/ should be ran by a third party that isn't google.

The licensing issue on https://pkg.go.dev/ coupled with the intended redirection of godoc.org requests to pkg.go.dev; seems to imply that the only/main reason why https://pkg.go.dev/ was created was so as to give goole a legal way out.

[networkimprov]

Yes, they got, erm, unprecedented legal advice ;-)

The Go team wants to own *.go.dev. There's talk of moving godoc.org to a third party in this thread: https://groups.google.com/forum/#!topic/golang-nuts/OA-7lUbZJMY

[komuw]

I'm still yet to hear a concrete reason as to why https://pkg.go.dev/ seems to be affected by licensing issues while other module/source explorers such as https://docs.rs/ aren't.

I think I finally found an answer. In [1], @rsc writes;

Why does pkg.go.dev require a detected license to show docs? Why doesn't godoc.org?

The teams working on the proxy and on pkg.go.dev have spent a lot of time talking to Google's lawyers about what we can and can't do with Go source code downloaded from the internet. The rule we've been given to follow is that serving a pretty HTML version of the docs is displaying a modified version of the original, and we can only do that if there's a recognized known-good license that gives us that permission.

When we adopted godoc.org from Gary Burd back in 2014, it did not occur to any of us to put it through that kind of review. If we had, maybe the community would have gone through this licensing pain earlier. For now we are focusing on making changes to pkg.go.dev rather than correcting past mistakes on godoc.org. (At this point, more scrutiny of what godoc.org does is not likely to have an outcome that anyone likes.)

1. https://groups.google.com/d/msg/golang-dev/mfiPCtJ1BGU/qtCrqlrEEwAJ

[rsc]

Not directly on topic, but something I need to say. If you are not a lawyer and not used to talking to lawyers and how they think, then it's easy to fall back on criticisms like "they got unprecedented legal advice" and try to caricature the Google lawyers we've consulted as the typical risk-averse, answer-is-always-no stereotype. I have had the displeasure of dealing with lawyers like that at other companies (ask me about Plan 9 some time); they really do exist. But, I am quite happy to report, not at Google. They work incredibly hard to help us do what we want to do, legally, instead of working to keep us from doing it at all. I have nothing but great things to say about every lawyer I've ever interacted with in 13 years at Google. They have all been fantastic. Please don't go around maligning them with no actual first-hand experience or data.

[rsc]

Back on topic of showing docs and alternate licenses, we continue to recognize and accept licenses that are used or even just depended upon frequently in the Go ecosystem. This issue was originally filed to try to handle the case of "Unlicense"-licensed code, which we weren't showing. Based on the Go ecosystem usage, we've added Unlicense as a recognized license text and can show docs for those packages. So there is no longer a need for a "docs-only" license to use with Unlicense.

In general, again, if a license is important to the Go ecosystem, we will do what we can to recognize and accept it. That's much better than just having a "docs-only" exception. I'm not aware of any major licenses that are missing at this point, although I am sure there are a few small ones in the tail still. We'll get there.

Closing this issue because we'd rather just recognize the direct licenses, and we've fixed the important case in the initial report. Once we've done a bit more work making sure of the low-order bits in the data, I intend to publish information about the relative usage of different licenses in the Go ecosystem. If important licenses remain that you think we are missing, feel free to file bugs about them, but rest assured that we're crunching the data whether you file a bug or not.

[networkimprov]

I'm relieved to learn that this problem is (mostly?) solved, and glad to hear that your attorneys are so supportive.

For the record, to the best of my knowledge, the opinion that "serving a pretty HTML version of the docs is displaying a modified version of the original" is not widely held, altho I haven't done enough research to say that it's literally unprecedented. As the intent is to faithfully represent the original within an index, I believe the common perspective is that it's "fair use". FWIW, I am accustomed to talking with lawyers :-)