search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
122.94k stars 17.53k forks source link

go.dev: support packages with unknown/unsupported licenses #36840

Closed eliasnaur closed 4 years ago

eliasnaur commented 4 years ago

There has been work on this, but @mvdan suggested I create an issue for it for tracking.

The motivating example is the dual-licensed (MIT/UNLICENSE) gioui.org packages. See https://pkg.go.dev/gioui.org/op. The problem is that go.dev refuses to display certain meta-data if the license is not detected or in a list of supported licenses. In particular, the blocking of "Doc" and "Overview" sections make the pkg.go.dev listing almost useless.

Compare godoc.org which, while inferior to pkg.go.dev in other ways, has no problem displaying documentation regardless of licenses.

The Go team seems to have a manual process of vetting packages such as gioui.org. However, that process is slow for all parties and might not cover every weird license out there.

@rogpeppe because pkg.go.dev also blocks some of his packages.

julieqiu commented 4 years ago

We discussed this on the golang-tools call - but posting here for posterity.

At the moment, we can only post metadata about a package/module (imports, importers, published date, etc.) when it does not have a recognized license per the requirements at https://pkg.go.dev/license-policy.

If any package authors are having issues, feel free to email go-discovery-feedback@google.com for manual review consideration by our legal team.

mvdan commented 4 years ago

I think Julie misclicked - from her comment, I don't understand why we should close this issue. It seems useful to leave it open, so that other module authors having similar issues can subscribe for updates.

julieqiu commented 4 years ago

I closed this issue because we don't intend to support unknown/unsupported licenses in the foreseeable future.

The most up-to-date information on our license policy will be at https://pkg.go.dev/license-policy. Authors can feel free to file an issue, or email go-discovery-feedback@google.com for any questions about their specific package.

Any major updates will be posted on the tracking issue for pkg.go.dev, so folks should subscribe to that issue for project updates.

eliasnaur commented 4 years ago

Would you consider alternatives? It seems to me the issue that go.dev needs permission to a very limited use of the copyrighted work, namely displaying the textual documentation embedded in the repository. For example, how about a file that clearly states go.dev (or source code explorers in general) permission to display the package contents? Say, <reporoot>/.allow-sourceexplorers.

eliasnaur commented 4 years ago

FWIW, other source explorers such as https://docs.rs/ doesn't seem to be affected by licensing issues either.

julieqiu commented 4 years ago

We (and the proxy team) considered that option and decided that it wasn't viable. We would rather not solve this problem with a "just for us" solution, because it doesn't encourage package authors to fix their licenses for the broader open source ecosystem if there are issues, and other users have concerns about a package or module's licenses.

/cc @jba @findleyr @katiehockman @heschik @hana

eliasnaur commented 4 years ago

a "just for us" solution

I deliberately named the file without reference to go.dev. I assumed that if Google has this legal issue with displaying source code meta-data, other similar explorers have it as well. Otherwise, what's special about Google?

package authors to fix their licenses for the broader open source ecosystem if there are issues, and other users have concerns about a package or module's licenses.

I'm sorry, but I can't see what I can do to "fix my license" for the gioui.org modules. If anything, go.dev is being overly restrictive in not accepting dual licenses as OR, rather than AND (gioui.org is dual licensed MIT OR UNLICENSE and MIT is supported by go.dev).

Secondly, I don't think it proper for a source code explorer to dictate the "proper" licenses for Go package.

Of course, please disregard my comments if you consider free, manual legal review by Google an acceptable solution for every package that needs it. If so, I still think there should be an automatic, faster (and cheaper for Google?) way to smooth the process; hence my .allow-sourceexplorers strawman.

julieqiu commented 4 years ago

gioui.org will be fixed on our next deploy, so there isn't any further action needed on your part.

I will have to defer to our legal team on their decision for how to the AND vs OR decision for dual licenses because IANAL.

Our current volume of special case issues is low enough that we plan to keep the manual review process for now (rather than building an automatic solution). We can always revisit this strategy as other specific issues come up.

mewmew commented 4 years ago

gioui.org will be fixed on our next deploy, so there isn't any further action needed on your part.

Good that gioui.org is now supported by pkg.go.dev. However, what about other packages that are dual-licensed, that are put in the public domain (through UNLICENSED) or otherwise blocked by pkg.go.dev?

There are currently 2,485 Go repositories released into the public domain (through UNLICENSED). For comparison, the total number of open source Go repositories hosted on GitHub is 232,299, so ~1% are released into the public domain.

To give another comparison, there are more Go projects released into the public domain (through UNLICENSED) than there are Go projects licensed under AGPL3, of which there are currently 2,244. And AGPL3 is recognized by pkg.go.dev.

There are currently 39 Go repositories licensed as Artistic License 2.0, another license recognized by pkg.go.dev.

For a full set of statistics, see below:

Total number of open source Go repositories on GitHub:

Total number of open source Go repositories with license recognized GitHub search:

Total number of open source Go repositories with license approved by pkg.go.dev:

Total number of open source Go repositories with license not approved by pkg.go.dev:

Personally, I think that this helped me gain a better understanding of the license landscape of the Go ecosystem. Prior to this quick evaluation, I felt that the list of licenses was too restrictive. However, given that it is possible to dual-license projects (although it will require manual effort to get them on pkg.go.dev), I think that the list of licenses is good enough as is.

I would very much like to see a public domain option added to the list, as it is the license used by most Go projects which is not yet supported by pkg.go.dev.

With kind regards, Robin Eklind

--

Edit: I do wonder though, what those 94,422 (32%) of the projects without a license recognized by GitHub search are.

jba commented 4 years ago

Thanks for that analysis. Very illuminating.

We appreciate what motivates people to reach for a public-domain style license, and in our own analysis, we've also identified that as a key issue. We'll try to have an update on that topic soon.

networkimprov commented 4 years ago

@mewmew do those numbers count github "forks"?

mewmew commented 4 years ago

@mewmew do those numbers count github "forks"?

Good question. The number is directly taken from the linked GitHub searches. I did not think of further refining the search to account for forks (i.e. duplicates). Feel free to post a follow-up analysis :)

Edit: @networkimprov, the numbers do not include forks. If we were to include forks, then e.g. for MIT, the number would be ~400,000: https://github.com/search?q=language%3AGo+license%3Amit+fork%3Atrue&type=Repositories

mewmew commented 4 years ago

Edit: I do wonder though, what those 94,422 (32%) of the projects without a license recognized by GitHub search are.

I can give at least an approximate answer to this. At least part of the number includes dual-licensed projects, as they seem to be excluded from the GitHub search results. For instance, https://github.com/llir/llvm does not show up when searching for 0-clause BSD, but the project is dual-licensed into the public domain (through UNLICENSED) and under a 0-clause BSD license. The specific issue tracking the llir/llvm project is #36983.

logrusorgru commented 4 years ago

because it doesn't encourage package authors to fix their licenses for the broader open source ecosystem if there are issues

It's called "force", not "encourage". Please call things with their real names.

So, why I should be forced to use an "approved license"? I did my best to choose my license, having considered all the possible problems:

And I can conclude, that the limitation is just the imposition of Google/Go’s will.

networkimprov commented 4 years ago

@logrusorgru please file a new issue to request inclusion of your packages on go.dev.

arp242 commented 4 years ago

We (and the proxy team) considered that option and decided that it wasn't viable. We would rather not solve this problem with a "just for us" solution, because it doesn't encourage package authors to fix their licenses for the broader open source ecosystem if there are issues, and other users have concerns about a package or module's licenses.

I think the biggest problem here is that the LICENCE file is intended for humans to read and review, and not for programs to get data from.

Especially larger projects have more complex licensing than "just MIT" or "just GPL": they may be dual-licensed, have difference licenses for different files, may include 3rd-party code, and so forth. It's almost impossible to communicate this clearly to humans in a LICENSE file and remain usable by pkg.go.dev.

I think "package authors should fix their license" is just one reason licenses aren't being detected, and to be honest I think the above is too dismissive of the complexities of real-world licensing @julieqiu.

I don't know what the best solution for this is; I know there's https://reuse.software/spec, but it's fairly new without much adaptation (AFAIK), but it would solve the problem without a "just for us" solution and I think it's worth taking a look at. I don't know if there are other similar standards.

At any rate, I think it's worth trying to find a better solution to this, which may take a while, rather than just closing the issue. Just because no one can think of a good solution today doesn't mean a good solution can't be thought of tomorrow.

logrusorgru commented 4 years ago

@logrusorgru please file a new issue to request inclusion of your packages on go.dev.

I have already got the reject by email some time ago.

Me.: Hello. Please, add WTFPL license to the https://pkg.go.dev/license-policy list. For example https://github.com/logrusorgru/aurora you can see GitHub detects the...

Jonathan Amsterdam jba@google.com: Thanks for your request, Konstantin. Unfortunately, we're not able to use WTFPL.

Regardless the github.com/google/licensecheck detects it ok.

networkimprov commented 4 years ago

Things may have changed since then. And you can't necessarily trust email :-)

jba commented 4 years ago

@logrusorgru, licensecheck's job is just to recognize licenses. pkg.go.dev has stronger requirements: the license must grant us permission to display the README and documentation, and must have the right legal properties. I'm afraid nothing has changed with regard to our treatment of WTFPL.

jba commented 4 years ago

https://reuse.software/spec ... would solve the problem

@arp242, That spec mandates that a license file's name is the SPDX identifier for the license it contains. Presumably you're suggesting that we use the SPDX identifier and ignore the contents.

The problem is: what if they don't match? What if the license forbids redistribution but its filename says it's OK? It's unclear what the legal ramifications are, but even we non-lawyers can understand that a human judge might give more weight to a complete natural-language document than to a short name.

And frankly, license recognition is not the biggest problem. We have a couple of ways to recognize non-standard license text.

logrusorgru commented 4 years ago

@logrusorgru, licensecheck's job is just to recognize licenses. pkg.go.dev has stronger requirements: the license must grant us permission to display the README and documentation, and must have the right legal properties. I'm afraid nothing has changed with regard to our treatment of WTFPL.

So, I can find arguments against this position. But, seems, other corporations and users can stuck with the same fears. It's better to move my projects to Unlicensed, I think. It's the shortest and the simplest one of approved.

Thanks you @jba, @networkimprov.

ddevault commented 3 years ago

I am maintaining an independent fork of godoc.org at https://godocs.io, and will do so indefinitely. Feel free to use it if your project is being bullied into changing its license or else having docs taken away from it.

ddevault commented 3 years ago

Sharing information about independent forks which address features that the Go team does not want to address, in issue threads about those same features, is not spam. Please provide more detail if you believe otherwise.

I get that you're proud of pkg.go.dev and uncomfortable with those who don't like it, but that doesn't change the facts.

andybons commented 3 years ago

Both comments are off topic from the discussion. You have other, more appropriate forums to advertise your service. Please use them. Thanks.

ddevault commented 3 years ago

Re-iterating is not the same thing as providing rationale. This is hardly an advertisement. The moderation tactics here smell of salty maintainers rather than dispassionate rules enforcement.