proposal: decide policy for sub-repositories

[adg]

[this is not a proposal yet, just a problem statement]

There are several proposals to add new packages to the x/foo "sub-repositories". (See: #16353, #13432, #12332, #16257, #15985, #15406, and more.) The proposals are all different, but they circle around the same issue: what should and should not be in a sub-repository.

Currently, the sub-repositories contain x different things:

1. Packages and tools that are part of a loose set of Go tools (godoc, goimports, benchcmp, x/debug, etc)
2. Packages and tools that support the Go web services (godoc, present, x/playground, x/tour, etc)
3. Tools that support Go project development (x/build, x/benchmarks, x/review, etc).
4. Packages that augment the standard library (x/text, x/image, x/crypto, x/sys, etc).
5. Other projects worked on by Go team members at Google (x/exp, x/mobile, etc).

To me, it's clear that 1-2 belong in the sub-repositories, and should be an official part of the Go project.

For 4-5, this code only lives in the sub-repositories either for historical reasons or mere convenience. The Go contributors are accustomed to the process (tools and reviews) of writing code in that style, so the sub-repos were just the obvious place to go.

(3 is a fringe case, as the tools could be worked on elsewhere, but the people working on them work on Go full time and work on that code solely to support the project, so I am inclined to ignore this code for the purposes of this issue.)

As the project grows, it attracts people who want to contribute new packages to these sub-repositories (see the issues cited at the beginning of this issue). The reasons for inclusion include:

1. To provide an official package for doing X.
2. To provide a supported package for doing X.
3. To contribute to the Go project.
4. To use the project's development processes (gerrit, etc).
5. To have their code reviewed by other Go contributors and to benefit from their expertise.
6. To use the same consistent license (with CLA) that the project uses.
7. To provide a single package for a single purpose, and avoid duplicate effort.
8. To provide an official package for interoperating about (but not necessarily doing) X.
9. To act as a dependency for something else under x/.

[What are the other reasons? I'd like to enumerate them all here.]

---

My opinion begins here:

None of the above reasons are good arguments to add new packages to the sub-repositories. Taking each in turn:

1. The "official" packages, such as they are, are generally of high quality because a small group of dedicated contributors have invested constant energy in them. But there are other high quality packages elsewhere in the Go ecosystem (many of higher quality than some of those in the sub-repos), it's just harder for people to find them and they don't have "prestigious" import paths. Rather than moving more stuff into the Go project, I'd think we should address this issue by making it easier to find and recognize high quality packages. (One way to do this is to improve godoc.org, but there are many other ways.)
2. Putting something in the sub-repositories does not imply support. People are not more or less inclined to work on a package just because it's in a sub-repository. The current set of Go contributors are stretched pretty thin as it is. I think that by moving more stuff into the sub-repositories we set expectations of support where there is none. That's bad for our users.
3. Moving more code into the project just for the sake of contributing is not a net win. There's plenty of work to be done on the code that is already part of the project (and the many, many open issues).
4. If people want to use our processes, I think that a better approach is to find a way to use those processes outside of the project. That's a solution that scales much better.

[see discussion below for other points]

In general, I'd like to make the Go project smaller rather than bigger. (In the same sense that we would remove stuff from the standard library, if we could.)

As I said, this is just my opinion and perspective. I have created this issue to gather feedback from other [potential] contributors.

Personal opinion ends.

---

We need to set a policy for what belongs in the sub-repositories, and in doing so more clearly define the boundaries of the Go project.

[rhedile]

The current attitude to /net/rpc appears to be "give a dog a bad name the shoot it ". The favoured "production rpc mechanism" is gRPC which does a very different job. Its slower, needs a manifest, and does not like dedicated server port stacks. net/rpc, whether with gob or JSON, ist working code for us. Please don't sunset it.

On 30 September 2016 at 00:36, Andrew Gerrand notifications@github.com wrote:

@StabbyCutyou https://github.com/StabbyCutyou thanks for clarifying. It's hard to divine the context behind everyone's posts, and I'm sure I misinterpreted you. Sorry. 😦

Go has a lot of really strong libraries that come out of the box - as a developer the idea of seeing Go "shrink" in this capacity would be disappointing

When I talked about taking stuff out of the standard library I was referring to things like net/rpc which was an early experiment and is now clearly inadequate as a production rpc mechanism, having been superseded by several other rpc systems.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17244#issuecomment-250611740, or mute the thread https://github.com/notifications/unsubscribe-auth/AEAf7CaR6guzv4y0cugoj8jdzk6IEr5Cks5qvD1VgaJpZM4KHJNI .

[ianlancetaylor]

@rhedile This is not the issue on which to discuss the net/rpc package in particular. That is issue #16844.

[adg]

@nigeltao I think this

Speaking of the OP, another reason for inclusion would be to provide an official package for interoperating about (but not necessarily doing) X.

is what @rakyll was referring to. I'll add it to the list.

I think that should be in scope for /x, and that it would make sense to eventually move such packages to the standard library.

[crawshaw]

I put mobile under golang.org/x because it gave us common tooling with the Go project, in particular gerrit and the issue tracker. I think that was reasonable at the time, given that golang.org/x didn't have any clear meaning.

Looking at where @adg is going with this proposal, I suspect it will not belong there any more. (Especially as a lot of us working on it have had our time taken by other projects, it should have a separate issue tracker.) It could move to github.com/google/gomobile, though I will miss gerrit.

I would miss gerrit even more for x/exp/shiny. It would be nice if there were some other place that had the common tools of the Go project but did not come with the fancy golang.org/x prefix (or any other implications of being part of the standard library.)

[dgryski]

It would be nice if there were some other place that had the common tools of the Go project but did not come with the fancy golang.org/x prefix (or any other implications of being part of the standard library.)

That is precisely point (4) (in the problem and personal opinion) in the above proposal.

[nathany]

"If people want to use our processes, I think that a better approach is to find a way to use those processes outside of the project. That's a solution that scales much better." - @adg https://github.com/golang/go/issues/17244#issue-179371715

Regarding my own proposal (#15406), the goal was to utilize the Go builders to test across multiple architectures. Making the tools accessible outside of the x/ packages (#17312) is a better solution than moving a package into x.

"It is actually a medium-term goal of mine to support third-party repositories" - @bradfitz https://groups.google.com/d/msg/golang-dev/SVrNHQU1oEM/b13BeZHVBwAJ

Putting something in the sub-repositories does not imply support. People are not more or less inclined to work on a package just because it's in a sub-repository." - @adg https://github.com/golang/go/issues/17244#issue-179371715

Case in point, the fsnotify and winfsnotify packages in https://github.com/golang/exp haven't been updated in over 2 years. I've abandoned them in favour of the fsnotify organization.

"I think it would be better to have a scalable mechanism for identifying and maintaining good packages, and I don't see why that should be, or even can be, associated with golang.org/x." - @ianlancetaylor https://github.com/golang/go/issues/17244#issuecomment-249923601

Godoc.org already provides a start. A search for websockets ranks gorilla/websocket higher than x/net/websocket. It is imported more and has more stars on GitHub.

Yet people may gravitate towards the x/ package due to it's "official" status, even if gorilla/websocket is faster and subjectively better:

This already happened with websocket, where the package in /x/net is not the best implementation, but people use it because it's there. - @adg https://github.com/golang/go/issues/17244#issuecomment-250603608

From my perspective, being able to work under an x allows me to reach out to the go approvers and this is what I care when I am designing ecosystem-wide core libraries for a niche topic. I don't think it is possible once the experimentation is not considered a part of the Go project. - @rakyll https://github.com/golang/go/issues/17244#issuecomment-250005746

I see no technical reason why Go team members and other skilled gophers with the time and interest in a USB package couldn't provide code reviews and feedback to a package located under /rakyll or a /gousb organization.

"I think Gerrit might not be critical given Github PRs are becoming better." - @rakyll https://github.com/golang/go/issues/17244#issuecomment-250260049

Agreed on that. From my perspective, GitHub has caught up with what advantages Gerrit once provided. https://groups.google.com/forum/#!msg/golang-dev/Tk1rSAdiz10/

"Shouldn't they be able to put their high quality code in a repo somewhere else, and have it be regarded just as highly as anything in /x? Code should be judged on its own merits, not by its location." - @adg https://github.com/golang/go/issues/17244#issuecomment-250009629

Absolutely.

Instead of proposing what should or shouldn't be in the subrepos, I'd personally like to see a series of proposals that remove any advantage of using the x/subrepos.

That includes making tools like the Go builders, Gerrit (for those who want it), and licensing (like CLAHub) available to Git repositories anywhere.

It also means improving discovery. A curated list still isn't scalable, but I've read several mentions of reviews and ratings in this thread. There are other metrics, like number of imports, GitHub stars, passing tests, test coverage, and linting (lint/vet/fmt). Whether as part of GoDoc.org or a separate endeavour, I think discovering the best packages can be solved in a scalable way the embraces the whole community.

There is no reason why the Go team should be responsible for vetting all the foundational libraries for Go. The Go community is filled with intelligent people with a wide range of experience. Resources like CodeReviewComments encourage more gophers to use good practices. What else can be done in the area of education?

Ultimately, I'd like to see less of a divide between how the Go team and the larger Go community operate, with everyone having access to the best tools.

Instead of placing demands on the Go team through the x/ subrepos, why don't we give the Go team space and as much help as we can? Allow them to be involved in and with the community wherever their interests lay.

[dlsniper]

I wrote this message to Andrew on Slack but after reading this issue I'm not sure if this is a better place for it or not so here it goes:

I think I have a middle ground for getting the: "follows guidelines" label for projects while not having the Go team / contributors mandatory involved in the process.

I'm thinking of adding a flag to projects that would show them in as such on godoc.org (and maybe have a badge for that to add to the readme).

The reviewers should be part of the community and it should work on a trusted model and only reviewers can add new reviewers.

For an issue to be flagged as an issue at least a few reviewers would need to agree on it and the reviewers could get some form of recognition (karma) for having the most accurate reviews, basically gamify this but not for the number of repos reviewed or for the owners of the repositories but for the reviewers themselves.

As for the original issue, what to do with x/ I think of x/ as an intermediate step between really a really good project and adding it to the stdlib. Let's say gorilla/websocket could make it one day in it as it's known to be better than the standard implementation for websockets. Or things like Go mobile. We already have x/net/http for the http2 support there, no? I'd like to still be able to have a: we'll break compatibility if needed that is less strict than Go's one. And those packages don't have to make it in stdlib for 1.x lifecycle, maybe Go 2.x will be good enough. There are few things which Go misses and for that x/ would be good. Maybe a similar tools/ will be added to move the supporting tools.

[nathany]

@dlsniper Along those lines, a few years ago @garyburd had the idea of including a special token in project READMEs along with the GoDoc.org badge. https://github.com/golang/gddo/issues/107#issuecomment-23953268 The gist is that if a project doesn't include the badge/token for GoDoc.org, it's just a hobby project, and not intended to be a supported library. That doesn't guarantee a high quality library, but having a way to opt-out of the index makes sense.

Forks of a project also would be hidden from GoDoc.org based on the same mechanism, though I think that's being solved in other ways these days.

It would probably be a good idea to start one or more new threads over at https://github.com/golang/gddo/issues or golang-dev?

So far I've created one new issue based on this discussion thread:

• https://github.com/golang/go/issues/17312 go builders and trybots for third-party repositories

[cznic]

The gist is that if a project doesn't include the badge/token for GoDoc.org, it's just a hobby project, and not intended to be a supported library.

No.

[namsral]

In order to be brief and clear, my thoughts in a Q&A:

Q: What should go in golang.org/x ? A: Fundamental packages which may end up in stdlib; bootstrap anything else outside the tree

Q: What to do with current golang.org/x? A: Stop accepting non-fundamental packages; move unmaintained packages outside the tree and assign domain experts as maintainers

Q: How to fix the Go team bandwidth issue? A: Keep stdblib small; remove packages like log/syslog in Go 2.0; to stay relevant, a language must evolve, which will be easier and faster with a smaller codebase

Q: How to make quality packages discoverable A: Good code should live anywhere; extend godoc.org with package ranking to reflect code and documentation quality as proposed in golang/gddo#320 (Proposal for Package Ranking)

As a Go developer, i'd prefer the core team members put as much time as they need on the evolution of the language, stdlib and the go tools.

[adg]

@cznic comments like just "no," are not constructive. If you disagree, provide a rationale. Thanks.

[matrixik]

Q: How to make quality packages discoverable A: Good code should live anywhere; extend godoc.org with package analysis to show code quality, API stability and maybe popularity

You mean something like joining godoc.org and http://goreportcard.com ? That would be nice. API stability marker also would be nice (like last breaking change happen in commit ...).

[namsral]

@matrixik Yes, that's what I have in mind.

One thing I'd like to add is to take into account a package's documentation and examples; doc.go, example_test.go.

[dlsniper]

@nathany et. al. I've opened https://github.com/golang/gddo/issues/445 as per suggestion you had so that we keep this one clean. Looking forward for comments there. Thank you.

[SamWhited]

Q: What should go in golang.org/x ?

As the proposer of #16257 (and other subrepo requests in the past, some accepted, and some not), my criteria for if I propose a project be moved into the x/ namespace is generally "do I think this would be useful in the standard library eventually, eg. context, or do I think it could be vendored in the standard library to support other packages, eg. that sasl proposal?)

If the answer to both of those questions is no, I probably would prefer to keep it to myself.

The only place I make a slight exception to this is when I consider security and crypto related packages. Eg. with the SASL proposal I feel that it's important to have a single, known-good, well-vetted implementation. Fair or not, having it under golang.org/x/ will make people assume it's the "recommended" solution, and the process (and intelligent developers that are a part of this community), would hopefuly ensure that nothing horrible slips through the cracks, so security packages get slightly different considerations for me.

Q: How to fix the Go team bandwidth issue?

I think the answer hear is to allow third parties and trusted domain experts to act as the maintainer of some packages in the subrepos. There's no reason the Go team has to manage every single package themselves. This doesn't mean all authors immediately get commit access or review access, but it might mean that eventually long-time contributors and others that have shown that they can be responsible and maintain the standards expected in the subrepos could take some of the review and merge burden off the core team.

[nigeltao]

I don't think it's in your enumeration: another reason for something to live under x/ is being a dependency for something else under x/.

For example, Brotli is a compression format. A Go "package brotli" might or might not meet the bar for x/ inclusion on its own merits, but e.g. if we wanted WOFF2 (Web Open Font Format 2) support somewhere under x/, it would depend on a Brotli decoder, so I'd expect a "package brotli" to also have to live somewhere under x/.

[mpvl]

Data point: recently the dependencies of core on x/net, x/crypto, and x/text have been steadily increasing. There are now 9 vendored packages in core with more to come.

[adg]

@nigeltao I added that to the list. FWIW, I think we can use internal/ to side-step that issue.

[nathany]

@adg I am curious how licensing may impact the use of internal/ in core or current x/ tools. For example, if fsnotify wasn't under the Google CLA, could guru or godoc or go build still use fsnotify? Likewise with brotli or other core/low-level packages like USB.

Either way, (IMO) the CLA used should be orthogonal to the location of the code, e.g. grpc-go still has the Google CLA.

[bradfitz]

@nathany, yes, we can only use CLA-compliant code in the core. That's briefly addressed by the bullet at top:

To use the same consistent license (with CLA) that the project uses.

But it's not very explicit.

[nathany]

The fsnotify codebase originated from code contributed to Go, then moved to x/exp, and now maintained in a separate GitHub organization. At one point we had all contributors sign the Google CLA, but I haven't been good about enforcing it for patches since. 🙁

If a project like gomobile were moved outside of x/, I assume the CLA would still be required. Is there tooling to help enforce it? Does it require a Gerrit workflow or is it independent of Gerrit?

Contributions to GitHub's repositories have a CLA check, so it is something that can be done on GitHub, though I don't know the implementation details.

That would address the technical issue of enforcing the CLA -- independent of repository location.

The other issue I personally have is that the non-endorsement license and Google CLA feel awkward for fsnotify specifically. It isn't currently a Google-run project, but needs the CLA and license for a potential future use in core. Which ends up feeling like an endorsement. Heh.

I'm not sure if this applies to any other projects that may be moved out of the x/ subrepos? Projects that may no longer under the review of Go team members (that's kind've the point of all this), but are low-level enough that they still may be useful as internal/ packages in core.

[mdlayher]

Hey all, apologies for the ping, but has there been any movement on this at all? I can see that this proposal has been referenced from several others (including one of my own).

[bradfitz]

No movement yet. Updates will be posted here.

[rakyll]

Similar to my previous point earlier on this thread, @mattetti says on https://github.com/golang/go/issues/18497#issuecomment-270202473, a central point to discuss ecosystem-wide APIs and libraries is a hard requirement for the success of anything beyond the core project.

I think regardless of what's our decision for the new policy regarding to the x repos, we need a migration/replacment plan for those who finds it productive to use Go mediums.

[mattetti]

I'd like to resume this discussion since it seems to be critical for the evolution of the stdlib and the extension of the ecosystem. There are a lot of very valid points made in this thread and I believe everyone cares to keep the burden on core go developers as light as possible while also benefiting from a great ecosystem.

From what I read I saw two proposed ideas to address the expansion of "core" packages:

1. improve godoc.org (@adg)
2. create the equivalent of Linux subsystems / official interest groups (suggested by @rakyll / @jessfraz )

What's clear to me is that we need to scale and even though my proposal #18497 was to provide an official package for interoperating about (but not necessarily doing) X to quote @adg, I honestly don't care too much where it's going to live as long as it's clearly discoverable and have a thorough review process. Doing that outside of the Go project umbrella is extremely difficile and I think that's the heart of the issue.

At this point I think we could have 2 non exclusive strategies:

1. The first one would look at what can we do to make godoc.org a more useful place to find high quality, non-std library packages, start on that work in an incremental manner.

2. The other would be to create official interest groups which would have for mission to support "subsystems". Their role would go from maintaining and organizing a list of existing libraries. Defining standards and advocating for them. Code reviewing submitted 3rd party libraries and implementing interop solutions or actual features. These groups might benefit from current tooling and occasional access to core team members but wouldn't be restricted by release cycles and core team vetting. Off the top of my head, here are interest groups I can think of: media (audio/video), images, hardware, mobile, GUI, crypto, web services (similar to go-kit), dev tooling, machine intelligence... Having those groups defined, official and the process well documented should help us scale. That would obviously require a clearer delimitation between the standard library and non-standard packages. Really high quality packages that became standard don't ever have to "graduate" to the standard library but they have to be looked after by more than 1 owner (company or developer) and I believe those groups should be under the official Go umbrella.

If this idea sounds good, I would suggest to start by a couple special interest groups with momentum such as media and let them figure out how they want to get organized. I would assume an official team, repo, toolset and mission would be required. Then exposure with guidance are probably going to be the most valuable things that the project can provide to those teams.

What do you think?

[rakyll]

/cc @spf13

[cep21]

Is there a longer, more abstract, article or general guidance around what code or libraries belong in Go's standard library?

[nathany]

"Once you have experience with the API being good, then it might make sense to promote to a subrepo or eventually the standard library" - @rsc https://github.com/golang/go/issues/18497#issuecomment-270387898

One big issue I see with this approach of "build it elsewhere, then promote it" is the licensing and CLA. If all contributors need to sign the Google CLA for inclusion into the subrepos/std, that not only involves a lot of administrative work, it could block promotion if some past contributors decide against it or are unreachable. (I don't know the remedy to that -- not a lawyer)

If the current system of subrepos is replaced with special interest groups (SIG) and the packages they support and recommend, this idea of promotion would only be relevant for standard library.

Then perhaps the licensing/CLA could be more relaxed (whatever each SIG chooses), with no expectation of the licensing or location of the code changing, or any of the overhead required to "promote" a library.

"all of these are on hold for #17244, which in turn is essentially blocked on understanding the long-term plan for package management. I intend to focus on that for the next six months." - @rsc

Thanks for putting so much time into this. I'm really looking forward to kicking the tires on the new dependency management tool once the initial version is made publicly available.

I :heart: the Go standard library, especially net/http as compared to the myriad of http server and client options in other language communities I'm familiar with. Go makes it a lot easier to get started.

My long-term hope is that through a combination of processes (SIGs) and tooling, the Go community can reap many of the benefits of the "batteries included" (large std library) approach for a wide range of libraries developed by the community in a distributed fashion.

If it works extremely well, I could even imagine parts of the std moving to external SIG-supported libraries in Go 2.0 (i.e. image/*). An ambitious goal might be to make third-party package management so seamless and special interest groups so well organized that there no longer is a discernible advantage for a library to be in std vs. located elsewhere.

[glycerine]

Summary: a search engine that finds and ranks Go projects by code quality is being called for.

[dgryski]

Rust's bug tracking discoverability and ranking: https://github.com/rust-lang/rfcs/pull/1824

[rsc]

As noted above, this issue is basically on hold for package management to settle. I think a lot of the pressure on the x repos is caused by bad broader support for package management.

[ianlancetaylor]

Taking off hold to see if we can make any progress on this as 1.13 rolls out, as prompted by a comment on #19700.

[kardianos]

I'd like to find resolution on this topic, at least for a time. I see the current categories of content in the x repos being:

• Tools used by the Go project (x/review, x/build, x/tools, x/website, x/perf)
• Packages that are or may be vendored into the final build (x/crypto, x/net, x/sys, x/text, x/arch) but are also generally useful.
• Foundational packages the Go team wants to support, but may not want to tie to a specific release and backwards compatibility promise) (x/sync, x/time, x/oauth2)
• Experiments (x/xerrors, x/mod, x/net/context) that may make it to the std lib in some way (or graduate to a tool), often part of some proposal or larger process.
• Experiments (x/exp, x/mobile) that probably won't make it to the std lib.

I think the effective policy these last few years has been:

• Allow tools used by the go project
• Allow packages that may be vendored by the Go std lib / runtime.
• Allow experiments that may make it it to the std lib.
• Don't allow other additions.

In light of the effective policy, issues on hold for this issue should be rejected unless they meet that criteria. Secondly, existing repos such as x/exp and x/mobile should be considered to be moved outside or explicitly marked as grandfathered in as part of the policy.

The x/time/civil proposal does not meet these policies, I would expect it and the CL to be closed.

[bradfitz]

@kardianos, thanks! That looks pretty accurate.

Keep in mind that a major reason (perhaps the only one) for having x/exp not be on GitHub already is for the same Gerrit code review tool, same builders/trybots/etc. So if we moved it, we'd need to decide whether we're just changing its import path (e.g. golang.org/x/exp/utf8string -> goexperiment.net/utf8string) or also moving it from the https://go-review.googlesource.com Gerrit instance to the misc https://code-review.googlesource.com Gerrit instance. But it'd need to remain on Gerrit w/ builders, because it's where many of the Go team are most comfortable.

As for x/mobile, it was originally meant to be official, and for some time had official staffing. It wasn't until some reprioritizations happened that the Go team stopped working on x/mobile. It might do more harm than good to move it at this point.

[comaldave]

There is a great need for standard libraries, but they do not need to be done entirely by the Go team. If you increase the size of the Go Team, you also increase internal communication and decrease productivity. I am very much in favor of the divide and conquer approach. You do not need more people, but you do need more teams, SIG they are following standards and are producing reliable code that everyone can use. BindData was a very useful package, so I vendored it and wrappered it, one day it was gone. Not a problem for me, but an outcry from the community. I love using other peoples packages, but for my clients, I only do so after vendoring and code review unless the package is extremely popular and the maintainers well respected. I would respectfully suggest that when someone has a mature package of what they consider high quality, that they be able to submit it to a review process. I have some very specific ideas of how to conduct the review process, track the reliability of the coders and the code reviewers. For me, selecting packages to use is a painful process, that should not be the case.