proposal: Go 2: disallow imports of external packages in library packages

[metakeule]

Proposal for Go2: Disallow imports of external packages in library packages

Definition of the term main package

A package with the name main containing a function main (aka a program).

Definition of the term library package

A package that is not a main package.

Definition of the term external package

A external package is a library package, that is neither part of the standard library, nor a package that has the importing package as a subpath.

Examples

• package foo/bar/baz would be an external package when imported to bar/foo but not be an external package when imported to foo/bar

• package fmt would not be an external package since it is part of the standard library

Proposal

This proposal would not change the rules for imports of standard packages, it would not change the rules for imports of subpackages and it would not change the rules for imports from any main package.

It would only forbid a library package to import an external library.

Examples

We have the following packages:

• A foo/bar/baz (a library package)
• B foo/biz (a main package/ a program)
• C foo/bar (a library package)
• D bar/bop (a library package)
• E fmt (a standard library package)

According to this proposal the following imports are allowed:

• B importing any of A, C, D, E
• C importing A and E
• A,B,C,D importing E

The following would be rejected:

• D importing C or A
• A importing C or D
• C importing D

Benefit:

• No dependencies between external libraries.
• No references to symbols and functions of an external library in a library package
• No package management necessary (apart from reproducible builds for main packages).
• Decoupling, better maintainance
• Dependencies are owned by the user. May easily be replaced by changing glue code.
• Leads to larger repos in order to avoid the hassle of lots of glue codes for the users.
• Larger repos means less dependencies in the main package.

The standard libraries are not affected and since they are released as a whole, there are no package management issues with them anyway.

But how can a library package foo then depend on a library package bar?

It won't. However a function of foo can consume an interface that is implemented by some type of bar.

The main package then would import both library packages, passing the required value to foo. In order for that to work, the developer of foo would offer example glue code.

The developer using package foo, copies the example glue code for the integration to his main package.

So what happens, if any of foo and bar changes in an incompatible way?

We assume that the principal functionality offered of bar would not change. If so, it would make sense to rename it.

However what could change is the exported symbols, the initialization routine etc.

If so, the main package would not compile. Since the glue code is now owned by the developer of the main package, it can be easily changed without foo having to be updated. In the worst case one could create a wrapper implementing the needed interface.

In combination with reproducable builds (e.g. vgo) main would not simply stop working without intervention of the user.

UPDATE

After I bit of reasoning, it seems like it would be better to apply this restrictions only if the importing library is "published", where "published" would be defined as having a domain name as part of the package path. These would give some freedom to mono-repos and the standard library (which was excluded anyway).

[metakeule]

@4ad Well that was for practical reasons, but probably the rules could be harmonized with the rules existing today for internal packages (which have an internal path element).

However it is important to be able to distinguish de facto at least on the repo level. Since go import paths are just paths, that may happen to be URLs there is nothing within the import path indicating the top level of a repo (or the "project boundary"). But the compiler should not need to check repos and also a repo should not be a requirement.

Because of this and of the nature, how code hosting plattforms like github are typically organized, the hierarchie seemed reasonable. But that is not the core of the proposal, if there is a better way to find out project boundaries.

[metakeule]

@dpinela What you are saying is the first what comes to mind, if one is exposed to such an idea. It is a bit annoying. But it depends how the code is organized. An the real challenge is not to dismis such a proposal, but to try out, if and how code could be written at large with such restrictions in a way that minimizes the annoyance and to have a look, how much annoyance we are left with. I written some libraries this way and used them and it was not a big deal and I am not convinced that it wouldn't be worth the effort.

Everybody interested in this question "Could we avoid 2nd level dependencies completely with Go" make your tests, rewrite some code the way I suggested or come up with own ways and share your results.

[metakeule]

@AlexRouSg
Sorry, I've the impression that you missed the entire point of the proposal. I don't even know where to start...

[metakeule]

@ianlancetaylor

I really appreciate the effort of the Go team to keep the compatibility promise of Go1. However I think, it is a big mistake to determine that Go2 will be compatible to Go1 at this point where it is completely unclear what Go2 will be.

There is the great chance to revise some unfortunate design decisions of Go1 that would be totally missed with this commitment.

I am not expecting that this proposal is followed, however it shows a way to get rid of 95% of the package management problems we are facing and that let to complex tools like dep and similar tools (I am talking about complexity in code and workflow in difficult situations).

I think, this proposal is worth recognizing and maybe you get some inspiration that may influence some decision for Go2 that would result in lesser dependency problems.

I wouldn't throw it out of the window just because you can and there are so many objection from the standpoint of an existing eco system.

The proposal is an thought experiment (like every proposal?) and shows, that it is theoretically possible with the current Go1 to avoid 2nd level dependencies completely (apart from the standard library). Admittedly in a provoking form - as a proposal for Go2 - but with the intent to influence/inspire the design of Go2 in some non predictable way.

[metakeule]

@dpinela

programming at scale usually requires composing libraries, which this idea intentionally makes difficult and annoying.

In a surprisingly large number of cases, you can replace method definitions with top level functions that receive the needed object as normal argument. Now if your then top level functions have only a single purpose, they mostly need not that much behavior from an object. That behavior could easily abstracted via an minimal interface.

If this approach is taking by the immediate libraries and the low-level ones, it should not be hard or require much boilerplate to compose libraries.

[comaldave]

This is not a proposal that I like. I can appreciate that it is good to avoid 2nd level dependencies. There are times that using 2nd level dependencies saves time and makes development much easier, especially when creating reusable code shared by multiple teams.

If you choose to use this proposed standard for your code, I have no objection as you can restrict what you do without restricting what I do. If you are proposing that I have to do the same, then I object, I am a very lazy programmer. In fact my laziness is why I became a programmer. I will spend all day automating a one hour job.

If I am publishing, then I am likely to vendor any 3rd party packages without semver. Or any 3rd party package that I feel needs a quality review. Packages published by my team members, I do not vendor because I am already a stakeholder supported by the programmer.

I do thank you for offering some very sound advice. Developing useful abstractions is always a bit of a challenge.

[metakeule]

@comaldave I respect your opinion. Just saying:

Go did already teach us in lots of places (e.g. code formatting, error handling, missing of generics, static compilation) that restrictions can lead to freedom in the end. The idea is followed here.

[metakeule]

Ok, I've identified a real bummer:

With this proposal, it is not possible to create libraries that are easier abstractions over functions in a low level library, even if direct dependency on the low level types could be avoided. That means, use friendly simplifying libs have to be part of a low level project/repo and vice versa - which would be good from a maintenance and user viewpoint but is typically a social problems since the folks preferring low level work are typically not good at creating simple easy to use APIs and vice versa.

It is an interesting challenge to find a solution for this social problem.

[metakeule]

Also, it prevents you as a user to organize your glue code and own project internal abstractions over 3rd party libs in project specific libs.

So maybe we should allow packages that have no domain name as part of their path to import anything and only restrict the ones that have domain name (and are therefor considered to be "published"). Then the standard library exception would automatically included in that definition. Also mono-repos should get no restrictions as long as they choose their pathnames properly.

[metakeule]

See my UPDATE section in the proposal. This way an abstraction could be build internally and when it is finished, a pull request could be made to include it in the low level repo for general consumption (to solve the social issue mentioned above).

[ianlancetaylor]

I don't see why this proposal would address package dependency issues. Just because all interactions between packages must be mediated through the main package does not mean that a package can not change to being incompatible.

[deanveloper]

Seems way too Object Oriented... Also, nobody wants to write 500 interfaces before they start their code.

For instance, a Slack bot. In order to use a 3rd party slack library, I'd need to write an interface for Users, Groups, Channels, Messages, Emojis, etc. It defeats the purpose of using a library.

I don't want it to be in the main package though, since if I wanted to make a Twitter AND a Slack bot (managed by a single app), I don't want my twitter bot and slack bot to be in the same package. And as I said earlier, I don't want to write tons of interfaces before I get to write my program. It's just boilerplate.

[metakeule]

@ianlancetaylor

1. You get no diamond dependency problem.
2. Problems could always be fixed by the user in main.
3. No need to fork and change the upstream library.
4. No need to dig deeply into an upstream library in order to be able to fix it, since the dependent parts are visible through the interfaces.
5. In combination with the removal of init functions:
   • Freedom to include different version of a library side by side.
   • Freedom to make wrappers between incompatible types of two versions of a library.
   • Freedom to fine tune initialization order. ....

[arp242]

Even if we assume that the basic idea of this proposal is a good one, it's still extremely impractical.

If I develop library lib which calls pkg.Helper, then I would have to test this against two, three, or more different Helper implementations. If someone would report a bug then it can be very hard to track down, as it may depend on the specific implementation of the Helper function they're using (it could have a bug, different assumptions, etc.) Even worse, it may depend on the interaction between pkg.Helper and otherpkg.Helper.

Dependency injection with interfaces can be very useful, but like most good ideas it becomes a bad idea when pushed to the extreme and applied to every single case. This is no exception.

[OneEricJohnson]

Really interesting proposal, but it feels like handling of the issue is happening in the wrong place. Developers already have the ability to avoid importing packages outside of the standard library, simply by not importing packages outside the standard library. Which means, if that is already happening, then developers have a reason (good, bad, or horrible) for doing so.

In the packages I develop, the code imports various flavors of other packages:

• Utility libraries - "assert" and "errors" packages come to mind - build upon what's available in the standard library, and make certain use-cases easier.
• shared models and interfaces - packages that define structures used elsewhere
• large implementations of other people's APIs for which maintaining interfaces would be impractical - for example - Go implementation of the GitHub API.
• implementation details matter - using a specific implementation of an API to extract information not available by default - in the context of database drivers, extracting driver-specific error information for recovery logic or testing.

There are good reasons for all of these types of imports. Since the act of adding an export is an explicit act of coding, developers already have the opportunity to choose not to do that.

Treating the "stdlib" as somehow blessed assumes that the majority of Go code is being written for open-source consumption. In practice, who knows how much is written for private use, and in the context of those private uses, companies may build up their own extensions to the standard libraries. The Go team's appropriate reticence to add to the standard library makes this scenario very likely.

On top of that, the "main" package isn't really that privileged. When building larger projects, I've found a fairly logical approach is to build a "main" package that doesn't have much logic in it, and mostly accomplishes its work by delegation to a different package in the project. If nothing else, the existence of the library package that encompasses the functionality of the "main" package means that it is possible for downstream users of the "main" program to instead invent their own version of "main" based on importing that other library. However, for ease of implementation, that library package called by main is going to include concrete packages that it depends upon, not build up a whole additional layer of interfaces. That would be extra work that any sensible developer would try to avoid.

Experience shows that smaller interfaces are better. However, the approach of forcing everything into interfaces necessarily would lead to an increase in the average number of functions defined in an interface. These would not be well-designed interfaces, as they would, in many cases, simply be substitutes for large lists of functions defined on structures. I'm fairly certain someone clever would build a tool that would automatically define an interface in package A that matches all the function signatures of a structure in package B. In short, forcing this approach in libraries would likely lead to badly designed interfaces that still happen to be tightly coupled to implementations.

Overall, I'm intrigued by the design aim of the proposal, but the solution seems to miss the mark. I think it doesn't actually fully solve the intended problem, introduces a whole bunch of new ones, and would end up with too many poor interfaces. This calls for alternate solutions - perhaps options such as these:

• Improvement to linting tools to identify places where narrow interfaces could be substituted for concrete types (I'm a huge fan of gometalinter)
• Improve lint to flag packages that lead to diamond dependency problems. That is, library package A could import library package B, so long as library package B doesn't itself import packages outside of the standard library. Allowing the one level of external package reference would allow for all the most common use-cases, and generate pressure to introduce interfaces when the complexity increases.
• Adding "flavoring" to import statements - perhaps a "no-structs" import tag - which would hide all concrete structures from being imported, and only allow for constants and interfaces. Or perhaps a "function only" tag (so packages like "assert" could work.)

In general, to align with the totally practical approach that Go has taken, of requiring a demonstration of the the value of the proposal. In this case, perhaps by way of using some sort of linting to catch and discourage the specific presumed bad practice. When experience has borne out the value of eliminating the presumed bad practice, then go ahead and consider it for introduction in the language.

[metakeule]

Experience shows that smaller interfaces are better. However, the approach of forcing everything into interfaces necessarily would lead to an increase in the average number of functions defined in an interface. These would not be well-designed interfaces, as they would, in many cases, simply be substitutes for large lists of functions defined on structures. I'm fairly certain someone clever would build a tool that would automatically define an interface in package A that matches all the function signatures of a structure in package B. In short, forcing this approach in libraries would likely lead to badly designed interfaces that still happen to be tightly coupled to implementations.

Yeah, that might well happen.

In general, to align with the totally practical approach that Go has taken, of requiring a demonstration of the the value of the proposal. In this case, perhaps by way of using some sort of linting to catch and discourage the specific presumed bad practice. When experience has borne out the value of eliminating the presumed bad practice, then go ahead and consider it for introduction in the language.

I encourage everyone interested to try this out and see how far you can get. I got pretty far in my experiments, but obviously that would require a bunch of new lib designed with a different mindset.

[ianlancetaylor]

We aren't going to do this. It might possibly have been practical several years ago. Today it would just break all existing Go code, with no simple path forward. It's infeasible.