search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
122.84k stars 17.51k forks source link

proposal: Go 2: disallow imports of external packages in library packages #25588

Closed metakeule closed 6 years ago

metakeule commented 6 years ago

Proposal for Go2: Disallow imports of external packages in library packages

Definition of the term main package

A package with the name main containing a function main (aka a program).

Definition of the term library package

A package that is not a main package.

Definition of the term external package

A external package is a library package, that is neither part of the standard library, nor a package that has the importing package as a subpath.

Examples

Proposal

This proposal would not change the rules for imports of standard packages, it would not change the rules for imports of subpackages and it would not change the rules for imports from any main package.

It would only forbid a library package to import an external library.

Examples

We have the following packages:

According to this proposal the following imports are allowed:

The following would be rejected:

Benefit:

The standard libraries are not affected and since they are released as a whole, there are no package management issues with them anyway.

But how can a library package foo then depend on a library package bar?

It won't. However a function of foo can consume an interface that is implemented by some type of bar.

The main package then would import both library packages, passing the required value to foo. In order for that to work, the developer of foo would offer example glue code.

The developer using package foo, copies the example glue code for the integration to his main package.

So what happens, if any of foo and bar changes in an incompatible way?

We assume that the principal functionality offered of bar would not change. If so, it would make sense to rename it.

However what could change is the exported symbols, the initialization routine etc.

If so, the main package would not compile. Since the glue code is now owned by the developer of the main package, it can be easily changed without foo having to be updated. In the worst case one could create a wrapper implementing the needed interface.

In combination with reproducable builds (e.g. vgo) main would not simply stop working without intervention of the user.

UPDATE

After I bit of reasoning, it seems like it would be better to apply this restrictions only if the importing library is "published", where "published" would be defined as having a domain name as part of the package path. These would give some freedom to mono-repos and the standard library (which was excluded anyway).

pciet commented 6 years ago

Do you have a real world example that is improved by this feature?

metakeule commented 6 years ago

@pciet

Well we have the package management problems with Go since day one. Countless variants of "solutions" and the complexity involved.

If these dependencies could be avoided in the first place (and that's what would be inforced by this proposal) the better for us all.

The only unavoidable dependency is from the main package. If you write it, you "own" your imported external libraries, and you make sure they work. Only after a while, when updating, the problems arise.

Now if you have library package foo depending on type Bar of package bar and you also need package baz that is depending on type Bar of package bar that changed in an incompatible way, you have a problem. With this proposal foo and baz could not reference type Bar so there would be no dependency and no problem. Inside main you could easily wrap around bar.Bar to fullfill the incompatible interface of foo without any action from the developer of foo.

mbenkmann commented 6 years ago

If neither bar nor foo may depend on the other, nor on a 3rd library, where is the definition of the common interface known by both foo and bar. Your proposal would only work with interfaces from the standard library.

metakeule commented 6 years ago

@mbenkmann

If foo is meant to be used with bar, foo would define an exported interface type that is implemented by bar at the point when they are compatible.

BTW: This encourages a culture of similar interfaces for similar tasks within the community, so libraries can be swapped.

metakeule commented 6 years ago

A good extension of that proposal would be to forbid init functions within libraries. Then every initialization inside a library would be done via a function call from main. That would make it more transparent, that there is initialization code and if this code changes there is a chance that the function name changes and such a change would get noticed at compilation time (compared to a change within init which would get unnoticed).

4ad commented 6 years ago

I appreciate the sentiment behind decoupling external libraries and the general idea of making the dependency graph wider rather than deeper, but this is unviable without covariant types, without generics, without disallowing packages that have side effects, and without much stronger type inference (and without surely many other things). I don't see how it would ever be possible in a language like Go.

This would make more sense in a purely functional language.

Since this is much too radical, I'd prefer some mechanism, tooling, or policy (or a combination of all three) that would encourage, or somehow help development of these "independent" libraries through some other means rather than by adding restrictions to the language.

metakeule commented 6 years ago

@mbenkmann The interface of foo does not have to be known to bar. This is the beauty of interfaces in Go.

mbenkmann commented 6 years ago

You are assuming interfaces that do not include ANY custom types. How do you define useful interfaces for a graphics library if you don't even allow abstractions for Rectangle. You want those interfaces to use [2]int for a Point?

metakeule commented 6 years ago

@4ad Concerning generics it seems the Go team wants to have them in Go2. Your statement is very broad. Can you give a concrete example where two packages could not be used in tandem with this proposal?

metakeule commented 6 years ago

@mbenkmann Why not? Or x,y int. For sure the libraries would look a bit different, but I question, that they would look worse. One would definitely make more use of builtins which would be another good side effect IMHO.

4ad commented 6 years ago

@metakeule I think @mbenkmann's last example is a good one. In general, to be really useful packages need to share data (through types), not just behavior (through interfaces). You can do everything solely with interfaces, but that doesn't make for a good programming model in a language like Go (Lisp would be fine here).

metakeule commented 6 years ago

@4ad I fail to see why that would result in worse programming. Decoupling is a good thing and it is why we have interfaces in the first place. And decoupling is far more important between packages. Where would we be, if not every package used io.Reader and io.Writer?

mbenkmann commented 6 years ago

And what about a Dialog? With x,y,width,height, a title, a message text, buttons...?

justinfx commented 6 years ago

I don't understand how this would work if library foo needs to build its implementation details on top of library bar. There is no main involved here. Someone is trying to provide others with a library that may be consumed by another library or a main. Your proposal would make it impossible to have private implementations that use existing libraries without asking some other person to pass you through an interface?

So let me make sure I understand this. If there is library "A" which wants to use some sort of embedded kV database or cache, it would instead code against an interface and ask the consumer of this library to pass it in? Now let's say you have someone else with library "B" that uses "A" and third person with main that uses "B" 

main -> B -> A

According to your proposal, main would have to import the transitive key value db dependency, and pass it through to B as an interface, which would then have to pass it though to A as an interface?

I'm very lost as to why we would want this situation.

mbenkmann commented 6 years ago

Maybe we should just exchange []unsafe.Pointer in our interfaces. Yay. Who needs type safety.

metakeule commented 6 years ago

@mbenkmann It has nothing to do with type safety.

package foo

type Rect interface {
   X() int
   Y() int
   Width() int
   Height() int
}

func UseRectangle(r Rect) {
...
}
package bar

type rect [4]int

func (r rect) X() int {
return r[0]
}

func (r rect) Y() int {
return r[1]
}

func (r rect) Width() int {
return r[2]
}

func (r rect) Height() int {
return r[3]
}

func MakeRect(x,y, width, height int) rect {
   return rect{x,y,width,height}
}

If the Rect interface would be used and offered by other libs you could combine them easily (not possible with depencies on structs).

Perfectly type safe. (an [4]int would also be type safe BTW)

metakeule commented 6 years ago

@justinfx

Yes you did understand correctly.

We would want it for 3 reasons:

mbenkmann commented 6 years ago

So you are suggesting that instead of depending on a 3rd library implementing standard data structures, every library should contain a copy of the relevant code. Look at how much code your package "bar" needs just for a silly little class like rectangle. Take a real world example:

https://godoc.org/github.com/veandco/go-sdl2/sdl

Now I want to offer a sprite library. You're telling me I can't use sdl.Rect. I have to copy the code into my library, standard code for computing intersections, unions etc. of rectangles. Your suggestion comes down to NOT USING LIBRARIES.

metakeule commented 6 years ago

@justinfx


func main() {
   b.New(a.New(kv.New()))
}
justinfx commented 6 years ago

@metakeule yuk. So that means the main has to now be aware of how to initialize library B with the transitive library A dependency. And you have to do this for every dependency that library A wants to use, which means every library in between has to expose an injection point.

Furthermore, it means that if library A wants to use a 3rd party dependency, it has to now create its own interface definition to match that dependency. But there is nothing to say that any other similar suitable replacement will conform to that interface you have just invented for exposure.

metakeule commented 6 years ago

@mbenkmann

The sdl library would have to be rewritten in order to be useful for other libraries with this proposal. (e.g. offer methods on Rect etc.) But since we are talking about Go2, it would probably have to be rewritten anyway. And it would make perfectly sense to make a sprite library independant from a specific sdl implementation, doesn't it?

mbenkmann commented 6 years ago

Okay, so we've reached the point where every existing library has to be rewritten to be used with Go2. Not gonna happen. Go2 will be compatible with existing code.

metakeule commented 6 years ago

@justinfx

What do you mean with "conform to that interface"? Do you mean "conform to the semantic of the interface", because otherwise the compiler tells you...

metakeule commented 6 years ago

@mbenkmann Not every, but a lot. But nothing from the standard library.

Also I heard, Go2 should be able to import Go1 packages. So there would be a way to distinguish them and the rewrite could be done incremental.

metakeule commented 6 years ago

@justinfx Also that is just like normal Go interface work. Never had an issue with it. I guess the trick would be to avoid relying on 3rd party libs whereever possible (which is also a good practice today) and make the exposed "API" as small as possible. Probably be making use of builtin types where possible. Mind you: that is just in between packages; inside a package hidden structures could have all the custom types.

Whatever, I think it is an interesting thought experiment. We can see, if something useful arises from it.

justinfx commented 6 years ago

What I meant is that if I use an external library Foo in my own library, I have to spec an interface for it so that the chain of dependers above me can supply me with an implementation. Now let's say there is exactly one existing solution for Foo. The interface I spec out basically says "I know there is nothing out there besides Foo to match this. Please just pass me Foo so I can work. And let's hope that something other than a mock will also conform to this interface that I have now been forced to expose as my public api.

My point is that your suggestion turns a private implementation consuming a private dependency into the requirement to expose a public interface so that dependers can pass you everything you need. Your example of passing A to B in main just illustrates how transient types now have to be leaked into the main. Before, main never needed to worry about the private types of B. Now your main has to reach into all the dependencies to pass through chains of types. Yes I would have originnaly seen all the dependencies listed in my dep manager lock file, but I never had to concern myself with their apis. Now I would have to chain them up to satisfy that embedded cache implementation that I didn't know I had to think about, which exists two levels of dependencies away from my main.

The goal is noble, to try and force people to limit their use of external dependencies in libraries, but it seems this solution is meant to make it annoying and gross to even use external deps in a library so as to deter people from doing it.

metakeule commented 6 years ago

@justinfx Ok, before we can agree to disagree, let me just point out that IMHO what you call a "private dependency" isn't a real private dependency since its crossing the package borders. That becomes apparent if the dependency breaks and 3rd party users are affected. Then it is not private anymore and the user of your code needs to dig through your code in order to understand what the problem is. A real private dependency can IMHO only be within a repo and packages and subpackages (which could depend on each other without restrictions according to this proposal).

With bubbling up the loose behavioral dependencies to the main package, you pay the price of importing packages in the beginning (and it could be a deciding factor which library to use). I think this is more adequate then paying the price / biting in your ass after months or years when you are in maintenance mode and on other projects. Also keep in mind that the more dependencies you have (= your project gets larger), the more likely any version conflicts are, some of them might not even be solvable.

I prefer to know my risks upfront.

metakeule commented 6 years ago

@justinfx It would be the duty of the package expecting a certain semantic from an interface to document the expectations. Also to offer example code for integration, so that the user can simply copy the code and does have to figure out the dependency by searching/looking up.

So in your example, package a would have the example glue code:

package main

import 'kv'
import 'a'

func main() {
   a.New(kv.New())
}

and package b would have the example glue code

package main

import 'kv'
import 'a'
import 'b'

func main() {
 b.New(a.New(kv.New()))
}

so users of package b would not need to look up the example glue code from package a but could just copy the example glue code from package b straight away.

It is just a question of culture and documentation.

metakeule commented 6 years ago

The problems described here: https://sdboyer.io/vgo/failure-modes/ (diamond problem) could be completely avoided.

dans-stuff commented 6 years ago

This is impossible with Go as it stands, for one main reason - interface{Get() Interface} is not satisfied by Get() Implementation. This means main has to write, each time, wrapper code between almost all packages. Here's a simple example using hypothetical "MySQL" db package and "Sqlx" package.

https://play.golang.org/p/zYaBPs6nuKz

package main

import "fmt"

// MyDB.GetTransaction has no way of returning a SqlxTransaction.
// Write wrapping code to allow them to work together.

type DBWrapper MyDB

func (d DBWrapper) GetTransaction() SqlxTransaction {
    t := MyDB(d).GetTransaction()
    return &t
}

func main() {
    fmt.Println( Get(DBWrapper(MyDB{}), "nineteen characters") )
}

// package sqlx

type SqlxDB interface {
    GetTransaction() SqlxTransaction
}

type SqlxTransaction interface {
    Get(string)
    Commit() int
}

func Get(db SqlxDB, key string) int {
    t := db.GetTransaction()
    t.Get(key)
    return t.Commit()
}

// package mysql

type MyDB struct {}

func (m MyDB) GetTransaction() (tran MyTran) {
    return 
}

type MyTran string

func (m *MyTran) Get(key string) {
    *m = MyTran(key)
}

func (m *MyTran) Commit() int {
    return len(*m)
}

The only realistic solutions to this are: change Go to allow methods to satisfy an interface if their return values satisfy the interface; always return interface{} and do typecasting in sqlx; or have sqlx provide "wrapper.go" that must be copied into the main package to allow it to actually work correctly.

justinfx commented 6 years ago

Ok, before we can agree to disagree, let me just point out that IMHO what you call a "private dependency" isn't a real private dependency since its crossing the package borders.

No this is confusing private/public api with physical files contain source code. In the current ecosystem I would not be leaking details of the dependency api at all. My api would be smaller. But in your proposal, I would now have to create a larger api to allow dependency injection, via interfaces that I have to define so that dependers can chain up calls across the package boundaries. 

package main

import 'kv'
import 'a'
import 'b'

func main() {
 b.New(a.New(kv.New()))
}

What was once a private implementation for B to use A has now become a public contract for main. If B wants to stop using A and switch to C, it cannot unless A and C share an interface. Main would be broken if B wanted to change implementation details. But your proposal implies that for any usage of a 3rd party lib, it must be exposed as a public interface and becomes harder to be refactored out. What was once just dependency source code on disk is now a real public api concern all the way up to main.

What you suggest makes sense as an optional approach a library could take to allow for alternate implementations of a major component like an external database. But for something concrete and embedded like text processors, or other supporting utilities, it is heavy handed to require all of them be abstracted into interfaces and the dependers have to figure out what projects to use to pass to them or be told "try using these external deps".

metakeule commented 6 years ago

@dantoye Yes, it has to be in conjunction with https://github.com/golang/go/issues/8082

davecheney commented 6 years ago

No dependencies between external libraries

You want this, if library a cannot depend on library b then you cannot compose software unless

This proposal is unworkable and should be rejected.

davecheney commented 6 years ago

No dependencies between external libraries

If library a cannot depend on library b then you cannot compose software unless

This proposal is unworkable and should be rejected.

metakeule commented 6 years ago

@justinfx It encourages to reuse the same interfaces for the same tasks: A good thing. Yes main would be broken and point the user to the new library C instead of hiding it. As a user I want to know such things. Maybe malicious code would be injected and I have to audition the new dependency?

metakeule commented 6 years ago

@davecheney You know enough of Go to know that this 

an since you’ve propose that libraries cannot depend on each other, all those interfaces would have to be defined in the standard library.

is not true. You may as well define the interface in the receiving package without any other package having to depend on the interface type. That is the beauty of Go interfaces that the definition of the interface type is independent from the place where it is used.

Also there is "unnamed", "in the place" interface definition.

func F(in interface{ A()}) {
}

Can't believe you forgot that.

metakeule commented 6 years ago

@davecheney They would get really useful in conjunction with https://github.com/golang/go/issues/8082

dans-stuff commented 6 years ago

@metakeule #8082 only speaks about interface matching, no?

Whereas this would require func() Implementation to be assignable to func() Interface, which is impossible.

What would happen if I declared var X func() Interface; x = func() Implementation {}? X just doesn't have the same type...

metakeule commented 6 years ago

@dantoye Yes you are right.

It would have to be rewritten like this:

package main

import "fmt"

type DBWrapper MyDB

func (d DBWrapper) GetTransaction() (interface {Get(string);    Commit() int}) {
    t := MyDB(d).GetTransaction()
    return &t
}

func main() {
    fmt.Println( Get(DBWrapper(MyDB{}), "nineteen characters") )
}

// package sqlx

type SqlxDB interface {
    GetTransaction() (interface {Get(string);Commit() int})
}

type SqlxTransaction interface {
    Get(string)
    Commit() int
}

func Get(db SqlxDB, key string) int {
    t := db.GetTransaction()
    t.Get(key)
    return t.Commit()
}

// package mysql

type MyDB struct {}

func (m MyDB) GetTransaction() (tran interface {Get(string);Commit() int})) {
    return 
}

type MyTran string

func (m *MyTran) Get(key string) {
    *m = MyTran(key)
}

func (m *MyTran) Commit() int {
    return len(*m)
}

I admit, it is a bit ugly, but maybe we could come up with a better API in this case.

metakeule commented 6 years ago

I also would prefer to have a nicer syntax for "in place" anonymous interfaces.

dans-stuff commented 6 years ago

@metakeule so your proposal also requires that people only ever return interfaces, never structs?

metakeule commented 6 years ago

@dantoye No. Why do you think that? The normal case for an implementation would be to return structs. We just couldn't accept interfaces with methods returning or receiving structs (in exported functions/methods).

dans-stuff commented 6 years ago

@metakeule in your example, you now have the issue of privacy. If MyDB wants to be able to work with a MyTransaction, it will have to accept an interface (because sqlx would be the one passing it back), and then MyDB would have to type-assert it as a MyTransaction in order to access private methods or fields.

Type safety goes out the window for this, no?

Additionally, it seems like it would be a near-requirement for the entire surface of any library to be abstract, just consisting of interfaces, correct?

metakeule commented 6 years ago

@dantoye That is no different to Go1: sqlx would return an interface anyway in your example. That would have to be type casted to a specific type, if MyDB wants more. That cast may go wrong, thats why you do

if my, ok := trans.(*MyTransaction); ok {
}

The ok is only true if the cast is successful. Nothing new.

Additionally, it seems like it would be a near-requirement for the entire surface of any library to be abstract, just consisting of interfaces, correct?

No:

// implementation: no need for interfaces here
package a

type s string

func (s) String() string {
  return "hi"
}
func New() s {
   return
}
package b

type S interface {
   String() string // just the receiver of a dependency needs an interface
}

func B(s S) {
x := s.String()
....
}
package main

import (
  "a"
  "b"
)

// no need for interfaces here too
func main() {
  b.B(a.New())
}

Also there are in place struct definitions like

func New() struct{ A string; B int } {
    return struct{ A string; B int }{"A", 3}
}

that might also be used without importing a package


func Use(s struct{ A string; B int }) {
   a := s.A
  ...
}
AlexRouSg commented 6 years ago

Why should everyone be forced to code in such a manner and prevent almost all Go1 packages to be upgradeable to Go2 without manually rewriting? This will essentially split the ecosystem and force people to choose between using external packages and having the benefits of Go2.

I do not see a reason why this cannot just be a opt-in flag or a third party tool to warn against importing external packages.

metakeule commented 6 years ago

@AlexRouSg I think it would just have the benefits, if it is enforced. Maybe a translator could be written that would automatically rewrite library code, so that just the glue code must be rewritten.

AlexRouSg commented 6 years ago

@metakeule

If it is possible to create such a tool, then why can you not simply fork the packages and use the tool on it thereby only enforcing it on your packages/programs?

Or why can't people just release 2 packages, one normal and one translated?

ianlancetaylor commented 6 years ago

Go 2 must be largely if not perfectly compatible with Go 1. A change that breaks pretty much every existing Go package and a good chuck of existing Go documentation is basically a non-starter for Go 2.

If I'm reading the proposal correctly, the main package is in charge of importing essentially every top level package that it uses, and is, further, responsible for somehow hooking them up. If package "a" needs values created by package "b", then the only wait it can get them is if the main package calls "b" functions to create them and passes them to "a". Requiring the main package to do this in all cases seems impossible awkward.

4ad commented 6 years ago

I want to point of that this proposal implies some implicit ordering between example.net/foo/bar/baz, example.net/foo/bar and example.net/foo. IIUC you propose that example.net/foo can import example.net/foo/bar/baz as a non-external package, but the other way around example.net/foo/bar/baz can import example.net/foo only as an external package. In other words, you are introducing restrictions on who and how can a package import based on the syntax of the import path. (internal also restricts who can import what using import path syntax, but it simply disallows some imports, it doesn't change the normal rules about the nature of a package, the how.)

While perhaps some sort of distinction between external and non-external packages is warranted (I am not convinced) making this distinction based on the fact that import paths appear to be hierarchical is a deep departure from the way Go works today (internal and vendor notwithstanding). Today, package topology is undetermined by the apparent hierarchy present in import paths (in fact this independence is a sometimes a pain point for newcomers to the language, who expect it). foo/bar can import foo, but just as well foo can import foo/bar (but not at the same time), and people make frequent use of this. Your proposal forces people to chose only one possible option.

In fact, foo/bar and foo might not even be related at all. In general, for clarity and simplicity people try (more or less) to keep the import path hierarchy related to the actual dependency graph, but this is not required. foo/bar might be more related to quux than to bar and quux/baz might be more related to foo than to quux.

To summarize, to a good enough approximation import path syntax merely tells us how to find a package. This includes special cases like internal and vendor which add special rules but still are about how to find packages (or whether to find them at all). You propose that the import path syntax would have additional meaning than simply telling us how to find a package, and this meaning you are proposing is incompatible with the way Go is used today.

dpinela commented 6 years ago

Besides the compatibility problems, this proposal would make the language almost unusable for all but the simplest of programs; programming at scale usually requires composing libraries, which this idea intentionally makes difficult and annoying.