prasannavl commented 7 years ago

This has been talked about before, and often referred to the FAQ, which says this:

Method dispatch is simplified if it doesn't need to do type matching as well. 
Experience with other languages told us that having a variety of methods with
the same name but different signatures was occasionally useful but that it could
also be confusing and fragile in practice. Matching only by name and requiring
consistency in the types was a major simplifying decision in Go's type system.

Regarding operator overloading, it seems more a convenience than an absolute
requirement. Again, things are simpler without it.

Perhaps, this was the an okay decision at the point of the initial design. But I'd like to revisit this, as I question the relevance of that to the state of Go today, and I'm not sure just adding a section in the FAQ fully justifies a problem.

Why?

Complexity level is low to implement it in Go

Function overloading doesn't have to complicate the language much - class polymorphism, and implicit conversions do, and that with functional overloading does so even more. But Go doesn't have classes, or similar polymorphism, and isn't as complex as C++. I feel in the spirit of stripping to simplify, the need for it was overlooked. Because overloaded functions, are, for all practical purposes, just another function with a suffixed internal name (except you don't have to think about naming it). Compiler analysis isn't too complicated, and literally every serious language out there has it - so there's a wealth of knowledge on how to do it right.

Simpler naming and sensible APIs surfaces

Naming things is one of the most difficult things for any API. But without functional overloading, this makes this even harder, and in turn encourages APIs to be named in confusing ways and propagate questionable designs so much that its repeated use makes it perceived to be okay.

I'll address some cases, going from debatable to the more obvious ones:

Case 1: Go's standard library is filled with functions such as this
```
func (re *Regexp) FindAll(b []byte, n int) [][]byte { ... }
func (re *Regexp) FindAllIndex(b []byte, n int) [][]int { ... }
func (re *Regexp) FindAllString(s string, n int) []string { ... }
func (re *Regexp) FindAllStringIndex(s string, n int) [][]int { ... }
```
I believe this is confusing and there is no way to know what the functions' intents really are, unless we read the docs. (This is a common pattern we've become accustomed to over-time, but still isn't convenient in any way). Even worse, by avoiding function overloading, the language itself is now unfortunately encouraging people to create APIs with insensible names.

With function overloading, this could be
```
func (re *Regexp) FindAll(b []byte, n int) [][]byte { ... }
func (re *Regexp) FindAll(s string, n int) []string { ... }
func (re *Regexp) FindAllIndex(b []byte, n int) [][]int { ... }
func (re *Regexp) FindAllIndex(s string, n int) [][]int { ... }
```
I think this is much nicer, and simplifies the API as its essentially now just FindAll and FindAllIndex. It can reduced, and this also has implications that go into auto-completion, and grouping it in the API docs, etc.

The other part of the redundancy in this specific example relates to generics or variant types, but that's out of scope of this issue.
Case 2: The above example is actually one of the better ones.

http package:
```
func Handle(pattern string, handler Handler) { ... }
func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) { ... }
```
We've used it long enough so that this API has been ingrained into us by habit as to what it does. But let's think about it from a pure semantic perspective for a moment. What does HandleFunc mean? - Does it mean "Handle the Func itself", or "Handle it using the func"?. While, the team has done what's best without the capability of overloading - It's still confusing, and it's a poor API design.
Case 3:

Even worse:
```
func NewReader(rd io.Reader) *Reader { ... }
func NewReaderSize(rd io.Reader, size int) *Reader { ... }
```
I'm saddened that this style even became popular and into the standard library. Does this really mean "create a new reader of size", or "create a new reader size" (that is, semantically - an integer)?. And this list doesn't just go on, it encourages user to utilize the same design which creates a nasty world rather quickly.

Better default patterns for APIs

Currently one of the patterns that's considered a good way to pass options into APIs is: "The functional options" pattern by Dave Cheney here: https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis.

While it's an interesting pattern, it's basically a work around. Because, it does a lot of unnecessary processing, like taking in variadic args, and looping through them, lot more function calls, and more importantly it makes reusing options very very difficult. I don't think using this pattern everywhere is a good idea, with the exception of a few that "really fits the bill" and keeps its configurations internal. (Whether it's internal or not, is a choice that cannot be generalized).

The most common pattern that I'd think as a general fit would be:

  type Options struct { ... opts }
  func DefaultOptions() { return &Options }
  func NewServer(options *Options) { ... }

Because this way, you can neatly reuse the structures, that can be manipulated elsewhere, and just pass it like this:

  opts = DefaultOptions();
  NewServer(&opts);

But it still isn't as nice as Dave Cheney's example? Because, the default is taken care there implicitly.

If there was function overloading, it basically can be easily reduced to this

  func NewServer() { opts := DefaultOptions(); NewServer(&opts) ... }
  func NewServer(options *Options) { ... }

This allows me to reuse the structures, manipulate them, and provide nice defaults - and you can also use the same pattern as above for APIs that fit the bill. This is much much nicer, and facilitates a far better ecosystem of libraries that are better designed.

Versioning

It also helps with changes to the APIs. Take this for instance https://github.com/golang/go/issues/21322#issuecomment-321404418. This is an issue about inconsistent platform handling for the OpenFile function. And I find that a language like Rust has a much nicer pattern, that solves this beautifully - with OpenFile taking just the path, and everything else solved using a builder pattern with OpenOptions. Let's say hypothetically, we decide to implement that in Go. The perm fileMode parameter is a useless parameter in Windows. So, to make it a better designed API, let's hypothetically remove that param since now OpenOptions builder handles all of it.

The problem? You can't just go and remove it, because it would break everyone. Even with major versions, the better approach is to first deprecate it. But if you deprecate it here - you don't really provide a way for programs to change it during the transition period, unless you bring in another function altogether that's named, say OpenFile2 - This is how it's likely to end up without overloaded functions. The best case scenario is you find a clever way to name - but you cannot reuse the same good original name again. This is just awful. While this particular scenario is hypothetical - it's only so because Go is still in v1. These are very common scenarios that will have to happen for the libs to evolve.

The right approach, if overloaded functions are available - Just deprecate the parameter, while the same function can be used with the right parameters at the same time, and in the next major version the deprecated function be removed.

I'd think it's naive to think that standard libraries will never have such breaking changes, or to simply use go fix to change them all in one shot. It's always better to give time to transition this way - which is impossible for any major changes in the std lib. Go still has only hit v1 - So it's implications aren't perhaps seen so strongly yet.

Performance optimizations

Consider fmt.Printf, and fmt.Println and all your favorite logging APIs. They can all have string specializations and many more like it, that opens up a whole new set of optimizations possibilities, avoiding slices as variadic args, and better code paths.

I understand the initial designers had stayed away from it - but I feel it's time that old design decisions that the language has outgrown or, is harmful to the ecosystem are reopened for discussions - the sooner the better.

Considering all of this, I think it's easy to say that a tiny complexity in the language is more than worth it, considering the problems it in turn solves. And it has to be sooner than later to prevent newer APIs from falling into the insensible naming traps.

Edits:

Changed a few words which have incorrectly set an offensive tone to the issue. Thanks to those who pointed out, and my apologizes to anyone, whom I may have inadvertently offended - was not my intention.
Added versioning point
Added optimizations point
Better formatting

prasannavl commented 6 years ago

@pciet, great example! And we're only in v1 of the language. I'm scared of how these APIs will evolve without overloading.

pciet commented 6 years ago

For the context in database/sql case I’m not convinced function overloading is the right pattern to fix repeating APIs. My thought is change *DB to a struct of reference/pointer types and call methods on an sql.DB instead, where the context is an optional field assigned before making these calls to Exec, Ping, Prepare, Query, and QueryRow in which there’s an if DB.context != nil { block with the context handling behavior.

neild commented 6 years ago

The database/sql case seems to me like an argument for better API versioning rather than for function overloading.

prasannavl commented 6 years ago

@neild, precisely. And overloading is one of the most helpful ways to achieve API versioning. (As mentioned in the initial post already)

The problem? You can't just go and remove it, because it would break everyone. Even with major versions, the better approach is to first deprecate it. But if you deprecate it here - you don't really provide a way for programs to change it during the transition period, unless you bring in another function altogether that's named, say OpenFile2 - This is how it's likely to end up without overloaded functions. The best case scenario is you find a clever way to name - but you cannot reuse the same good original name again. This is just awful. While this particular scenario is hypothetical - it's only so because Go is still in v1. These are very common scenarios that will have to happen for the libs to evolve.

This just so happens to be an example, similar to the hypothetical scenario I mentioned. It's not that you can't do it without, but you have to jump through hoops, possibly with new packages even, to correct mistakes of old.

quasilyte commented 6 years ago

@neild, precisely. And overloading is one of the most helpful ways to achieve API versioning. (As mentioned in the initial post already)

@prasannavl, as already pointed out, this is not fair solution as it can break existing code that assigns function referring to it by its name. The whole "versioning" part can be misleading as overloadeding can be backwards-incompatible in unexpected ways (think about C code that calls your Go functions for example of less expected/popular example).

If you know the solutions to those complications or this is an acknowledged risk/tradeoff, please mention it in the first message. Current solutions do not cause these troubles, so the alternative should consider that.

That being said, I can think of a few ways - the obvious one being to pass the signature along - which might seem rather tedious, but the compiler can quite easily infer it - and in cases of ambiguity won't compile. These are "solvable" problems. But API design restrictions imposed by the language itself is not.

I also don't get you point how function overloading helps tooling.

It will get clumsier to "goto definition" of overloaded function because of candidates list.
If there will be much smart inference from the compiler, public API for tools should be provided in order for them to use that information. Otherwise very few tools will adopt it properly.
Tools that rely on the unique property of {package name}+{function name} combinations will break. And I would not presume that it is very trivial to fix all of those cases. It can be impossible to remedy by go fix.

I have a feeling that you underestimate the associated complexities at the whole picture. Sorry if I am wrong, but things like "quite easily" or "Complexity level is low to implement it in Go"/"just another function with a suffixed internal name" are confusing.

offtopic

My understanding is that C++ resolutions are hard due to other reasons (templates, namespaces/dependent name lookup are better examples). > But Go doesn't have classes, or similar polymorphism, and isn't as complex as C++.

prasannavl commented 6 years ago

as already pointed out, this is not fair solution as it can break existing code that assigns function referring to it by its name. The whole "versioning" part can be misleading as overloadeding can be backwards-incompatible in unexpected ways (think about C code that calls your Go functions for example of less expected/popular example).

Can be backwards-incompatible? Yes. Should it backwards-incompatible? Not really.

The key here is in how it's implemented. Let's think about what the variables here are - It's only the function parameters. So, if we can come up with a way, where the generated function names are tethered to the function parameters, this can solved nicely. (I do vaguely remember mentioning something on these lines in one of the comments before). That said, I can imagine this being a big problem in a language like C where dynamic linking is extensive. In Go, a majority of the use cases are static and ABI compatibility can also be forgiving (though I don't imply that things should break).

Let take an oversimplified example (oversimplified because this can't work due to other problems yet to be solved discussed in the comments before)

func DoSomething(in string)
func DoSomething(in int32)

And it's internal implementation would look something on the lines of:

func DoSomething__In_String(in string)
func DoSomething__In_Int32(in int32)

This shouldn't break C calling Go, or Go calling Go, or binary compatibility.

I also don't get you point how function overloading helps tooling.

I think it'd be fair to just refer to C# here. The tooling around C# does exactly what I mention, and it's a language with one of the most stellar language services and tooling. (PS: It handles a lot more complexity that isn't needed in Go, as the language is far more complex).

It will get clumsier to "goto definition" of overloaded function because of candidates list.

Sorry, I really don't see how. Each reference directly is tied to one definition, or it's incorrect code that won't compile.

If there will be much smart inference from the compiler, public API for tools should be provided in order for them to use that information. Otherwise very few tools will adopt it properly.

Possibly. While I'd like to explore the possibility on how the "tax" from this can be reduced, I am certain the best approach in most scenarios is what you mention.

Tools that rely on the unique property of {package name}+{function name} combinations will break. And I would not presume that it is very trivial to fix all of those cases. It can be impossible to remedy by go fix.

There are no cases that will break existing code. Go 1 tooling will remain compatible with Go 1 code. Hypothetically if Go 2 implements function overloading, the unchanged tooling just won't detect the new function that use overloading, which will of course require updated tooling.

quasilyte commented 6 years ago

Sorry, I really don't see how. Each reference directly is tied to one definition, or it's incorrect code that won't compile.

Well, maybe that point is not very important for others anyway.

In very-very short form: In Emacs (or any other editor/IDE, actually) M-x find-function foo.Bar can't work without additional prompt if multiple foo.Bar exist. This basically boils down to the fact that you need two things instead of one to find the function: it's name and actual arguments (or their types).

prasannavl commented 6 years ago

@Quasilyte - Ah. Thanks for that. I was thinking of only the goto-definition by pointing at a function use, and navigating to the definition from there. (F12 from the actual function usage, in VSCode for example)

Thinking of the scenario you mentioned, yes, an additional prompt would be needed, when appropriate. Though most tooling assisted editors I tend to use - VS Code, Gogland, etc have as-you-type fuzzy search anyway that boils it down where I wonder if this is even noticeable. (C# has it, and never felt it to hinder ease of use or the speed of navigation. So does C++, Rust (traits), JavaScript etc). But yes, I suppose it might involve an extra key-press, and some people value it a great deal than others - though I really really wish one wouldn't state that as an argument against overloading.

quasilyte commented 6 years ago

@prasannavl, did you answered https://github.com/golang/go/issues/21659#issuecomment-325485157? Please, consider this case. It's a technical detail, not subjective or "religious".

I am not a good advisor here, but maybe technical design document may be a better argument than repeating the ones that proven not to be working (not everyone will agree on "API just get's better"). You may browse existing design documents to get inspiration.

Can be useful: adhoc polymorphism in the context of Haskell Not all kinds of polymorphism play well with each other. If generics are desired, they should be somehow designed together.

quasilyte commented 6 years ago

There is a backwards-compatible way to introduce adhoc polymorphism into Go without some problems mentioned above though.

Introduce a new keyword that defines overloadable function. This function can not be assigned assigned unless type elision is possible (see https://github.com/golang/go/issues/12854). No existing code is affected.

xfunc add1(x int) int { return x + 1 }
xfunc add1(x float32) float32 { return x + 1 }

f := add1 // Error: can't assign overloadable function
var f1 func(int)int = add1 // OK
var f2 func(float32)float32 = add1 // OK

func highOrder(f func(int) int) {}

highOrder(add1) // OK

xfunc is just a placeholder for a better (and new) keyword.

When overloadable function is assigned, it's type is concrete and it can't be distinguished from normal function. The only differences are in the way functions are defined (to tag their names and store function info in separate data structure during compilation) and assigned. No issues with "overloadable functions as parameters", because the value can't have a "overloadable function" type.

Overloadable functions share same namespace as normal functions, hence it is not possible to have f as overloadable and ordinary function.

If considered in the context of original proposal goals, there is a downside. Programmer must know beforehand which function may require overloading in future.

creker commented 6 years ago

@Quasilyte I'm not @prasannavl but the answer is obvious - compile time error. If type inference doesn't work then compiler should generate an error. Go has var [name] [type] syntax to solve that. No need for new keywords or anything like that. Existing code will continue to compile as long as you don't start adding overloads. And when something breaks it will be trivial to fix by hand. But probably not something that can be fixed by tooling.

quasilyte commented 6 years ago

@creker, maybe it's my personal problem with "compatibility" term and "API versioning" solution. If overloading is sold as a solution for that, why it makes it so easy to break code that is depending on the library?

This is why "compile error" is obvious, but suboptimal, in my opinion. Also, some languages do it in other way and defer error until there is no way to infer the actual type. These cases should be a part of proposal to avoid misunderstanding. Obvious things are not exception.

creker commented 6 years ago

@Quasilyte type inference can be as sophisticated as it can but when there's no other way then compiler should throw an error. The comment that you mentioned is ambiguous without proper context and should not compile. How exactly sophisticate it is should probably be in the proposal document. I agree with you on that.

About breaking client's code, that's definitely something to think about and should be covered extensively in the proposal. Even language like C# has problems with that. But it most cases examples of such breaking are more telling about bad library design rather than problem with overloading. MS adds overloads with every .Net release and doesn't break anything because if you design your API properly then adding an overload is not breaking change.

bcmills commented 6 years ago

if you design your API properly then adding an overload is not breaking change.

That depends on how strictly you want to define “breaking change”. If someone, say, assigns a function to a variable, and later invokes that function by name, the two references may resolve to the same overload in one version and a different overloads in the next.

That pretty much implies that if you want a strong compatibility guarantee, you must prohibit users from ever referring to a specific overload, including by assigning it to a variable. The compatibility guidelines for Google's Abseil C++ libraries make that explicit: “You cannot take the address of APIs in Abseil (that would prevent us from adding overloads without breaking you).”

creker commented 6 years ago

@bcmills you could still design your API to solve that. If your overloads are ambiguous then it's your problem to make them compatible. Even if reference resolves to different overload client's code is still working.

Say, you had one function

func foo(interface{})

Now you add an overload

func foo(io.Reader)

Obviously client's code that passed io.Reader before will now reference the latter overload. It's your problem to retain compatibility between the two even if they're used interchangeably. If your overloads are so incompatible that you can't even solve that then it's obviously API design problem.

Another solution could be on the language level. Forbid referencing overloads without explicitly specifying the exact overload. In other words, f := foo will not compile in any case even if you can infer the type. Also, forbid type conversion between overloads even if arguments are compatible. In the case above, two overloads have different and incompatible types even though you can pass io.Reader as interface{}.

prasannavl commented 6 years ago

@prasannavl, did you answered #21659 (comment)? Please, consider this case. It's a technical detail, not subjective or "religious".

It just doesn't compile. If there's ambiguity you add signatures to compile. (But, this does suffer from the same compatibility issue. More on that below)

@Quasilyte type inference can be as sophisticated as it can but when there's no other way then compiler should throw an error.

@creker, I think @Quasilyte was referring to the part where the function that isn't overloaded, is then overloaded, that could prevent code that compiled before from not compiling? While this isn't really an issue at all when statically linking - it is a significant issue during dynamic linking.

Also since you mentioned C#, just want to point out here, while I think it's fair to compare tooling, I don't think it may be fair comparison to compare on a deeper level. Since C# has IL code in the middle, it provides with more flexibility that Go cannot achieve.

maybe technical design document may be a better argument

@Quasilyte - I do agree. I think there's significant feedback that has been gained from this thread, to start thinking about a technical document. I hope to find time soon to collect things from this and potential impl possibilities into a document. 22/30 for/against a the time of this comment. Hmm.

PS: While I'm quite obviously advocating FOR overloading, if it HAS to introduce a new keyword, to me personally that would tip the scales, as that defeats the outer language simplicity enough for me to not purse it.

@bcmills - now, coming to the compatibility issue - thinking out loud here, there is one approach to make sure overloading doesn't break compatibility. But that'll break compatibility with existing Go. It's rather simple, but radical - change the internal repr of every Go function to have it's parameters as a part of the name

Eg:

func Hello(text string)
func Hello1(text string)
func Hello1(name string, text string)

Now the compatibility is an issue only when Hello, and Hello1 follow different methodologies for internal naming. But if you keep it consistent and change the internal representation of both to look like this

func Hello__$CMagic__string()
func Hello1__$CMagic__string()
func Hello1__$CMagic__string_!_string()

This solves compatibility. But introduces slightly more complicated debug tooling. (Not debugging, but debug tooling). The tooling will then have to actively convert the internal repr to human readable form of Hello1(name, text).

This does raise the cost of internal complexity more than I'd like. But does solve compatibility.

ianlancetaylor commented 6 years ago

Any approach here will add significant complexity to the spec. We will need to define something along the lines of the complex C++ rules to choose which overload is desired. It won't be as complex as C++ but it will be complex. One of the key reasons that Go code is easy to read is that it is easy to understand what every name refers to. This proposal will lose that property to some extent.
In code like f := F where F is overloaded you will need to explicitly state the type of f, which is unfortunate.
In code that uses method expressions that refer to an overloaded method, you will have to specify the type of the desired method, but how? This is a long issue but I don't see any good suggestion above for this problem.

Proposal declined.

golang / go

proposal: Go 2: function overloading #21659

Why?

Complexity level is low to implement it in Go

Simpler naming and sensible APIs surfaces

Better default patterns for APIs

Versioning

Performance optimizations