proposal: Go 2: generic native types (GNTs)

[rcoreilly]

This proposal is to add generic native types (GNTs) to Go: generic versions of each of the native builtin types (e.g., map = generic map, slice = generic slice, number = generic number, float = generic float, signed = generic signed integer, unsigned = generic unsigned integer, etc).

Edit: Can also use generic types to specify container types, e.g., []number is a slice with number elements, map[string]number, etc.

GNTs support a substantial proportion of generic programming functionality, without adding any new syntax to the language: generic code looks identical to regular Go code, except for the use of these generic type names, and a few extra functions that access the type parameters of container objects such as maps and slices (e.g., elem(x) is the type of the elment in a slice or map; key(m) is the type of the key of a map; field(s, fieldName) is the type of the given named field in a struct; return(f) is the type of return value from function, and type(x) is the type of a generic var, e.g., for use in return signature).

The generic struct is the most challenging case, because of its unlimited number of type parameters. The proposed strategy is to adopt the interface concept, but specialized for the struct type, as a struct interface -- see https://github.com/golang/go/issues/23796 for discussion of a related proposal, where most of the objections were about "polluting" the existing interface concept with struct-specific functionality -- this problem is avoided by having a special type of interface only for structs, which provides all the benefits from that proposal, as well as supporting the definition of a generic struct type.

Edit: It might be clearer to use prototype instead of struct interface due to the differences between this type and the existing interface -- they do share some properties but also differ significantly.

Here are some examples of some of the main generics use-cases under this proposal:

func Keys(m map) []key(m) {
    s := make([]key(m), len(m))
    idx := 0
    for k := range m {
        s[idx] = k
        idx++
    }
    return s
}

// note: all args in same list (x,y) must be of same concrete type
func Min(x, y number) type(x) {
  if x < y {
     return x
  }
  return y
}

// A struct interface is like an interface, specialized for struct -- any struct
// with the specified field names and types gets the generic methods,
// instantiated for the concrete type
type Vec2 struct interface {
    X, Y number // or float
}

Edit: or prototype version:

// A prototype provides an abstract implementation based on a struct,
// using generically-typed fields, and methods operating on those fields.
// Any concrete struct with the specified field names and types matches the prototype
// (similar to how interfaces are "magically" matched, except based on fields,
// not methods).
// The concrete struct then gets the generic methods from the prototype,
// instantiated for the concrete field types
type Vec2 prototype {
    X, Y number // or float
}

func (v Vec2) Add(u Vec2) Vec2 {
    return Vec2{v.X + u.X, v.Y + u.Y}
}

// Vec2f32 instantiates the generic Vec2 struct interface [prototype] for the float32 type
type Vec2f32 struct {
    X, Y float32
}

func myfunc() {
    v1 := Vec2f32{1,2}
    v2 := Vec2f32{3,4}
    sum := v1.Add(v2) // this is automatically instantiated from prototype
}

package graph

// Edge defines a connection between two nodes, as a struct interface.
// Any type with fields named From and To that are pointers to a 
// concrete struct that implements the Node struct interface will satisfy
// the Edge interface.
type Edge struct interface {
    From, To *Node 
}

// Node defines a node in a graph, which has Edges connecting it to other
// nodes.  Any concrete struct with a slice of structs satisfying the 
// Edge struct interface, named Edges, will satisfy this interface.
type Node struct interface {
    Edges []Edge // concrete slice of a struct interface type..
}

// Unlike the draft proposal, methods can be defined with struct interface args
// just like any interface type can be passed around currently.  A boxing penalty
// is likely but perhaps not inevitable.
func (n *Node) ShortestPathTo(to *Node) []Edge {
    for _, e := range n.Edges {
        if e.To == to {
            return []Edge{e}
        }
        se := e.To.ShortestPathTo(to) // all just std interface virtualization method calls..
        if len(se) > 0 {
            return se // actually would need to sort by path length and return shortest, but whatever..
        }
    }
    return nil
}

In summary, GNTs leverage the "contracts" associated with the existing native types in the language, instead of introducing an entirely new syntax to specify novel such contracts. This makes programming with these types immediately intuitive and natural for any Go programmer, and the code as shown in the above examples inherits the simplicity and elegance of the existing Go syntax, as compared to other generics proposals that require additional type parameters etc.

Edit: The key difference between GNTs and the draft proposal (and other similar approaches using separate type parameters) is that these other approaches split the type information into two separate parts, while GNTs keep the type specification unitary and in one place (where it normally is for non-generic code). For example, in the draft proposal:

func Print2(type T1, T2)(s1 []T1, s2 []T2) { ... }

You have to go back and forth between the type args and the regular args to understand what the types of s1 and s2 are -- the information is split across these two places. In contrast, under the GNT proposal:

func Print2(s1 slice, s2 slice) { ... }

the type is fully specified (albeit still generic) in one place.

To use a colorful metaphor, the draft design is like a horcrux that splits key information painfully across multiple places, whereas GNT's preserve the "natural" unity of type specification: you don't have to go searching across multiple hiding locations to find the type :) More scientifically, reducing cognitive load by not having to integrate across these separate pieces of information is a well-established principle: https://en.wikipedia.org/wiki/Split_attention_effect

The major limitation is that generic functionality is limited to these existing native types. You cannot define Min / Max functions that work across all native numbers and on non-native numbers such as big.Int. The bet here is that GNTs solve 80% or more of the use-cases in the cleanest, simplest way -- i.e., the defining feature of Go.

Several of the required keywords are repurposed from existing ones, and thus would have no compatibility impact:

• Type kinds: map, chan, interface, struct, struct interface (combination)
• Type parameters: type(x), return(f) (with optional 2nd arg specifying which return val if multiple -- could be index or name if named)

These new ones would create potential conflicts:

• Type kinds: slice, array, number, unsigned, signed, float, complex, prototype
• Type parameters: elem(x), key(x), field(x, fieldName)

See GNTs gist for a full discussion of the proposal -- would be happy to write a design doc according to standard template if requested.

[av86743]

Also compare with https://github.com/golang/go/issues/32862 just prior to this one.

[dotaheor]

It is some like this one without the outer gen part.

[ianlancetaylor]

One of the goals of generics is to permit people to write type safe containers that are not already provided in the language. A typical example would be a concurrent hash map, such as a type safe version of sync.Map. I don't see how this proposal permits that.

Am I correct in thinking that this is much the same as https://gist.github.com/rcoreilly/bfbee2add03c76ada82810423d81e53d ?

[rcoreilly]

@ianlancetaylor Yep I'm submitting an updated, more fully worked-through version of that gist idea here in order to get a complete, considered evaluation of this idea from the entire team, per the proposal process description. Here is my response to that question from the go-nuts email list (you didn't reply to this response so I don't know if it is satisfactory or not?)

you can do any valid map operation on the generic type “map”, so at a minimum it seems like you could just add an appropriate mutex around each of the different operations, using a struct interface:

type SafeMap struct interface {
    mu sync.RWMutex
    m  map
}

func (m *SafeMap) Store(k key(m.m), v elem(m.m)) {
    m.mu.Lock()
    if m.m == nil {
        m.m = make(map[key(m.m)]elem(m.m))
    }
    m.m[k] = v
    m.mu.Unlock()
}
…

any concrete map that shares that struct interface type signature (i.e., the same field names and types) automatically gets the associated methods:

// MySafeMap implements the SafeMap struct interface
type MySafeMap struct {
    mu sync.RWMutex  
    m  map[string]string
}

func MyFunc() {
    m := MySafeMap{}
    m.Store(“mykey”, “myval”) // at this point the compiler knows the concrete type of “m”, can compile Store
}

Robert Engles replied:

That is not sufficient. You can't require the usage of the built-in map, as specialized maps may have a completely different structure (special hash tables when the keys are ints, or more commonly special CAS techniques for highly concurrent lock-free maps).

To which I replied:

That is a good point of clarification about the specific limitation of this proposal: it specifically reifies the existing Go type system and does NOT enable the generic-ification of arbitrary novel types. If you can build a specialized type out of generic versions of existing Go types, then you’re good, but if you need something entirely different, then this won’t do it. The key question is whether having a really simple and tractable “native types” version of generics solves enough problems, while avoiding the challenges of going “fully generic”, to hit that “Go” minimalism sweet spot..

[ianlancetaylor]

Thanks, I did see that discussion, and didn't reply further because I thought the point had been made. Sometimes you want a type-safe container that is entirely different. I think that is part of the 80%, not the 20%.

[typeless]

I like this proposal. It adds minimal syntax complexities and zero emission of noises for generic code. It does not change much how Go1 code looks.

[rcoreilly]

@ianlancetaylor Have you considered that, as long as said container is composed out of existing container elements, GNTs would handle that case? How many containers would not have a map or slice as the underlying storage?

Also, it would be possible to adopt this proposal on its own minimalist merits, and postpone a more "generic" generic solution, after real-world experience with how far this can go. It seems likely that anything that is "fully generic" is going to incur a significant syntactic penalty in terms of type specifiers and contracts, so perhaps it would be better to put that off until we can see to what extent it is really that valuable in real-world cases?

And @typeless thanks for the support!

[ianlancetaylor]

The maps provided by the language don't permit you to specify a hash or equality function, so they are limited in how they handle keys.

You can't implement a proper concurrent hashmap on the basis of maps or slices; you need a more complex data structure.

I find these to be serious difficulties with this approach. Of course others may not agree.

[rcoreilly]

Not to belabor the point but if these features were deemed not important enough to include in the language itself then perhaps that indicates their status relative to the 80 / 20 tradeoff?

Could we use some Go corpus stats to decide these kinds of questions - how frequently are those features actually used?

[av86743]

In your opinion, professor O'Reilly, what is the degree of cohesiveness between your proposition and original proposition of Go inner circle? On a scale from 10 to 0, 10 being a sound and most welcome fit, and 0 being a completely irrelevant nonsense.

[mikeschinkel]

@ianlancetaylor

The maps provided by the language don't permit you to specify a hash or equality function, so they are limited in how they handle keys. You can't implement a proper concurrent hashmap on the basis of maps or slices; you need a more complex data structure.

Is it not possible to consider a solution for the subset of the problem? Seems that is the approach the try() proposal chose?

[ianlancetaylor]

@av86743 Please be polite. Please do not suggest that other people's ideas might be "completely irrelevant nonsense." Please see the Gopher Code of Conduct at https://golang.org/conduct. Thanks.

[ianlancetaylor]

@rcoreilly The language has a limited set of generic types. The generics problem is that Go provides no way for people to write generic code other than using that limited set. Any 80/20 tradeoff here, if there is one, doesn't have anything to do with what Go already provides. It has to do with a tradeoff among concepts that Go does not provide. I am suggesting that the ability to write compile-time-type-safe containers is part of the 80% that needs to be provided by any solution to the generics problem. The fact that Go does not provide such containers doesn't tell us anything about whether compile-time-type-safe containers are in the 80% or 20% of what Go does not provide.

Could we use some Go corpus stats to decide these kinds of questions - how frequently are those features actually used?

I don't think I understand this question. Since Go does not provide these features, they are never used.

[ianlancetaylor]

@mikeschinkel We can consider steps that solve a subset of the problem, as long as there is a clear path forward to addressing more of the problem as we gain experience. The try proposal, for example, is careful to explain that we can add additional arguments to the try builtin function to add additional functionality, such as wrapping errors, if that seems desirable.

[av86743]

@ianlancetaylor

@av86743 Please be polite. Please do not suggest that other people's ideas might be "completely irrelevant nonsense." Please see the Gopher Code of Conduct at https://golang.org/conduct. Thanks.

I do not suggest either (other being "sound and most welcome fit".)

On the other hand, what other individuals are doing, is out of my control. If certain proposition grades at level 0, it is what it is. I cannot change that.

[ianlancetaylor]

@av86743 You can't change what something is but you can change how you talk about it. Please read https://golang.org/conduct, especially the section on "Avoid destructive behavior". Thanks.

[av86743]

@av86743 You can't change what something is but you can change how you talk about it. Please read https://golang.org/conduct, especially the section on "Avoid destructive behavior". Thanks.

And which of the below, in your view, applies?

Derailing: stay on topic; if you want to talk about something else, start a new conversation.

Unconstructive criticism: don't merely decry the current state of affairs; offer—or at least solicit—suggestions as to how things may be improved.

Snarking (pithy, unproductive, sniping comments)

Discussing potentially offensive or sensitive issues; this all too often leads to unnecessary conflict.

Microaggressions: brief and commonplace verbal, behavioral and environmental indignities that communicate hostile, derogatory or negative slights and insults to a person or group.

[mikeschinkel]

@ianlancetaylor Not in reply to your reply. but as an additional comment.

However, since this is off-topic of this proposal I am hiding it in a details section.

Click to read

I think everyone in the Go team understands that generics are a very hard problem to solve without recreating the complexity of generics in Java et.al. And I for one very much do not want Go to become so complex that its code is hard to reason about in order to add generics. One thing Go has done that is relatively unique in the realm of serious programming languages is provide builtin functions with features that a Go programmer cannot develop on their own, e.g. `append()`, `make()`, etc. Earlier in my career I would have thought that to be a negative but now I think one of the strengths of the language is that it can offer special case solutions so that its language does not have to become too complex. As such, it is my hope that the Go team **will address most of the need for generics with builtin functions** to handle the 80+% of use-cases. Here is my list of proposed new builtins: - A func that returns a slice of **property** values given a map or slice of structs, e.g. `props(myMap,Id)` and `props(mySlice,Id)` where `Id` is a property of the struct that `myMap` and `mySlice` contain. - A func that **converts** a map of structs to a slice, or a slice of structs to a map; e.g. `convert(myMap)` and/or `convert(mySlice,myKeyExpression)` where `myKeyExpression` is either a property name, or an expression that assumes property names that are not qualified, e.g. `fmt.Sprintf("%s:%s", id, FullName())` where both `id` is a property and `FullName()` is a method of the struct that `myStruct` is a type of. - A func that **filters** a map or a slice, e.g. `filter(mySlice,myFilterExpression)` where `myFilterExpression` is equivalent to `myKeyExpression` above but that returns `bool` but if `false` eliminates the element from the returned slice. - A func that **iterates** a map or a slice, e.g. `mySlice, err = iterate(mySlice,myIterateExpression)` where `myIterateExpression` could be `Validate()` and returns an `error` or `nil` and if `!= nil` short-circuits the internal loop and return with the `error` as 2nd return value. - A func that **unwraps** an `interface{}` containing a map or slice to a map or slice containing the same type contained in the `interface{}`s, *or* a map or slice of `interface{}`s containing identical types to a map or slice of that type, e.g. `unwrap(myInterfaceOfMap)`, `unwrap(myInterfaceOfSlice)`, `unwrap(mySliceOfInterface)` and/or `unwrap(myMapOfInterface)`. - A func that **wraps** an a map or slice into a map of `interface{}` values containing the types, e.g. `wrap(myMap)` and/or `wrap(mySlice)`. I am assuming that the compiler could see all the above and be able to optimize it hence why there would be no need to qualify properties or methods. OTOH, for the `unwrap()` method that works with interfaces the Go compiler might not know what types it will contain so I would propose having the compiler recognize an `Unwrap()` methods as a special interface for any type that needs to be unwrapped thus allowing the unwrapping code to be precompiled. If an `interface{}` was attempted to be unwrapped and its contained types did not implement the `Unwrapper` interface then it should throw a `panic()`. Of course the names I chose are open for bikeshedding, but the functionality is really what I wanted to present. There maybe be one of two more that I have forgotten, but this is all I've got at the moment. The above would probably eliminate at least 80-90% of my needs for generics and it would probably reduce the number of lines of code in my codebases by 1/3 to 1/2 current size.

[ianlancetaylor]

@av86743 Happy to continue this on private e-mail at iant@golang.org. I'm now going to hide this thread from the issue. Thanks.

[rcoreilly]

@ianlancetaylor Haven't people written non-type-safe interface{} based versions of containers that they really need? That is what I'm suggesting could be analyzed to determine the prevalence of demand for containers outside the standard. Also, it would be important to see how many of those weren't backed by a standard slice or map in a way that would be amenable to this approach. Or easily handled using a struct interface as in the Graph example, which also handles linked lists and all manner of tree-like structures. I've got very limited internet here in Yellowstone now so can't easily look up examples but I'm sure there are lots of them that I've seen..

Also, in terms of a pathway to full generic types, it occurs to me that this overall approach could be extended as in Michal Straba's generics proposal to use the keyword type for a fully generic item (while using the type(x) function to access the actual type, instead of defining an extra type parameter, to eliminate need for additional syntax) -- this would then forgo the benefits of having a known "contract", but would be available for "extreme" cases. But I'm struggling to see how you would compute a hash function on a fully generic type? Probably you'd want to make specialized versions for number, string keys? Seems like it would still be a lot better to not include this at the start, to see if it is really worth "going there".

[ianlancetaylor]

I see, sure, that is worth asking. It's not a great comparison since interfaces generally require boxing values, so there is a real cost to building containers based on interfaces. So people avoid doing that. But it's worth asking. For example, can this proposal be used to build a compile-time-type-safe version of sync.Map?

[rcoreilly]

Yep definitely can do a compile-time-type-safe version of sync.Map -- when the struct interface is instantiated by a concrete struct it should have everything it needs to compile all the methods.

[ianlancetaylor]

I don't understand how that works. And looking back at your SafeMap example, I don't understand how that works either. I can sort of see how struct interface defines an generic interface that other types can implement, and I can sort of see how MySafeMap implements SafeMap. But that's not what I want. I want SafeMap to be the complete implementation, written in terms of a generic type parameter. I don't want to have to write MySafeMap, which has specific key and element types. See the "containers" example in https://go.googlesource.com/proposal/+/master/design/go2draft-contracts.md .

[rcoreilly]

It is just a different way of achieving the same end result. Either you add new syntax to specify type parameters (following the paradigm used in C++ and other languages), or you name a concrete instantiation of a struct interface type (which seems like a more Go way to do it, following the interface paradigm). In both cases, you are specifying the same information.

In other words, the move here is to adopt the idea that an abstract interface is much like a generic type specification, and concrete types that implement the interface are what you actually want to use in any given case. By using the struct form of the interface, we can specify concrete types for the generically-typed fields, but otherwise it should work much like an existing interface.

A drawback is that you need to come up with relevant names for the concrete instantiations (probably something better than MySafeMap) -- that type name effectively replaces the generic type and the type parameters in the standard approach.

But the advantage is that you can further customize the concrete type, just like something that implements an interface can also do various other things -- i.e., the polymorphism / mix-and-match model instead of just a "type-parameterized instantiation of a canonical generic type". Also, this approach can naturally support multiple different generic fields, which would start to get awkward for the type parameter approach. Furthermore, it is very clear exactly when and where the generic type is being instantiated with concrete type args -- this could simplify a lot of the issues with duplicate code generation etc.

For many cases involving generic number or float types, e.g., the Vec2 example, it would be natural for the package to provide default named instantiations (Vec2f32, Vec2f64) so it functions much like existing code generation techniques.

So anyway, yes this is definitely not the standard type-parameterized generics syntax, and this is also why I think it is likely to be the most "Go-like" of any of the existing proposals, most of which follow in that general type-parameter mold..

[lootch]

On 7/4/19, rcoreilly notifications@github.com wrote:

It is just a different way of achieving the same end result. Either you add new syntax to specify type parameters (following the paradigm used in C++ and other languages), or you name a concrete instantiation of a struct interface type (which seems like a more Go way to do it, following the interface paradigm). In both cases, you are specifying the same information.

Any chance you could explain by example?

Lucio.

[ianlancetaylor]

When I say that I believe that any generics implementation must be able to support compile-time-type-safe containers, I mean that people must be able to write the code for a container once, and be able to instantiate that code many times for many different types. I think the ability to do that is a requirement for any generics proposal in Go.

[rcoreilly]

@ianlancetaylor I'm not sure I follow what you're saying. Per the SafeMap example above (@lootch do you want more than the examples here and at the start -- if so, what?), the code is definitely only ever written once, and instantiated any number of times for different types. Unlike a standard interface, the struct interface has fields (with generic types) so you can write the full code implementing the methods of the interface. Furthermore, the matching rule for a struct interface is only based on the fields, not the methods -- if you define a type that matches the fields, then the methods come along for free and are instantiated for the concrete types of the fields. Here is that example with more annotation:

// any concrete struct type matching these field names and types implements the SafeMap
// struct interface, and automatically instantiates the methods defined on SafeMap for
// the specific type of map used for field m
type SafeMap struct interface {
    mu sync.RWMutex
    m  map
}

// this is the only place we ever need to write the Store method!  GNTs allow us
// to write the entire implementation, which is then instantiated with concrete types
// by a specific struct type that implements this interface. 
func (m *SafeMap) Store(k key(m.m), v elem(m.m)) {
    m.mu.Lock()
    if m.m == nil {
        m.m = make(map[key(m.m)]elem(m.m))
    }
    m.m[k] = v
    m.mu.Unlock()
}

// define Load, Delete etc here, in the same way...

Then the user does something like this:

// MySafeMap implements the SafeMap struct interface, because it has the same field
// names and types as SafeMap.  At this point, the compiler now has a concrete type for
// the generic map defined in SafeMap, and can compile all the methods defined on SafeMap
// for that specific type of map.  Furthermore, this is the exact point at which this code must
// be instantiated, removing any ambiguity otherwise present in generics with type params
// as to when to instantiate a given parameterized version of a generic type.
// The user can also add their own fields and methods to extend the functionality in any 
// way they want, just like a type that implements an interface can also do lots of other stuff.
type MySafeMap struct {
    mu sync.RWMutex  
    m  map[string]string
    metaData map[string]string // User can add their own additional fields and methods too!!
}

func MyFunc() {
    m := MySafeMap{}
    m.Store(“mykey”, “myval”) // we automatically got this method from SafeMap, did not re-write
}

[av86743]

// [...]
type SafeMap struct interface {
    mu sync.RWMutex
    m  map
}

// [...]
func (m *SafeMap) Store(k key(m), v elem(m)) {
    m.Mu.Lock()
    if m.Map == nil {
        m.Map = make(m.Map)
    }
    m.Map[k] = v
    m.Mu.Unlock()
}

@ianlancetaylor

Professor O'Reilly must be joking. There is no Mu in SafeMap. No Map in SafeMap either.

[rcoreilly]

Sorry - copy-paste err from a diff case - Map -> m and Mu -> mu.

[typeless]

I think what @ianlancetaylor means is that we cannot implement a type-safe generic container with arbitrary underlying implementation (with this proposal). Say, the built-in map implementation https://github.com/golang/go/blob/master/src/runtime/map.go#L115-L129 uses unsafe pointers and arrays as the underlying data structures. This proposal seems to be impossible to make such code type-safe so that users can implement it as a package.

[rcoreilly]

@typeless I'm not sure why you think it would not be type safe? The MySafeMap defines the map as specifically map[string]string so it is just as strongly typed as any native map of the same type. LIkewise, the Store method is defined in terms of the actual key and element types of the map, so it will only take args of those types.

func (m *SafeMap) Store(k key(m), v elem(m)) {

Am I missing something?

[typeless]

@rcoreilly what if I want to implement a map based on an underlying slice or a B-tree? I am not sure if it's possible. Because I thought, the struct interface requires the member to be a map?

[ianlancetaylor]

Thanks for the example. I didn't realize that a struct interface could also have methods.

func (m *SafeMap) Store(k key(m), v elem(m)) {

I don't understand how you can write key(m) here. `m' is not a map.

type MySafeMap struct {
    mu sync.RWMutex  
    m  map[string]string
    metaData map[string]string // User can add their own additional fields and methods too!!
}

func MyFunc() {
    m := MySafeMap{}
    m.Store(“mykey”, “myval”) // we automatically got this method from SafeMap, did not re-write
}

I don't understand how you can call m.Store here. There is nothing that goes from MySafeMap to SafeMap. Somehow you are taking the methods of SafeMap and adding them to MySafeMap. I don't think that could happen automatically. SafeMap is likely in a different package. I don't think we can just pick up all methods of struct interface types and automatically add them to types that implement that struct interface. That's not how interfaces work.

[av86743]

func (m *SafeMap) Store(k key(m), v elem(m)) {
[...]
        m.m = make(map[key(m.m)]elem(m.m))
[...]

I don't understand how you can write key(m) here. `m' is not a map.

Use of derived types key(...) and elem(...) was earlier corrected in the body, but not in the header.

[rcoreilly]

@av86743 thanks for noting that the correct signature should be Store(k key(m.m), v elem(m.m)) -- I went back and fixed that in the two examples above.

@ianlancetaylor the logic is the same as with existing interface, except a struct interface matches on field names and types instead of matching on methods. The same "magic" association between types occurs, based on this match, and presumably all the same overall issues associated with this kind of "magic" logic should be well understood by Go users.

Once the two types are linked, then it should be straightforward to compile all the methods defined on the struct interface using the concrete types provided in the matching struct. Yes this is not how interface works, but it is clearly how struct interface must work to provide generic functionality, and is hopefully fairly intuitive once understood?

For example, in the SafeMap case, it should be clear that someone defining a type with a mutex and a map, and importing the new sync.Map package that defines the struct interface, intends to create a concrete version of that interface, and comments can be added to clarify this as people often do for interface methods.

[rcoreilly]

@typeless Each struct interface defines a specific generic type with full implementation in its methods. So the SafeMap example defines a basic mutex-protected interface to a standard map, and that is all it does. If you want something else, it would have to be coded as such. For example, if you have a container that uses a slice as a backing element, you would include something like this:

type FancyContainer struct interface {
    s  slice // this is the generic slice that provides the underlying storage for fancy container
    ...  // some other fields needed to make fancy container work
}

func (f *FancyContainer) Store(v elem(f.s)) {
    // code that stores into slice, involving for example:
    f.s[idx] = v
    // where idx has been computed (and space allocated) according to some fancy algo
}

Then the user does the same kind of thing as in SafeMap:

type FancyF64 struct {
    s []float64 // concrete type of slice for this instantiation
    ... // other fields copied from struct interface
}

func MyFunc() {
    f := FancyF64{}
    f.Store(3.14) // methods have been compiled from struct interface...
}

Make sense?

[ianlancetaylor]

Interface work in that if a type has a certain set of methods, a value of that can be implicitly converted to the interface type.

I think you are suggesting very very different: if a type happens to have a certain structure, then a set of methods is automatically added to the type.

With interface, it does little harm that there is no explicit relationship between the interface I and the type with methods T. It would be unusual for someone who does not intend T to implement I to attempt to convert a value of type T to type I.

With your suggested struct interface, changing the imports of a package can change the set of methods on a type without any other change. That seems odd.

What happens if package A defines the struct interface, package B defines the type that happens to implement the struct interface without importing package A, and package C imports both packages A and B?

[typeless]

@rcoreilly That's cool. It reminds me of the GCC typeof and offsetof extensions, which are indeed useful to implement some generic code.

The elem and key, like typeof in C, are type functions which take types as input and produce a type. It is interesting to see that if there are other useful type functions.

Another question is why s in the struct interface has to be lowercase (private)? Would that be a problem if the method Store is public, but we have f.s in its method signature?

[rcoreilly]

@ianlancetaylor I would suggest that a file needs to directly import the package that defines the struct interface for it to take effect (or the struct interface is in the same package itself). Also, as a convention, generic-providing packages should be focused solely on defining a single targeted struct interface so you are explicitly making the decision to use that generic type by importing that package.

With this structure, there shouldn't be any unintentional consequences of the struct interface.

Keep in mind that this is all buying a way of specifying any number of concrete field types without requiring explicit type parameters decorating everything, so in my judgment it is well worth the resulting simplicity and readability of the code to have this one bit of "magic". It seems like a similar judgment that was made in not making the connection between an interface and a concrete type explicit in Go1.

[rcoreilly]

@typeless I'm not familiar with those GCC extensions so I'll have to look those up -- good prior art it seems! Regarding the unexported (lowercase) fields, there would have to be a rule that says that the unexported fields of a struct interface are exported to the struct that instantiates it (across package boundaries). Furthermore, the use of unexported fields (e.g., f.s in example code) in any methods in either the struct interface or the instantiating struct would be legal because these are all local to the current package. Anyone importing a package with the instantiating struct would not be able to refer to the unexported fields, as expected with the current rules.

Then it is just a question of design preference whether to use unexported or exported field names, as it is currently.

[rcoreilly]

Looks like typeof became decltype in C++11 -- relevant arguments about its utility: https://en.wikipedia.org/wiki/Decltype

[rcoreilly]

Perhaps it would be clearer to use a different keyword such as prototype instead of struct interface, given the differences between how the proposed struct interface works compared to the existing interface. An interface is an abstracted API defined strictly in terms of methods, and the struct interface is not really that -- you don't match or redefine its methods.

prototype captures more of the core idea of something that is an abstracted implementation, where the fields need to be concretized to make fully implemented type.

It is also cleaner to have a single keyword instead of a sequence of two.

[rcoreilly]

Just saw the updated contracts proposal: https://go.googlesource.com/proposal/+/master/design/go2draft-contracts.md -- given that contracts can now only specify existing types, wouldn't it be simpler to just introduce sensible generic type categories in the first place, and do away with the need for the contract syntax and the type parameters! :) How many times will people specify random combinations of number types? Or generic functions that operate on string and number types? This seems like a lot of extra parentheses and syntax, for potentially rarely needed extra degrees of freedom..

[alanfo]

given that contracts can now only specify existing types, wouldn't it be simpler to just introduce sensible generic type categories in the first place, and do away with the need for the contract syntax and the type parameters! :)

Well, last September shortly after the first draft design for generics had been published, I wrote an alternative proposal which did use what I thought at the time were sensible 'type groups'. However, it later became clear that, with the integer groups, there were problems with the smaller types not being able to deal with untyped literals outside their range and, in later proposals, I came up with the idea of being able to exclude such types from those which would otherwise have satisfied the contract.

When I look at the latest draft design, I think listing the types which can satisfy the contract is much simpler and more flexible (even if a trifle long-winded) than having predefined type groups though a standard contracts package can, of course, include the contracts which are most likely to be used in practice.

[ianlancetaylor]

This seems like a lot of extra parentheses and syntax, for potentially rarely needed extra degrees of freedom..

Note that the only extra syntax for contracts is the ability to write an identifier after the list of type parameters. The extra parentheses are still there to list the type parameters.

In exchange we get the ability to, for example, write a function that operators on either a []byte or string, which I believe is not available in this proposal. We also have the possibility of writing generic functions that operate on generic containers, though I admit that I don't know whether that will really work out in practice.

[rcoreilly]

@alanfo I remember some of that discussion which certainly inspired this current proposal. Within the context of the draft design, the ability to list types explicitly and also use predefined contracts that delineate the standard kinds of categories seems reasonable and is certainly more flexible than only having predefined categories.

However, in the context of the present proposal, there is a much bigger potential benefit available by using generic type names (signed, unsigned, integer, number, etc), which is retaining the current Go1 syntax entirely, and avoiding extra type parameters. Using lists of types and / or having the ability to define arbitrary new categories would probably be an excessive complexification here. I'm not sure the language itself should be expected to prevent numerical overflow if you use too small of a type for the job..

[rcoreilly]

@ianlancetaylor you should be able to write a lot of common functions that operate on slices of any type using the generic slice type: e.g., Reverse, Insert, Delete, etc. string could presumably be treated as a special read-only []byte slice in this case, and the compiler could catch any attempt to use non-read-only slice operations on generic code instantiated with the string type.

It should also be possible to extend the proposal to include generic types of the form []number or []integer, etc, which would allow more control over the element type. In which case, you could write generic code with explicit type conversions of the elements as needed to do just about anything, in methods that would accept any such type, including []byte and string (subject to the read-only constraints above).

I haven't looked into the guts of bytes vs. strings packages to evaluate this more concretely, but probably many of those methods could be written generically under this proposal. Presumably there are still a number of cases that must be specialized for string to deal with its read-only nature (and availability of the + operator), and that would be the case for any generics proposal.

[alanfo]

@rcoreilly

I'm not sure the language itself should be expected to prevent numerical overflow if you use too small of a type for the job.

The problem with using the smaller integer types was not so much that they might overflow but that (to take an example from the current design draft) they wouldn't compile at all:

func Add1024(type T integer)(s []T) {
    for i, v := range s {
        s[i] = v + 1024 // INVALID: 1024 not permitted by int8/uint8
    }
}

Nonetheless, unanticipated overflow is a problem with something like a generic sum function and, unless some new integer type(s) are to be introduced which panic on overflow, it might be better for such a function to always return (u)int64 rather than the element type of the slice which is being accumulated.

[rcoreilly]

Is it a problem if that code would not compile when you try to instantiate it with [u]int8? Seems like that would be a good thing -- you'd know at compile time that it doesn't make sense to instantiate that generic function with that type (compiling only happens when you instantiate with a specific type). Yep it does seem like a Sum function should return a (u)int64 to avoid overflow.

[ianlancetaylor]

One important question is: what does that compilation failure look like?

One of the major problems with C++ templates is the long nested compilation errors that result from using a template incorrectly. That is not something we want in Go.