golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
121.07k stars 17.36k forks source link

proposal: Go 2: immutable type qualifier #27975

Open romshark opened 5 years ago

romshark commented 5 years ago

This issue describes a language feature proposal to Immutable Types. It targets the current Go 1.x (> 1.11) language specification and doesn't violate the Go 1 compatibility promise. It also describes an even better approach to immutability for a hypothetical, backward-incompatible Go 2 language specification.

The linked Design Document describes the entire proposal in full detail, including the current problems, the benefits, the proposed changes, code examples and the FAQ.

Updates

Introduction

Immutability is a technique used to prevent mutable shared state, which is a very common source of bugs, especially in concurrent environments, and can be achieved through the concept of immutable types.

Bugs caused by mutable shared state are not only hard to find and fix, but they're also hard to even identify. Such kind of problems can be avoided by systematically limiting the mutability of certain objects in the code. But a Go 1.x developer's current approach to immutability is manual copying, which lowers runtime performance, code readability, and safety. Copying-based immutability makes code verbose, imprecise and ambiguous because the intentions of the code author are never clear. Documentation can be rather misleading and doesn't solve the problems either.

Immutable Types in Go 1.x

Immutable types can help achieve this goal more elegantly improving the safety, readability, and expressiveness of the code. They're based on 5 fundamental rules:

These rules can be enforced by making the compiler scan all objects of immutable types for illegal modification attempts, such as assignments and calls to mutating methods and fail the compilation. The compiler would also need to check, whether types correctly implement immutable interface methods.

To prevent breaking Go 1.x compatibility this document describes a backward-compatible approach to adding support for immutable types by overloading the const keyword (see here for more details) to act as an immutable type qualifier.

Immutable types can be used for:

Immutable Types in Go 2.x

Ideally, a safe programming language should enforce immutability by default where all types are immutable unless they're explicitly qualified as mutable because forgetting to make an object immutable is easier, than accidentally making it mutable. But this concept would require significant, backward-incompatible language changes breaking existing Go 1.x code. Thus such an approach to immutability would only be possible in a new backward-incompatible Go 2.x language specification.

Related Proposals

This proposal is somewhat related to:

Detailed comparisons to other proposals are described in the design document, section 5..


Please feel free to file issues and pull requests, become a stargazer, contact me directly at roman.scharkov@gmail.com and join the conversation on Slack Gophers (@romshark), the international and the russian Telegram groups, as well as the original golangbridge, reddit and hackernews posts! Thank you!

griesemer commented 3 years ago

Core team members have commented on this issue (e.g., see 1st comment by @ianlancetaylor). Virtually all language proposals are on hold while we're trying to get generics done.

ColourGrey commented 3 years ago

@griesemer: Does the core team have a threshold, some measure of the interest of the community for an issue, in order to implement that issue? For instance, if a petition addressed to the White House is signed by at least 100000 citizens, the White House must provide a formal answer. Does the Go community have anything similar?

I took a quick look at the issues labeled with "Proposal-Accepted" and "Go2", and at those labeled with "Proposal-Accepted" and "LanguageChange", and this proposal has ratio of "supportive reactions" (expressed as emojis: thumb up, confetti, heart) to "dismissive reactions" comparable to all those proposals that got accepted (even better, in fact). This shows not only interest in the issue, but also that the community seems to back this proposal. This leads to an ethical dilemma: what if the core team disagrees with the majority of the GitHub Go community? Are these informal votes expressed as emojis binding for the core team in other ways than merely morally (which is quite loose)?

DeedleFake commented 3 years ago

This leads to an ethical dilemma: what if the core team disagrees with the majority of the GitHub Go community? Are these informal votes expressed as emojis binding for the core team in other ways than merely morally (which is quite loose)?

I don't think there's much of a dilemma. It's an open-source project. If the users of the project by and large want something but the primary developers disagree, I would think that the best course of action would be for the people who do want the change to fork it and implement it. It seems a little strange to me to have them be 'bound' by the limited number of users of the language who actually pay attention to GitHub issues and proposals.

The White House, or really the general government, is only required to respond, not actually do anything else, based on a petition. The Go developers generally respond to pretty much every proposal, and they'll usually respond quite a lot to popular ones, especially if they heavily disagree with them. Though they won't be responding much at the moment as the whole team is on a social media vacation for the next few weeks.

ianlancetaylor commented 3 years ago

@concastinator A number of people from the Go team have responded on this issue. If you just want a response, I think you have one. Indeed, you have several.

As far as "level of interest leading to implementation," then, no, there is no level of interest that in itself leads to implementation. There also has to be an idea that works well with the rest of the language. All language changes have costs as well as benefits. As I've explained in several comments above I think that this particular proposal has some significant issues that make it a poor fit for Go.

I personally would like to see a language or analyzer change that addresses at least some of the issues involving immutability in Go, but I have not yet seen a change that seems to have benefits that outweigh the costs. That is just my own personal judgement, of course.

edvdavid commented 2 years ago

Hi, I found this proposal just now, I’m using Go for only half a year now. Previously I used the D programming language for more than ten years and C even longer. Version 2 of the D programming language introduced a const/immutable feature very similar to the one suggested here, I have worked with it for years, so I’d like to speak a bit from experience.

Don’t do it.

Just don’t. You are opening a can of worms you’ll never be able to close again.

A const or immut qualifier looks unproblematic in short code examples, just a few simple extra rules, and it will work just fine. But this extension of the type system has much more severe consequences than these examples show. I’d like to provide a list of what I’ve experienced.

Here are a few examples of complications introduced by qualifiers.

// library code
func f(p *const int) {/* … */}

// user code
func g(f func(*int)) {/* … */}

g(f) // Will this compile? It should!

f can be called with an *int parameter so it should be possible to pass it to g. But are the function types compatible? If yes, what if f accepts another function accepting an unqualified pointer, or if f accepts a pointer to or channel or slice of function pointer?

func f(p *int) {/* … */}
func g(p *const int) {/* … */}

var a = [...]func(*int){f, g} // Will this compile? It should!

What about this case? a[i](p) may or may not write to *p so it should be allowed to have both f and g in the array. Still the function types differ. And again, the argument could be chan []*int vs. chan []*const int as well.

In D2 the memchr problem is addressed as follows:

func f(a *inout int) *inout int {/* … */}

inout is a magic qualifier that works like const inside f, but at the caller’s site the return type qualifier is the same as the qualifier of the inout argument. But what to you do with multiple arguments and multiple return values (which don’t exist in D)? How do you say, I want the first return type to have the qualifier of the second argument and vice versa?

Also, I’ve encountered situations which inout does not solve, like this:

type S struct{
    p *const int
}

(s *S) Set(p *const int) {
   s.p = p
}

(s *const S) Get() *const int {
   return s.p
}

Suppose we pass an *int to S.Set() and get it back with S.Get(). It will be const, no matter what it was initially. But what if we need it to be the same qualifier? We can’t because the const-ness of the return value of Get would need to change at run-time to match the most recent Set call. If a D user encounters this situation then the consequence in practice is to write unsafe code and cast the const away because duplicating S and all its methods or re-designing this part of the project is not feasible.

How will this work with a type assertion or switch? Will an *int match x.(*const int), too? Probably yes, but what about composed types?

Finally, the burden imposed on the programmer it cannot be underestimated how much unnecessary bureaucracy type qualifiers bear because of the incompatibilities they cause if not used with much care and far-sighted planning. That is not to mention the potential for bikeshedding in code review.

Last words. I have used C and D earlier, each for many years. Changing to Go was an indescribable joy from the beginning because so many things causing subtle bugs and unnecessary interference with the actual task of implementing functionality have finally been done right. Simple things are reliably simple and safe. The absence of type qualifiers is a big part of it.

So don’t do the same mistake. Leave type qualifiers out.

[Edit] Bonus: interfaces as an impression of the additional burden of decisions const puts upon language and library designers and users.

Only const methods can be called with a const object so const is caller-friendlier. However, this restricts the method implementation: an additional design decision to make.

Should error.Error() be const or not? Shouldn’t returning const error be the new convention anyway—one shouldn’t manipulate a given error in hindsight, should one?

I’m happy to give pages of more examples if interest is there.

PS. Type qualifiers in the D programming language

ianlancetaylor commented 2 years ago

@edvdavid Thanks for the report. That is very helpful.

robpike commented 2 years ago

Yes, thank you. Very thorough, thoughtful, and helpful.

Having watched const qualifiers arrive in C a long time ago, and all the trouble they caused, just like the ones you list here, I would have hoped subsequent language designers would leave const-ness out of their type systems. We did that for Go, but we were not language designers, just programmers designing a language.

psantoro commented 2 years ago

I do most of my professional coding in F#, but I also use GO. Here's my 2 cents on this proposal.

As a long time F# programmer, I vote for immutability by default with explicit use of a mut keyword. I've developed/maintained/refactored dozens of enterprise critical F# applications in the last decade. From my experience, having immutability by default truly does improve code simplicity/readability, code safety/correctness (especially with concurrent programming), and ease of refactoring. I use the F# mutable keyword sparingly. When I do use it, it's for local/private use only.

I think it's great that go is getting generics and would like to see immutability by default some day. I do understand that adding too many new language features, especially those with limited real world benefit, is counter productive to the original GO language design goals. Given the GO language promise of backwards compatibility, perhaps immutability by default could be implemented in the future via an opt-in compiler option (maybe in conjunction with the compiler's -lang version option).

Lastly, thank you to the GO team for all your hard work.

FrankDMartinez commented 2 years ago

I agree with psantoro: immutability by default is safer and allows for more optimizations.

deanveloper commented 2 years ago

I think that immutability by default is definitely the right way to go. However, that would be an extremely large change, as well as not being backwards-compatible.

FrankDMartinez commented 2 years ago

In my experience, especially if it makes for a better experience overall, backwards compatibility tends to be overrated. To put this into an analogy everyone can understand, so many of the issues people associate with the Microsoft operating system which don’t exist on Mac are directly related to backwards compatibility. Meanwhile, Apple is not afraid to say “Yeah, the previous system may have been good for its time and it might be inconvenient for some developers to make the updates but it’s a one-time cost and makes for a better system for the ultimate users.”

deanveloper commented 2 years ago

backwards compatibility tends to be overrated

I largely disagree here. The last thing we would want is a "python 2 vs python 3" stagnation, granted a bit more went into that other than a backward compatibility issue. In terms of Java, they made some backward-incompatible changes between LTS releases (8 and 11). We are now on Java 17 and a huge portion of companies (even modern ones) are still using Java 8.

Sure, it is a "one-time cost". However, we cannot feasibly promise backwards compatibility, then require nearly every library author, tool author, company, and really every user of the Go language, to rewrite all of their code. Sure, we could say that they don't need to rewrite their applications, and that libraries written in older versions of Go would have all exposed functions and types be imported as if they were all mut types. But that is not a good solution.

edvdavid commented 2 years ago

As I tried to show in my speech above, I don’t think it is possible to add useful backwards-compatible immutability to a system which has been designed based on mutability by default. To be useful and not harmful, immutability is too deep a structural difference.

C++ keeps backwards compatibility at all costs, at least the last time I looked at it, including astronomic compilation times because the preprocessor is still supported and obvious nonsense such as virtual int f() = 0.

In general, backwards compatibility has always a scalability limit beyond which it adds more damage than benefit. Obstructing progress and innovation can quickly be a higher price to pay than starting from scratch. For example, for backwards compatibility reasons, both the mass ounce and the mass ton have each three different definitions.

I always thought of Go as a shining example of favouring innovation and progress over backwards compatibility – one could still do everything in C, with perfection in backwards compatibility as well as unreadable code when using a C library to manage multiplexed I/O with coroutines and synchronising FIFOs.

Go, the most Unixian programming language since C, breaks holy C/Unix compatibilities such as pointer arithmetic (unsafe guarantees no compatibility), null-terminated strings or the C calling convention, the latter being well known for the mess when trying to pass types which don’t exist in C. And main returns no value.

ianlancetaylor commented 2 years ago

Go may break compatibility with C, but it does not break compatibility with itself. See https://golang.org/doc/go1compat. I believe this is essential for Go's success.

FrankDMartinez commented 2 years ago

@ianlancetaylor As far as I can tell, that document doesn’t actually say Go will never break backwards compatibility.

dfawcus commented 2 years ago

From my perspective of mainly using C, and only occasionally Go, there is really only one scenario where I make much use of 'const', that is for a assigning a name to an expression within a block which I then know can not change later in the block.

This is for aid in comprehension, especially when coming back to the code some time later, so something like:

void function (void)
{
...
    while ( some_condition() ) {
...
        bool const is_a_green_fish = is_a_fish(x) && is_green(x);
...
        /* some later expression uses 'is_a_green_fish' */
    }
...
}

Hence this is a variable which can not be reassigned later in the block, and I'm expecting a later compiler error to enforce that such that I reduce the mental state I'm carrying while reading code.

I suspect that could be achieved with a 'let' (or maybe 'once') keyword operating just like 'var' but with the semantic check phase of the compiler enforcing the no subsequent reassignment constraint, but I suspect there may be niggles with reference types.

Or possibly just allowing the existing 'const' to be used within a function to assign from an arbitrary expression (e.g. another function call), not just from a compile time constant expression.

i.e. in the following both 'j' and 'xyz' would be treated as if they had been declared 'var', with the difference that a subsequent reassignment would be a compiler error:

func something() int {
    return 8
}

func main() {
    for i := 0; i < 3; i++ {
        const j = i + 3
        fmt.Println("i: ", j)
    }
    const xyz = 2 + something()

    fmt.Println("xyz: ", xyz)
}

The only other places where I occasionally use const in C is for an object to be allocated in read only memory by the linker (file scope definitions), and sometimes in leaf functions if it can aid documentation / comprehension, bearing in mind that one can escape const-ness on structs even without casts. It is the latter which is in part desired here, but one can never trust it anyway, so it has little real value, and the aforementioned poisoning issue.

As to mutable vs non mutable data, I actually quite liked the way that worked in the Objective-C Foundation classes, with paired classes. However that does depend upon classful inheritance behaviour for calling immutable methods on a mutable value.

The option to optimise the compiler code generation for methods with non pointer signatures such that they can pass by reference under the covers, while still preserving pass by value semantics sounds helpful. i.e. the method would simply use the implicit pointer in the ABI as long as no assignments occurred to the value, if the value was assigned to, then code would be generated such that a complete or partial copy could be made within the method itself - so preserving pass by value semantics.

ianlancetaylor commented 2 years ago

As far as I can tell, that document doesn’t actually say Go will never break backwards compatibility.

Perhaps not, but it takes a clear engineering position: Go will only break backward compatibility for an awfully good reason.

By comparison, C has not broken backward compatibility since the first C standard in 1989. C++ has not broken backward compatibility since the first C++ standard in 1998, if we ignore the effect of additional keywords. Jave has, to the best of my knowledge, never broken backward compatibility at all. This approach has clearly worked well for these languages, all of which remain very popular. I think that Go should aspire to the same level of dependability.

deanveloper commented 2 years ago

Jave has, to the best of my knowledge, never broken backward compatibility at all

Java is planning on breaking certain unsafe reflection operations "in the future", but haven't done it yet. Currently, performing one of these unsafe operations will print a message to stderr.

noblehng commented 1 year ago

There are already many related proposals, and I think here is a better place to put my thoughts on.

First I don't think using same keyword for different concepts is a good idea, as other languages have showed. There are as least several concepts mixed here, immutable vs readonly view, and use them as storage class vs type qualifier. For their differences, I like D's description.

For Go's const keyword, it is used as Immutable Storage Class as in D. It is a property of an object, which means it is immutable throuth out it's lifetime after initialization. Readonly views are different, the object can still be mutated by other references, it is a property of the assigned variable. As of type qualifier, we don't have a const int type but const int value in Go.

So I would like to extent const in Go to all types of objects as in https://github.com/golang/go/issues/6386, like what D does for immutable storeage class. It is transitive and addressable, shadow copied to a var can modify that level's value, but not nested ones. Unlike D, casting of mutable to immutable should not be allowed, or atleast without using unsafe, so all nested value of an const object should either be const or copied in when initialization, or maybe unique reference at the time too. Like D, package level const should be constant expression evaluated at compile time, others could be evaluated at runtime, but check at compile time. For implementation, all references of a const must not appear in left hand side, compiler could record which parameter is modified in function meta data to assist implementation.

For readonly views, D use the const keyword, since Go already use that for immutable, it should not be use for this purpose. I think a compiler intrinsic function could be a good fit: func ReadOnly[T any](T) T. There is a noescape function in the runtime works like this. I believe this could solve many problem mentioned in the Evaluation of read-only slices proposal and better express the intention. Immutable express constructor's intention, and readonly view express the caller's intention when using in a function parameter, not the callee's.

Take io.Writer for example, it's the caller's intention for the callee not to modify the provided buffer, so something like (*reciver).Write(constrains.ReadOnly(buf)) is used. But the caller could also provide a large buf for the writer to modify inplace. Whether the writer satisfy the readonly constrain can then be verified at compile time. A constrains.IsReadOnly can also be use for specialization at compile time.

The constrain is scoped, so the return value's writability is the same as before calling constrains.ReadOnly, not as the referenced passing in argument. This way, a return subslice of a readonly view of a slice can be writable, if it's writable before the call, which usually is what the caller want. Like in trimed := bytes.TrimSpace(constrains.ReadOnly(line)), trimed will be writable if line is writable in caller before the call. This could be more clean than D's inout.

Avoid using const and readonly as type qualifier could solve the const poisoning problem. The public type and function signatures are the same for const or readonly or not, just like we could pass in a const or var to the same function in Go now, and we don't have to write exponential number of one line adapter for every combination even if the underlying implementation is the same. Instead, constrains.IsConst and constrains.IsReadOnly can be use for compile time specialization.

Other properties mentioned in https://github.com/golang/go/issues/24889, like restrict, which could help auto-vectorization, can be implemented as intrinsics like constrains.ReadOnly, without breaking backward compactability too.

Edit: In short:

sanathusk commented 1 year ago

I personally feel immutabilty must be added to go , even if not be default.we could introduce a new keyword imm similar to const & var to declare immutable struct.

js10x commented 1 year ago

As a huge fan of the simplicity that's currently made available by Go, I'll throw my two cents in the pot. While it's clear the proposer spent a lot of time and effort working this up, this is really just a matter of adding syntactic sugar really.

To the core team, thank you so much for being so determined to keep the language simple, whilst listening to your community. Go is an amazing programming language. I use it every day at work, and hope that the language stays as simple as possible as the years progress. Let's not let the features win, haha! Absolute features corrupt absolutely.

ilackarms commented 1 year ago

PLEASE add immutability. more important than generics were imho. this would make a huge diff in large software projects where there exists no strong way of ensuring data is read-only

matthinrichsen-wf commented 11 months ago

I'd like to hear of some experience reports where accidental mutation actually caused serious problems in a large Go code base

Hi, chiming in to provide current woes of a somewhat large Go code base (>1,000 packages with ~2million lines not including dependencies), currently struggling with accidental mutation. We currently have situations where a struct contains a map (or slice or some other pointer-backed data) which is widely used throughout the codebase. At some point, that struct is passed to a utility which believes it is the owner of that instance of the struct and mutates it in some way (writes to a map, appends/writes to a slice, changes a pointer etc). This causes those mutations to be seen by other holders of that variable. This bugs are very difficult to track down and often present as unexpected and surprising behavior requiring a lot of time spent debugging.

We've had to resort to deep copying these maps/slices/etc so that each caller does not mutate another's held copy accidentally. For large objects, this can balloon memory and be rather slow (it routinely shows up in our pprof graphs). It would be much better to have the option to enforce read-only structs/pointers/etc. It also causes a bit of a dependency nightmare where you must ensure that any sub-struct contained within a struct is also responsibly doing a deepcopy.

I'm not a huge fan of the proposed syntax, however. I find it interesting that Go has already solved this problem for channels via <-chan and chan<-. I'd like to at least entertain the idea of extending that paradigm to types:

type myStruct struct {
    names map[string]string
}

func readOnly(m <-myStruct) {
    readOnlyMapFunction(m.names)
...
}

func readOnlyMapFunction(names <-map[string]string) {
...
}

in which reads would be allowed, but writes expressly denied.

This does present the problem of const-poisoning as others have mentioned. But even in a codebase as large as the one I work in, I would take that tradeoff.

DeedleFake commented 11 months ago

With the usage of const as a modifier, there's been some confusion regarding its position and usage. What if instead of a modifier, a naming convention was used? The idea that's been floating around in my head recently is to allow ! to be used in identifiers, much like it is in Ruby, and say that any var, which includes receivers and arguments, that has a name that ends with ! would be immutable. That not only fits in well, in my opinion, with capitalization denoting visibility, but it would solve several questions of syntax:

var a! int
a! = 3 // Error.

a!, b := f()
a! = 3 // Error.
b = 3 // Allowed.

a! := 3
b := a! // Allowed because simple types copy.

a! := "An example."
b := a! // Allowed because strings are also immutable anyways.

a! := []byte("An example.")
b := a! // Error.

var a A
b! := a // Allowed for everything because immutability is only a guarantee that that variable can't be used to change something, not a guarantee that it won't change ever at all.

// Returns can't be immutable because of how defer works.
func (a! A) Im(b! B) (c C) {
  return a! + b!
}

p := &a! // Error. Can't store address of immutable variable in mutable variable.
p! := &a! // Allowed.

var a! struct { b! int } // Error. Struct fields can't individually be immutable.

var a! struct { p *int }
p := a!.p // Error. a!.p is immutable just like a! is.
p! := a!.p // Allowed.

*p! = 3 // Error.

m! := make(map[any]any)
f(m![3]) // Allowed.
m[3] = 2 // Error.
m2 := m! // Error.

var a! [3]int
s := a![:] // Error.

type A int
func (a A) M()
var a! A
a!.M() // Allowed.

type A int
func (a *A) M()
var a! A
a!.M() // Error.

type A struct { p *int }
func (a A) M()
var a! A
a!.M() // Error.

And so on.

Unfortunately, those last ones mean that without code migrating to this manually, that feature just won't be too useful, as making a variable immutable would limit you to only being able to call methods that expect it to be possible for it to be immutable. Existing code that hasn't changed its receivers to have a ! would just simply not work with it.

davi4046 commented 1 month ago

ImmuGo

Branch of Go... or Go compiler option👀

All references are immutable by default like in Rust unless declared mutable.

Silly idea? (yes, i havent thought this through and don't know enough about the subject to do so)

lorena-mv commented 1 month ago

What if instead of a modifier, a naming convention was used? The idea that's been floating around in my head recently is to allow ! to be used in identifiers, much like it is in Ruby, and say that any var, which includes receivers and arguments, that has a name that ends with ! would be immutable.

Hey @DeedleFake, just wanted to point out that if instead of ! you use characters allowed in a name, then you could write a linter that tells you if an immutable variable is being modified, which wouldn't require a language change.

For example, let's say _ at the end of the name means it's a mutable var (I think immutable should be the default):

var a int
a = 3 // Error.

a, b_ := f()
a = 3 // Error.
b_ = 3 // Allowed.

a := 3
b_ := a // Allowed because simple types copy.

a := "An example."
b_ := a // Allowed because strings are also immutable anyways.

a := []byte("An example.")
b_ := a // Error.

var a_ A
b := a_ // Allowed for everything because immutability is only a guarantee that that variable can't be used to change something, not a guarantee that it won't change ever at all.

// Returns can't be immutable because of how defer works.
func (a A) Im(b B) (c_ C) {
  return a + b
}

p_ := &a // Error. Can't store address of immutable variable in mutable variable.
p := &a // Allowed.

var a struct { b int } // Error. Struct fields can't individually be immutable.

var a! struct { p_ *int }
p_ := a.p_ // Error. a.p_ is immutable just like a! is.
p := a.p_ // Allowed.

*p = 3 // Error.

m := make(map[any]any)
f(m[3]) // Allowed.
m[3] = 2 // Error.
m2_ := m // Error.

var a [3]int
s_ := a[:] // Error.

type A int
func (a_ A) M()
var a A
a.M() // Allowed.

type A int
func (a_ *A) M()
var a A
a.M() // Error.

type A struct { p_ *int }
func (a_ A) M()
var a A
a.M() // Error.