spec: less error-prone loop variable scoping

[rsc]

We propose to change for loop variables declared with := from one-instance-per-loop to one-instance-per-iteration. This change would apply only to packages in modules that explicitly declare a new enough Go version in the go.mod file, so all existing code is unaffected. Changing the loop semantics would prevent unintended sharing in per-iteration closures and goroutines (see this entry in the Go FAQ for one explanation). We expect this change to fix many subtly broken for loops in new code, as well as in old code as it is updated to the newer Go version. There was an earlier pre-proposal discussion of this idea at #56010.

For example, consider a loop like the one in this test:

func TestAllEven(t *testing.T) {
    testCases := []int{0, 2, 4, 6}
    for _, v := range testCases {
        t.Run("sub", func(t *testing.T) {
            t.Parallel()
            if v&1 != 0 {
                t.Fatal("odd v", v)
            }
        })
    }
}

This test aims to check that all the test cases are even, but it doesn't check them all. The problem is that t.Parallel stops the closure and lets the loop continue, and then it runs all the closures in parallel when the loop is over and TestAllEven has returned. By the time the if statement in the closure executes, the loop is done, and v has its final iteration value, 6. All four subtests now continue executing in parallel, and they all check that 6 is even, instead of checking each of the test cases.

Most Go developers are familiar with this mistake and know the answer: add v := v to the loop body:

func TestAllEven(t *testing.T) {
    testCases := []int{0, 2, 4, 6}
    for _, v := range testCases {
        v := v // MAKE ITERATION VALUE SHARING BUG GO AWAY
        t.Run("sub", func(t *testing.T) {
            t.Parallel()
            if v&1 != 0 {
                t.Fatal("odd v", v)
            }
        })
    }
}

Changing the loop semantics would in essence insert this kind of v := v statement for every for loop variable declared with :=. It would fix this loop and many others to do what the author clearly intends.

A subtler variation is when the code says testCases := []int{1, 2, 4, 6} (note the non-even test case 1):

func TestAllEvenBuggy(t *testing.T) {
    testCases := []int{1, 2, 4, 6}
    for _, v := range testCases {
        t.Run("sub", func(t *testing.T) {
            t.Parallel()
            if v&1 != 0 {
                t.Fatal("odd v", v)
            }
        })
    }
}

TestAllEvenBuggy passes today, because the test is only checking 6, four times. Changing the loop semantics would still fix the loop to do what the author clearly intended, but it would break the test, because the test is passing incorrectly. So there is the potential to cause problems for users.

Because of this potential for problems, the new loop semantics would only apply in Go modules that have opted in to the release with the new loops. If that was Go 1.22, then only packages in a module with a go.mod that says go 1.22 would get the new loop semantics. Packages in other modules, including packages in dependencies, would get the old semantics. This would guarantee that all existing Go code keeps working the same as it does today, even when compiling with a new toolchain. People only need to debug loop changes when they opt in to the new semantics in go.mod. This approach is in keeping with our backwards and forwards compatibility work, #56986 and #57001, specifically the principle that toolchain upgrades preserve the behavior of old code, and compatibility is based on the go line.

If this proposal is accepted, users will need additional tooling support for a successful transition. That would come primarily in two forms: a compiler mode that reports affected loops, and a debugging tool that identifies the specific loops whose changed compilation is responsible for causing a test failure. There is a demo of the tooling support at the end of this comment.

The tooling support is important and necessary, but we expect it to be needed only rarely. Analysis and conversion of Google's own Go source code found that only about 1 package test per 8,000 started failing due to the new semantics, and the bug was essentially always in the test itself, like in TestAllEvenBuggy. In contrast, the new loopclosure vet analysis that shipped in Go 1.20 flagged definite bugs in 1 test per 400. The rest stayed passing with their bugs fixed by the new semantics. That is, in Google's code base, about 5% of the tests that contain this kind of sharing mistake were like TestAllEvenBuggy, exposed as buggy by the new semantics. The other 95% of the tests with this kind of mistake still passed when they started testing what they intended to. Of all the test failures, only one was caused by a loop variable semantic test change in non-test code. That code was very low-level and could not tolerate a new allocation in the loop. The design document has details. This evidence suggests that changing the semantics is usually a no-op, and when it’s not, it fixes buggy code far more often than it breaks correct code.

Based on the preliminary discussion the further work summarized here, @dr2chase and I propose that we make the change in an appropriate future Go version, perhaps Go 1.22 if the stars align, and otherwise a later version.

More details can be found in the design document.

Update, May 10 2023: Note that the change will not break the common idiom of changing a loop variable in the body of a 3-clause for loops. See this comment for details and links before commenting about 3-clause for loops. Thanks.

---

Tooling Demonstration

To demonstrate the tooling we would provide to support a successful transition, here is a complete example of a test that passes today but fails with the new loop semantics:

% cat x_test.go
package x

import "testing"

func Test(t *testing.T) {
    testCases := []int{1, 2, 4, 6}
    for _, v := range testCases {
        t.Run("sub", func(t *testing.T) {
            t.Parallel()
            if v&1 != 0 {
                t.Fatal("odd v", v)
            }
        })
    }
    for _, v := range testCases {
        t.Run("sub", func(t *testing.T) {
            t.Log(v)
        })
    }
    for _, v := range testCases {
        t.Run("sub", func(t *testing.T) {
            t.Parallel()
            if v&1 != 0 {
                t.Fatal("odd v", v)
            }
        })
    }
}
%

Building the program with -d=loopvar=2 reports all the affected loops:

% GOEXPERIMENT=loopvar go test -gcflags=all=-d=loopvar=2 x_test.go
# runtime
go/src/runtime/proc.go:1815:7: loop variable freem now per-iteration, stack-allocated
go/src/runtime/proc.go:3149:7: loop variable enum now per-iteration, stack-allocated
go/src/runtime/mgcmark.go:810:6: loop variable d now per-iteration, stack-allocated
go/src/runtime/traceback.go:623:7: loop variable iu now per-iteration, stack-allocated
go/src/runtime/traceback.go:943:7: loop variable iu now per-iteration, stack-allocated
# runtime/pprof
go/src/runtime/pprof/pprof.go:386:9: loop variable r now per-iteration, heap-allocated
go/src/runtime/pprof/proto.go:363:6: loop variable e now per-iteration, stack-allocated
go/src/runtime/pprof/proto.go:612:9: loop variable frame now per-iteration, heap-allocated
go/src/runtime/pprof/protomem.go:29:9: loop variable r now per-iteration, heap-allocated
# command-line-arguments [command-line-arguments.test]
./x_test.go:7:9: loop variable v now per-iteration, stack-allocated
./x_test.go:15:9: loop variable v now per-iteration, stack-allocated
./x_test.go:20:9: loop variable v now per-iteration, stack-allocated
# internal/fuzz
go/src/internal/fuzz/fuzz.go:432:9: loop variable e now per-iteration, stack-allocated
--- FAIL: Test (0.00s)
    --- FAIL: Test/sub (0.00s)
        x_test.go:11: odd v 1
    --- FAIL: Test/sub#08 (0.00s)
        x_test.go:24: odd v 1
FAIL
FAIL    command-line-arguments  0.199s
FAIL
%

Note that some loops are in the Go standard library. Those are unlikely to be the cause, so we can limit the diagnostics to the current package by dropping all=:

% GOEXPERIMENT=loopvar go test -gcflags=-d=loopvar=2 x_test.go
# command-line-arguments [command-line-arguments.test]
./x_test.go:7:9: loop variable v now per-iteration, stack-allocated
./x_test.go:15:9: loop variable v now per-iteration, stack-allocated
./x_test.go:20:9: loop variable v now per-iteration, stack-allocated
--- FAIL: Test (0.00s)
    --- FAIL: Test/sub (0.00s)
        x_test.go:11: odd v 1
    --- FAIL: Test/sub#08 (0.00s)
        x_test.go:24: odd v 1
FAIL
FAIL    command-line-arguments  0.119s
FAIL
%

That trims the diagnostic output, but it still leaves us to check all our loops. In this trivial example, 2 of 3 are buggy, but in a real program there are fewer needles and more haystack. To solve that problem, we can use a dynamic tool that reruns a test, varying which loops compile with the new semantics, to identify the specific loops that cause the failure:

% bisect -loopvar go test x_test.go
bisect: checking target with all changes disabled
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=n x_test.go... ok (13 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=n x_test.go... ok (13 matches)
bisect: checking target with all changes enabled
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=y x_test.go... FAIL (13 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=y x_test.go... FAIL (13 matches)
bisect: target succeeds with no changes, fails with all changes
bisect: searching for minimal set of changes to enable to cause failure
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+0 x_test.go... ok (7 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+0 x_test.go... ok (7 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+1 x_test.go... FAIL (6 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+1 x_test.go... FAIL (6 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+01 x_test.go... FAIL (3 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+01 x_test.go... FAIL (3 matches)
...
bisect: FOUND failing change set
--- change set #1 (enabling changes causes failure)
./x_test.go:7:9
---
...
bisect: FOUND failing change set
--- change set #2 (enabling changes causes failure)
./x_test.go:20:9
---
...
bisect: target succeeds with all remaining changes enabled
%

Now we know to spend our attention on the loops at lines 7 and 20, which are in fact the buggy ones. (The loop at line 15 has been cleared of suspicion.) Bisect uses an adaptation of binary search that can identify the relevant loop in a relatively small number of trials, making it useful even on very large, very complicated tests that run for a long time.

[AndrewHarrisSPU]

@rsc You will be able to do it with //go:build go1.20 per file IIRC.

Thanks, I hadn't noticed https://github.com/golang/go/issues/59033 - per-file seems forgiving enough. I'm fairly skeptical that a rewrite tool is a better choice than playing with build tags.

[zigo101]

@atdiar The i iteration variable is not important here, I may remove it totally. It is just used to do the demonstration.

@rsc All the three new "footgun" exmaple code pieces have their own unique characteristics. I haven't seen an example like the third one before posting it here.

@DmitriyMV In your code, you chose a way to avoid the problem, not solve it. :)

Up to now, the discussion in this thread has proven that the discussion on the proposed changes is still not sufficient enough. We need more Go programmers to be aware of the changes before deciding to accept it or not. I estimate less than 1% Go programmers are aware of the changes now. Considering there are many private Go projects, we need to wait more Go programmers to participate the experiment.

Go supports pointers, large-size types, closures, and multi-value assignments. All of these make Go 3-clause loops have more variation uses than other languages. We should be careful to make their semantic changes.

[atdiar]

@go101 you can remove i but the question of which value is captured in the closure remains.

counter is not a pointer. So its lifetime being extended outside of the scope of the for... loop can be quite surprising as well and it might not be the behavior that should have existed in the first place.

I guess @rsc idea is for people to test their existing code to know whether people rely on the old scoping rules or not. Given that for the corpus he tested, iteration-local scoping was a net positive.

[ulikunitz]

At first let me state that I have no concerns regarding the proposal. I checked a number of my modules with GOEXPERIMENT=loopvar gotip and my unit tests were all successful. While that doesn't prove a lot, it strengthens my confidence there is no practical problem. I suggest others do the same.

I agree however that one must get used to the new semantics for the three-clause for-loop because my personal mental model was that

var a []*int
for i := 0; i < 3; i++ {
     a = append(a, &i)
}

is equivalent to

var a []*int
{
    i := 0
    for i < 3 {
       a = append(a, &i)
       i++
    }
}

This mental model will be wrong going forward. I don't regard it as an issue, because most loops will not change semantics at all, if there is no closure or an address reference in the loop. Where it is, there is a high likelihood that a lingering bug will be fixed.

[danieljl]

I'm all for the semantic change for range loops. As for 3-clause loops, I was indifferent at first. But the more I think about it, the more I'm convinced that the 3-clause loops semantic shouldn't be changed.

we are going to make this decision based on evidence [emphasis on original]

I agree with @rsc about this statement. The important numbers we should seek and compare are # bugs fixed vs. # bugs caused by the change.

But we don't live in an ideal world. It's hard to gather unbiased and representative evidence for many reasons:

1. It's easier to find cases where bugs fixed than cases where bugs caused by the change. For example, we can find the former cases by searching x := x, as done by @rsc. On the other hand, it's hard to formulate search terms to find the latter cases. In any case, we can only find things we expect to find. The problem is that there may be much more cases that we don't think of, or even when we can think of some of them, we can't formulate the appropriate search terms.

2. Maybe the only alternative to relying on code search is using tests to get the number of bugs fixed and caused by the change. However, we need to realize that many code is not well tested, much less having 90%+ code coverage. Not to mention that, as Dijkstra famously said:

   Program testing can be used to show the presence of bugs, but never to show their absence!

   That means the fact that all tests pass doesn't imply that there are no bugs. We can only prove there are no bugs using formal methods, which in most cases not practical at all.

3. So far the code we've been getting evidence from is only from GitHub public repos and Google's codebase. This implicitly assumes that code from them is a representative sample of all Go code. However, open source Go packages and Google's codebase may be of higher quality and more well-tested than code from many other companies, which—due to deadlines or very specific requirements—may use hacks and obscure tricks.

If the problems above apply to both range loops and 3-clause loops, why do I support the semantic change of one and not of the other? It's because there are much fewer ways you can use (and abuse) range loops than 3-clause loops. The breakages caused by the new semantic of range loops revolve around the use of the address of the loop variable. In contrast, you can do almost anything in 3-clause loops.

Let's look at the grammar of the 3-clause loops from the spec:

ForClause = [ InitStmt ] ";" [ Condition ] ";" [ PostStmt ] .

InitStmt = SimpleStmt .
PostStmt = SimpleStmt .

Condition = Expression .

As you can see, we can do much more with SimpleStmt and Expression than (ab)using loop variable addresses in the range loops case.

Of course at the end of the day we still can't get the exact numbers of bugs fixed vs. bugs caused by the change. However, the numbers we have now may underrepresent the real distribution. As much as I want the decision is based on empirical evidence (because it's more objective and no one can argue with the numbers), I don't think gathering the evidence is an easy task. So, we may need to discuss things more subjective, e.g. which is more intuitive, which is more common, which requires less mental overhead, etc.

[atdiar]

@danieljl Does it break really?

[Merovius]

I agree with @rsc about this statement. The important numbers we should seek and compare are # bugs fixed vs. # bugs caused by the change.

But we don't live in an ideal world. It's hard to gather unbiased and representative evidence for many reasons:

It's certainly true that it's hard to get unbiased evidence. But we have some evidence that this change not only fixes some bugs, but also doesn't introduce any, for 3-clause loops. Like, this isn't just the absence of evidence, it's evidence for absence.

For example, the pattern below is not uncommon and will break in the new semantic.

I might be daft, but can you explain the breakage? Because I can't see it. From what I can tell, IsDone and Done have pointer-receivers (as they have side-effects). So I'd expect InitFoo to also return a pointer, in which case this seems fine.

And even if InitFoo returns a value (which would be exceedingly rare), then its value would get copied back-and-forth, including all fields. So, that still seems fine. It might break if its methods start goroutines that operate on fields or escape pointers to its fields or have mutexes that should not be copied - but in those cases, returning a value from InitFoo is really bad idea.

So… I don't think this would break in any common usage of this pattern?

[danieljl]

But we have some evidence ... this isn't just the absence of evidence, it's evidence for absence. — @Merovius

If it is some unbiased enough (no need to be 100% unbiased) evidence, it's good to use. But in this case it's clearly biased evidence, which may be worse than no evidence, because it can be misleading.

From what I can tell, IsDone and Done have pointer-receivers (as they have side-effects). So I'd expect InitFoo to also return a pointer, in which case this seems fine. — @Merovius

I don't understand how you could come into that conclusion. The fact that some methods have pointer receivers doesn't require the "constructor" to return a pointer.

And even if InitFoo returns a value (which would be exceedingly rare), then its value would get copied back-and-forth, including all fields. So, that still seems fine.

You may have some misunderstanding of how shallow copy / memcpy works because it doesn't work like you think it is. If InitFoo returns a value, the new semantic will break the code. Demo: https://go.dev/play/p/sxrRvfeeRx6 (In the current semantic it will output "go" once, whereas in the new semantic it will output "go" indefinitely and will never finish the loop.)

[fzipp]

In the current semantic it will output "go" once, whereas in the new semantic it will output "go" indefinitely and will never finish the loop.

@danieljl I can't confirm this with GOEXPERIMENT=loopvar:

% go install golang.org/dl/gotip@latest
go: downloading golang.org/dl v0.0.0-20230502172222-5216546bad51
% gotip download
Updating the go development tree...
...
Building Go cmd/dist using /usr/local/go. (go1.20.3 darwin/arm64)
...
Success. You may now run 'gotip'!
% cat x.go
package main

import "fmt"

func main() {
    for foo := InitFoo(); !foo.IsDone(); {
        fmt.Println("go")
        foo.Done()
    }
}

type Foo struct {
    isDone bool
}

func InitFoo() Foo {
    return Foo{isDone: false}
}

func (f *Foo) Done() {
    f.isDone = true
}

func (f *Foo) IsDone() bool {
    return f.isDone
}
% gotip run x.go
go
% GOEXPERIMENT=loopvar gotip run x.go
go
% 

[atdiar]

@danieljl I think the new semantics are closer to this: https://go.dev/play/p/39xzBuu8Ou8

Which only prints "go" once as well.

[DmitriyMV]

If it is some unbiased enough (no need to be 100% unbiased) evidence, it's good to use. But in this case it's clearly biased evidence, which may be worse than no evidence, because it can be misleading.

This is derailing the discussion. Either you have a real world examples proving your point or you not. So far all opponents of this change didn't find provide anything other than "I can write this code, that will be broken with this change, so I don't like this change.". On the other hand, we have numerous bugs (even linked with that discussion) that will be fixed.

Also - before saying things "biased" please provide the evidence.

Demo: https://go.dev/play/p/sxrRvfeeRx6 (In the current semantic it will output "go" once, whereas in the new semantic it will output "go" indefinitely and will never finish the loop.)

Please read the proposal again and all previous discussions. @atdiar provided the correct example (tho assignment in the end should happen unconditionally even if loop contains continue).

[danieljl]

@atdiar Thanks for correcting my code emulating the new semantic.

---

@DmitriyMV

If it is some unbiased enough (no need to be 100% unbiased) evidence, it's good to use. But in this case it's clearly biased evidence, which may be worse than no evidence, because it can be misleading.

This is derailing the discussion. Either you have a real world examples proving your point or you not.

Did you even read my first long explanation of why I think the evidence presented so far is biased?

Also - before saying things "biased" please provide the evidence.

The evidence of what? The evidence that the evidence presented so far is biased OR the evidence against the semantic change of 3-clause loops? In either case, again, please read my first post: https://github.com/golang/go/issues/60078#issuecomment-1546606504

---

So far no one disagreed or even commented about my 3-point argument that the evidence presented so far is biased; but instead everyone focused on one specific example, which I admit, incorrectly demonstrates the new semantic. I wonder if that means no one really disagrees that the evidence presented so far is biased / not representative. If someone argues otherwise, I'd like to hear their argument.

[Merovius]

@danieljl

So far no one disagreed or even commented about my 3-point argument that the evidence presented so far is biased

I disagree. I did comment on it. Also, feel free to provide better evidence.

[DmitriyMV]

@danieljl

On the other hand, it's hard to formulate search terms to find the latter cases.

Not really. Go /x tooling gives you enough info about types, and there are tools like go-ruleguard and gogrep that can help you identify such cases. You don't need to actually perform text search and you can find very specific cases if you go with AST and types lookup.

In any case, we can only find things we expect to find.

We are getting dangerously close to Russell's teapot. So far we have seen plenty of evidence supporting fixing this behaviour, and none for not doing it. Even opponents failed to provided real world examples, and they had plenty of time.

So far the code we've been getting evidence from is only from GitHub public repos and Google's codebase. This implicitly assumes that code from them is a representative sample of all Go code. However, open source Go packages and Google's codebase may be of higher quality and more well-tested than code from many other companies, which—due to deadlines or very specific requirements—may use hacks and obscure tricks.

Not really. The fact that this behavior resulted in numerous bugs in Google services as well as many public repositories says that people tend to think alike in common scenarios. This is what we can actually see and observe.

However, the numbers we have now may underrepresent the real distribution.

I'm getting tired of this "may or may not happen". So far evidence, collected from several resources, have been overwhelmingly supportive of this change. Almost all people here, on reddit, on hacker news and other discussion platforms support this change, with some being surprised that it doesn't work that way currently. I have seen none real objections to making this change that do not start with with word "may" or "maybe".

I will stop here cause I see no point continuing this discussion. Not until some new real world evidence will arise.

[zigo101]

@atdiar per your comment. I agree with you sometimes people expect an iteration variable in a traditional 3-clause loop to be a single loop-step scoped. What I disagree with you is that this is not always true, which is a big different from for-range loops. This has been proven in the 3 new footgun examples.

@danieljl I agree with you viewpoints. But admittedly, the example in your comment looks not like a broken case.

I won't refute some other skewed viewpoints. In fact, I will feel ashamed if I hold those viewpoints.

I'm helping this project by showing some bad effects of this proposal. In fact, I have not the responsibility to do this, not to mention that I have not responsibility to find the use cases in real world. It is the responsibility of the proposal makers to do this. If they think this is hard to do, then we should encourage the whole community to distribute the work load.

[zigo101]

As mentioned in the last comment, iteration variables in traditional 3-clause loops are sometimes expected to be one single loop-step scoped. and sometimes expected to be whole-loop scoped.

Currrently, the third clause in a traditional 3-clause loop can't be a re-declaration statement. We can remove this limit and use a re-declaration statement to explicitly show which iteration variables are expected to be one single loop-step scoped. For example

    for i := 0; i < 3; i := i+1 {
        prints = append(prints, func() { println(i) })
    }

    for counter, i := 0, 0; i < 5; i := i + 1 {
        if i&1 == 0 {
            defer func() {
                counter++
                fmt.Printf("#%d: %d\n", counter, i)
            }()
        }
    }

This will make things keep explicit and not break backward compatibility.

---

[update]: just an unproven theory assumption: it looks the iteration variables expected to be one single loop-step scoped are declared in the first clause and modified in the third clauses through assignments. If this is true, we can let new versions of go fix to modify the assignment statements in the third clauses to re-declaration statements. This is the bottom-line to let me think the change is not reckless and devastating.

[atdiar]

@go101 the point is that anyone can build synthetic examples that somewhat rely on the old scoping rules but the question is whether anyone does in real programs.

That's why @rsc asked about real code examples. :)

You will be able to convince him and others if you can find examples in real code where people relied on the old scoping behavior.

[Merovius]

You will be able to convince him and others if you can find examples in real code where people relied on the old scoping behavior.

To be clear: I wouldn't put it that strongly. Currently, the evidence indicates that the overwhelming majority of programs would benefit from this change. Producing a single real-world example would not disprove that. It's just that the inability to not even produce a single one seems to strengthen the conclusion that there are vanishingly few.

I think to be convincing, an example would have to either 1. be a pretty common pattern that we can find in lots of real code, or 2. exhibit real novelty, a behavior we did not anticipate but that would be clearly natural to write. Which would demonstrate that we actually misunderstood how easy it is to make a mistake under the new semantics.

The examples so far don't do 1, as they are entirely synthetic. And they don't do 2, as they are either absurdly unreadable and their effect would be hard to tell under any semantic (the ones with many function calls and pointer-comparisons in the loop clauses) or actually pretty easy to reason through and not really surprising (the counter example).

[zigo101]

@atdiar Again, it is not my responsibility to show the real code example. It is the responsibilities of the proposal makers (to prove no such cases are in all the go code in the world). :) My help is just to show the possibilities.

[atdiar]

@go101 oh I don't think anyone is asking specifically that you find an example in someone else's codebase of reliance on such behavior.

It's mostly about you reporting cases extracted from your own real code or the dependencies you use.

That's the only practical way to go about it. (distributed).

But just thinking about when one would want such behavior, we can already intuit that it would be exceedingly rare if not always a bug.

[Merovius]

The claim has never been that there are no cases where this proposal would break real-world code. The claim was that this proposal will fix overwhelmingly more problems than it causes. Moving the goal post like that is clearly designed to create an impossible to fulfill standard of proof (it's impossible to prove a negative) to gridlock the discussion and sabotage consensus. There is no point in having an argument where one side is not interested in reaching a consensus. Therefore, I strongly suggest to stop engaging.

@rsc has provided strong evidence that this change - for both kinds of loops - fixes overwhelmingly more problems than it causes. If anyone is dissatisfied with that level of evidence, they are free to provide their own. Or at the very least, provide clear, falsifiable claims on how such evidence can be obtained. Until then, there's just no point in belaboring this argument.

[danieljl]

The claim has never been that there are no cases where this proposal would break real-world code. The claim was that this proposal will fix overwhelmingly more problems than it causes.

Yeah, that's why I support the semantic change of range loops. As I said before, because there are only a few ways range loops can be (ab)used, # problems fixed >>>>> # problems caused.

As for the semantic change of 3-clause loops, I hope at least we can all agree that, although the problems caused by both types of loops are not many, the semantic change of 3-clause loops will cause more problems than that of range loops will do. That's because, as I explained before, there are more ways 3-clause loops can be (ab)used. (For details, please read a later section in my original comment that explains the grammar of 3-clause loops: https://github.com/golang/go/issues/60078#issuecomment-1546606504 .) So, even if there will be more problems fixed than problems caused, it won't be as dramatic as in the range loops case. In the 3-clause loops case, # problems fixed >> # problems caused.

We must remember that we will break our stability promise. We should only do that if the change will break almost no code (which I believe true for the range loops case, but not for the 3-clause loops case). If we accept any change that will fix more than it will break, this proposal won't be the last that will break the stability promise. (Aside: Will it be the first proposal that will break it?)

To reiterate and summarize my points about sampling bias, basically there are two parts of the problem:

1. How can we be sure that we don't oversample # problems fixed and undersample # problems caused?

2. Even if we can solve the first problem, there is a problem that is harder to solve: How can enlarge our sample so that we don't rely only on public code and Google's codebase? It may require a large-scale questionnaire project involving many companies that heavily use Go. We need to realize that today only a fraction of Go users are aware about the planned change.

That's what I think we should do if we'd like to go with the empirical evidence route. Or alternatively, we can approach it analytically, more specifically, "combinatorically", i.e. acknowledging from the for grammars that there are much more ways 3-clause loops can be (ab)used than there are for range loops.

---

On a side note, I think more people will support the semantic change of 3-clause loops, if—as unlikely as it is—Go 2 is released to accommodate this proposal. (Why don't we want to do that, though? Because Go 2 is reserved for bigger breaking changes? But if we are to follow semver, we should increase the major version.) That way, virtually all Go users will be aware of the breaking change, and no promises will be broken. In contrast, much fewer users will be aware of the breaking change if only minor version is incremented.

[rsc]

@go101 and @danieljl, we have heard your arguments, many times now. The fact that most of the participants here still disagree with you does not mean you have been unclear or misunderstood. Please step back and leave room for others to participate and bring up other concerns. Thank you.

[danieljl]

we have heard your arguments, many times now. The fact that most of the participants here still disagree with you does not mean you have been unclear or misunderstood. Please step back and leave room for others to participate and bring up other concerns. Thank you.

@rsc I thought we still have an open and unfinished discussion. You—the very person who suggested that empirical evidence is more important than anything—didn't even respond to my original post that challenges the underlying assumptions of using empirical evidence. But sure, I'll step back if no one else want to discuss my concerns anymore. If you consider my comments as noises, I apologize.

Maybe if I can ask, when the proposal is accepted and the next Go version is released, please at least mention the semantic change in the title of the release announcement post. Also, in the changelogs please make the semantic change very conspicuous, e.g. by making the font bold and red. That way, hopefully more Go users will be aware of the broken stability promise and can plan accordingly.

[purpleidea]

Fantastic, and I'm really looking forward to this change!

My only real question: Can you provide more examples of situations where this breaks things? (Even when they are cases of less than ideal code.) I saw the allocation use-case mentioned. Very interesting, thank you. You also mention certain percentages above, so I assume you have lots more examples. In particular, I might have missed the example but you wrote:

Of all the test failures, only one was caused by a loop variable semantic test change in non-test code.

Could you show us this code, or did I miss it in the wall of text above? Cheers!

[ianlancetaylor]

Of all the test failures, only one was caused by a loop variable semantic test change in non-test code.

Could you show us this code, or did I miss it in the wall of text above?

As it says above, the design doc has more details. In this case see https://go.googlesource.com/proposal/+/master/design/60078-loopvar.md#fixes and search for "a test checks that the code contains no allocations or even runtime write barriers."

[purpleidea]

search for "a test checks that the code contains no allocations or even runtime write barriers."

Yeah I saw that one, thank you! I thought the original message alluded to some more, sorry. If there are any other known bugs at all created by this change, I'd like to see some!

Keep up the great work!

[zigo101]

@purpleidea in case you haven't read them:

• https://github.com/golang/go/issues/60078#issuecomment-1546019456
• https://github.com/golang/go/issues/60078#issuecomment-1544324607 (overlap with the last one)
• https://github.com/golang/go/issues/57969#issuecomment-1426821588

[dr2chase]

@purpleidea that failing example is an interesting one. If the analysis were a little pickier, it would not have applied there. The address is taken on a code path that ends in a break, so it is technically no longer in the iteration (though it is in scope). We implemented a minor optimization for an easier-to-recognize case if the addressed variable is instead returned, e.g., return &saved. I thought about doing it for break, but so far, have not, partly because it is trickier than return.

[thepudds]

Hi @dr2chase

@purpleidea that failing example is an interesting one. [...]

Just to confirm, you are talking about how a potential future improvement to the complier could have avoided a new allocation under the new semantics (which normally would be an optimization whose absence wouldn't cause a test failure, but in this case the test was explicitly checking that no allocations occurred)?

Related snippet from the design doc:

var original *mapping
for _, saved := range savedMappings {
  if saved.start <= m.start && m.end <= saved.end {
      original = &saved
      break
  }
}

Unfortunately, this code was in a very low-level support program that is invoked when a program is crashing, and a test checks that the code contains no allocations or even runtime write barriers.

[crhntr]

I’ve been thinking about this a bit. At first the tiny performance hit bugged me. A person in my company slack made a comment that we will will soon have "performance loops" and "range loops". I sat with that idea for a bit. I don't think it is not a bad thing. In addition to the common bugs with for-range loops that this proposal addresses, I have found the order of assignment variables in a for-range loop to be confusing for new gophers and new programmers. Last week I pair programmed with a senior engineer who paused trying to remember what the order of values on the left hand side of the for-range expression was. I felt that confusion. I used to write Vue.js templates and would get confused going back and fourth (they do it the other way around).

Go programmers tend to prioritize readability over compact code. I feel moving forward (regardless of whether this proposal is accepted) I will prefer to use for-index loops when iterating though slices/arrays and write them like so (this is less error prone, more performant (not that it usually matters), and I need to remember less syntax):

for index := 0; index < len(slice); index++ {
  element := slice[index]
  //...
}

I will continue to use the range loop for maps and channels. I've noticed the ambiguity on the left side of the range expression is not an issue in those cases.

Go programmers tend to prioritize welcoming new programmers. Making for-range loops less error prone for those who care to use it seems inclusive as does using loops that read more like C/Javascript for-index loops.

[dr2chase]

@thepudds yes, that one failure, if we/I had implemented that trickier optimization (I am not adept at tree-node transformations) would have avoided transforming that loop and thus also avoided that failure.

[markusheukelom]

I am programming full-time in Go for the last 5 years, and I have only once or twice run into a bug caused by this (which was really quickly solved) in all these 5 years. (Maybe dislike this comment if you have ran into it more then, say once a year).

To show I am really not a genius programmer, I have about 50 times more run into the stupid mistake of updating an array element that is iterated over by value in a range operation (ie. for v, i := range array { v = ...} ). Even that was not a big problem. So I wonder if the cure is worth the hassle. (Real world examples that illustratie it would really help decide this).

I am saying this especially because, if this is implemented, it would potentially force the programmer to lookup the Go version in the mod file, for every package worked on, all code reviews, etc. Then, what if you want to use say go >1.22 for other stuff but do not want to check all for loops in all your code (yet)? I mean you're kinda forced to it. I will add hooks for senseless team discussions too, maybe.

All in all, people will manage of course, I certainly won't have sleepless nights if this is added. I just want to make the note that it adds another dime to the bag of cognitive load without solving any significant existing problems in reality? It may also serve as "prior art" for other future "incompatible" language tweaks because "we did that for-loop thing too".

Finally, I find the example given in the proposal a bit misleading because safe&correct parallel programming without channels is really hard anyway (to me, anyways). So this proposal could be based off some more mundane example to do it more right maybe.

[zigo101]

Totally agree. This is indeed another concern point I plan to make. Using minor semvars to mark feature adding is a good idea, but it is not a good idea to mark behavior changes. I even think it is also not a good idea to use major semvars to mark language semantic changes.

[pkorotkov]

I'm not sure I'm getting a practical problem to be solved by this surgery. Despite seemingly good reasoning you are about to break the compatibility promise where clearly stated that "It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification." Such decisions open a Pandora's box of random "language improvements" with unpredictable fallout.

Along with issues for Go beginners, the worst scenario I expect is tricky bugs in third-party libraries accidentally/incidentally upgraded. Vast majority of my Go projects have tons of dependencies and some of them contain spaghetti code. This imperfect code is not under my control. The only merit of this code is that it's time proved and stable. Accepted this proposal, this stability would be terminated for long.

[Merovius]

I am programming full-time in Go for the last 5 years, and I have only once or twice run into a bug caused by this (which was really quickly solved) in all these 5 years. (Maybe dislike this comment if you have ran into it more then, say once a year).

There are two costs associated with issue: One is actual bugs slipping through reviews and causing issues, the other is having to put x := x into your code. For the former, we have pretty concrete data. The second is harder to quantify, but it's certainly annoying and confusing. The question "why is there an x := x here?" comes up a lot with newcomers.

There's also the point that something that may happen rarely individually, can happen frequently at scale. If this bug happens to every Go programmer ~once a year, at an estimated ~1M Go developers, that's ~1M preventable bugs per year.

I am saying this especially because, if this is implemented, it would potentially force the programmer to lookup the Go version in the mod file, for every package worked on, all code reviews, etc.

I don't think this is quite as big a problem in practice. The vast majority of loops are not affected. I'd also expect most people who do code reviews to be somewhat aware of the context of the project.

It's also a temporary issue. After a year or so, older Go versions will be unsupported and phased out.

Then, what if you want to use say go >1.22 for other stuff but do not want to check all for loops in all your code (yet)? I mean you're kinda forced to it.

That is kind of true. But note that there is tooling to make the upgrade as simple as possible. In particular, you don't have to check all loops - the bisect tool can give you a set to check on. And there are escape hatches. If everything fails, you could even put //go:build go1.22 into all existing .go files to get the old behavior and then put the stuff that needs a newer Go version into a separate file with //go:bulid go1.23.

But, really, if you are affected by the migration, that means you most likely currently have a bug you didn't notice. I'm not sure I can sympathize with that suddenly being a huge problem.

[Merovius]

@pkorotkov The proposal is aware that this constitutes a break of the compatibility promise. As for the slippery slope argument, it is also pretty clear that this is a one-time exception. No one wants a world where we do things like this more often.

For your concern about 3rd party code, note that nothing about that will change, unless that 3rd party itself decides to upgrade to a newer Go version. That is, if the go.mod of your dependency says go 1.21, it will be compiled with the current semantics for loops, regardless of what your go.mod says.

[thepudds]

Hi @markusheukelom

Finally, I find the example given in the proposal a bit misleading because safe&correct parallel programming without channels is really hard anyway (to me, anyways). So this proposal could be based off some more mundane example to do it more right maybe.

Here are some non-concurrency examples.

CL 40937 fixes a bug in x/crypto/ssh that is a good example of how the current semantics can bite you even without any concurrency or parallel programming. Here the bug is due to the pointer that is stored in the map:

    knownKeys := map[string]*KnownKey{}
    for _, l := range db.lines {
        if l.match(addrs) {
            typ := l.knownKey.Key.Type()
            if _, ok := knownKeys[typ]; !ok {
                knownKeys[typ] = &l.knownKey
            }
        }
    }

And here's a simplified version (taken from the CL description):

    var p *int
    a := []int{0, 1, 2, 3}
    for _, i := range a {
        if i == 1 {
            p = &i
        }
    }
    fmt.Println(*p) // Prints 3

That example happens to be one where I saw some long-time Go contributors stare at the original problem and the simplified version, and even knowing there was a loop capture bug, they concluded "I don't see the bug".

You can see some other real-world non-concurrency problems in the examples I gathered here in #57173. (The context there is I had sent an improvement for the go vet loopclosure analysis for Go 1.20, and that issue is talking about possible follow-up steps).

The first two examples in the original pre-proposal discussion in #56010 also show the current semantics causing bugs even without any concurrency.

In some sense, the non-concurrency examples are arguably more insidious, including because the race detector won't usually flag them.

[markusheukelom]

@thepudds Thanks for your example.

While this proposal might fix the immediate bug in your example, it potentially leads to much more obscure bugs later in the application.

Consider:

var p *int
a := []int{0, 1, 2, 3}
for _, i := range a {  // ! assume i is created every iteration, as per the proposal
    if i == 1 {
          p = &i
    }
}
fmt.Println(*p) // prints "1" correctly now

// let's update the entry in the slice 
*p = 99
fmt.Println(a) // prints [0, 1, 2, 3], instead of the intended [0, 99, 2, 3]

This is especially true in your original (non-simplified) example: if someone takes a pointer to a some value, it's likely because it is expected that the value to be updated at some point (the other reason is efficiency, but that doesn't make sense in the provide example). However, while it seems everything works fine, knownKeys now points to copies of the db.lines entries. This will lead to potentially very hard to understand bugs later if this map is used to update keys.

In contrast, today you are forced to write something like this:

// best
a := []int{0, 1, 2, 3}
for i := range a {
    v := &a[i]
    if i == 1 {
        p = v
    }
}

// - OR -

// often seen too
a := []int{0, 1, 2, 3}
for _, i := range a {
    i := i
    if i == 1 {
        p = &i
    }
}

The former is always to be preferred; this will not have "update p somewhere a lot later" bugs. The latter works as long as p is only read. However, at least this second form makes it much more explicit that i is copied and that you are pointing to a copy of a value.

Note that he real issue here seems that it's not possible to range over the address of the values. Somethling like for _, p := &range a .

[Merovius]

@markusheukelom I don't understand the argument. The code you said is incorrect, is just as incorrect today as it will be after the proposal. The first of your correct examples is just as correct after the proposal, as it is today. The second of your correct examples is actually broken (it behaves the same as your "broken" example), again just the same today as after the proposal.

So I don't understand how your example makes a case that the proposal would lead to more subtle bugs. Exactly the same code is exactly as broken, before and after.

What am I missing?

[atdiar]

@markusheukelom I'm not too sure about the semantics you have in mind but note that in the code you wrote, I don't see how p is pointing into the slice.

It is pointing to the iteration variable. So if you were to print *p again at the end, you would just get different values.

Still in both cases, a doesn't change unless I'm mistaken.

On the topic of go.mod, it only affects libraries that are being upgraded.

Since a compiler does not check the semantics of user-code but just what can be encoded by the type system, if people rely on the old semantics tests should break (only if the old behavior is being relied upon at that). (meaning code should be tested anyway) There is also tooling to help. In fact, it doesn't really break backward compatibility. It's forward compatibility which may need work.

Given that, there is no need to worry I think.

[thepudds]

First, as a reminder for anyone reading or posting here, it's pretty easy to try the loopvar GOEXPERIMENT, and to do so without impacting your default installation of Go:

$ go install golang.org/dl/gotip@latest  # install a light wrapper (in standard GOBIN location)
$ gotip download                         # clone & compile the latest dev version (stored in ~/sdk/gotip)
$ gotip version                          # run 'gotip' instead of your normal 'go' command
$ GOEXPERIMENT=loopvar gotip test ./...  # test with loopvar experiment enabled

On Windows, the last step can be broken into set GOEXPERIMENT=loopvar then gotip test ./... or similar.

---

@markusheukelom, just to make some of the feedback on your example more concrete.

If we compare Go 1.20 vs. gotip with GOEXPERIMENT=loopvar for your example from above in https://github.com/golang/go/issues/60078#issuecomment-1554769603, it prints the following (with some brief comments added by me):

$ go1.20.4 run .
3                 // surprising to many
[0 1 2 3]         // p never points into slice, so *p = 99 does not update slice

vs.

$ GOEXPERIMENT=loopvar gotip run .
1                  // improvement over Go 1.20
[0 1 2 3]          // same as Go 1.20

The 3 vs. 1 output is basically the example from CL 40937.

The slice output of [0 1 2 3] is the same, so you might have had a mistake in the "try to update the slice" part of your example, or I might have misunderstood what you were trying to show.

[DmitriyMV]

Despite seemingly good reasoning you are about to break the compatibility promise where clearly stated that "It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification." Such decisions open a Pandora's box of random "language improvements" with unpredictable fallout.

Those were broke before, for correctness reasons. And most likely will be done again.

[pkorotkov]

Those were broke before, for correctness reasons. And most likely will be done again.

This disallowance (and I think the others you meant) will result in compiler errors. @DmitriyMV, respectfully, it's quite different from a silent change in behavior under the question in the proposal.

[AndrewHarrisSPU]

it's quite different from a silent change in behavior under the question in the proposal.

Before behavior changes a version number has to be bumped somewhere, so I don't see this as exactly a silent change. Along the lines https://github.com/golang/go/issues/60078#issuecomment-1542809524, we could say bumping version numbers without adequate testing may or may not introduce bugs, but I'm wondering if it's more useful to assert that it definitely introduces a code quality defect.

FWIW I was web searching e.g. stackexchange or Go documentation or blogging to see if there's a concise formulation of best practices for bumping version numbers, in a way that's focused on minimizing introducing defects (not bugs, but more intangible code quality defects). IMHO go tooling generally promotes better decisions, but people (especially people new to Go) might have intuitions that don't transfer etc. I didn't find something that seemed excellent, although nothing blatantly wrong either.

[pkorotkov]

Before behavior changes a version number has to be bumped somewhere, so I don't see this as exactly a silent change.

I don't wish to appear negative but this sounds too artificial, in the course of time developers will have to upgrade their packages. Out of options, just because the Go language moves forward and its older versions get unmaintained. And that's where an ambush could be.

[markusheukelom]

@Merovius (and @atdiar)

@markusheukelom I don't understand the argument. The code you said is incorrect, is just as incorrect today as it will be after the proposal. The first of your correct examples is just as correct after the proposal, as it is today. The second of your correct examples is actually broken (it behaves the same as your "broken" example), again just the same today as after the proposal.

So I don't understand how your example makes a case that the proposal would lead to more subtle bugs.

Because with the new proposal you may be more easily fooled into thinking that

a := []int{0, 1, 2, 3}
var *p int
for _, i := range a {
    if i == 1 {
        p = &i
    }
}

is actually a good implementation, not realising that p does not point into a but to a copy of a[1] instead. (Note that this example is extremely simplified; in the real world it is a lot more noisy, see above).

You are possibly more easily fooled, because today this does not work at all: both reading p and writing p will give wrong or unintended results. With the new proposal however, the code above works as long as you only read p. But if somewhere far away from this loop p is updated, this is not reflected back in the slice. This is more subtle because this "update-p-bug" may only manifest itself when a is persisted for example, not persisting the update value. This can be far away from this loop, both in code and in time.

Instead, what the programmer likely intended to do is:

a := []int{0, 1, 2, 3}
var *p int
for i := range a {
        v := &a[i]
    if i == 1 {
        p = &v
    }
}

Now p can be used to update the slice element.

My argument is that the real issue in the example is that you actually just want an easier / more natural way to ge a pointer to the slice element, instead of a per-iteration variable copy of the element.

I do agree however, that technically speaking the proposal fixes the bug in the example.

I hope this clarifies my point a bit.

[zephyrtronium]

@markusheukelom See also #21537 and https://github.com/golang/go/discussions/56010#discussioncomment-3824489.

[DmitriyMV]

I don't wish to appear negative but this sounds too artificial, in the course of time developers will have to upgrade their packages.

You have to maintain your software somehow - Go already goes to a very serious lengths for supporting old codebases. This feature will not change that, in fact it even solidifies the current approach of slow evolution. But at the same time, the language either evolves and tries to fix old mistakes, becomes bloated with different ways of doing one thing, or people start to leave. I don't think we want C++ approach, of preserving old ways of doing things, while adding new - this results in a incomprehensible code.

[Merovius]

@markusheukelom I find that a thin presupposition, to be honest. It seems to me that it is chaining someone misunderstanding the language in several basic ways with writing relatively uncommon code and verifying that with a pretty niche way to double-check their assumption (by printing the value a pointer points at, instead of the pointer itself - or writing a test for behavior). Which is not to say it can't happen, but to me at least, that can't really offset the kind of real-world data and experience we have about the bugs caused by the current behavior.

My argument is that the real issue in the example is that you actually just want an easier / more natural way to ge a pointer to the slice element, instead of a per-iteration variable copy of the element.

Even if that was true (I'm always skeptical of talking about "the real cause" of a misunderstanding), this particular example is only one example of many. And most of those happen around closures, which definitely do not make this assumption.

@DmitriyMV @pkorotkov @AndrewHarrisSPU Personally, I don't think any of these are really debatable:

1. This proposal breaks the compatibility promise. Using the go directive in go.mod somewhat mitigates that (because the compiler can use the old behavior if the code was written under old assumptions) but there was Go before modules, therefore it definitely does break the compatibility promise.
2. We decided that we are okay to break the compatibility promise in a process dubbed "Go 2", as long as it happens in a well-defined set of ways - namely, it must be a "removal", not a "redefinition", so that the compiler can report a breakage.
3. This proposal violates the rules set there as well, by being a redefinition. It's literally the example given there for such a change.
4. The proposal text mentions all of this. It also clearly states that this is a one-time exception.

In my opinion, it is neither helpful to mince words and pretend that this proposal does not break promises we set. Nor is it helpful to belabor that point. The people in favor of this proposal are aware of this breakage. You won't convince them that this proposal should not be adopted by continuing to talk about it.

If you are worried about the precedent this proposal would set, the way to go about it is to point anyone who, in the future, tries to make the argument "but we redefined loop semantics" at that last point, to definitively say that this can't be used as a precedent for future redefinitions. We would only be adopting this proposal under the assumption that it doesn't set a precedent, so it would be an oxymoron to use it that way.