Closed ijc closed 5 years ago
The types are semver.Version which is a struct with a handful of fields (Major, Minor etc).
For completeness here is the actual full type which is slightly more complex than I let on above:
// Version represents a semver compatible version
type Version struct {
Major uint64
Minor uint64
Patch uint64
Pre []PRVersion
Build []string //No Precedence
}
ping @bradfitz PTAL
Tentatively milestoning as 1.12 since it appears to be a recent regression.
cc @randall77 @cherrymui
Thanks!
Thanks for the bug report! If you/anyone wants to try to minimize further, setting GOGC=1
might help reproduce more reliably. (Bisection would also be super useful.)
FWIW I just tested on linux/arm64
using a binary I cross-built with the linux/amd64
toolchain described above and the issue did not occur neither with qemu-aarch64-state
nor on a real machine (I tried 10,000 iterations on the latter).
The go tool objdump
output on arm64 lacks most of the symbolic information to (easily) eyeball the code.
GOGC=1
doesn't seem to make a noticeable difference to the reproducibility.
Interesting. I'll take a careful look.
The go tool objdump output on arm64 lacks most of the symbolic information to (easily) eyeball the code.
On ARM64, many address operations are split into multiple instructions, as it cannot load a 64-bit address in a single instruction, and currently objdump is not smart enough to recombine them.
The same "volatile" stack temporary is used in two stores, both with write barriers. I think we don't generate this before, so the write barrier pass doesn't handle this case. Now we optimize X =f(); Y = X
to storing the result of f()
to both X and Y directly. I think we can fix this in the write barrier pass.
My bisect fingered 6c631ae227cb6f15d5b0b8cc39c1715b8f4e9187 "cmd/compile: in wasm, allocate approximately right number of locals for functions", but that doesn't pass the sniff test and I couldn't confirm manually. I suspect that is a misbisect (I probably got distracted and failed to test some rev but still entered a result).
I've scripted it this time and git bisect run
is underway now, although I'll have to head off soon so it might be tomorrow before I see the result.
It's not done but I really have to run. It's narrowed it down so far to:
$ git log --oneline bisect/bad...bisect/good-6c631ae227cb6f15d5b0b8cc39c1715b8f4e9187~1
3bf9b77c0c (refs/bisect/bad) encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines
7c2718b12a doc: tweak example in Effective Go
f5df0a9575 cmd/internal/xcoff: don't use io.SeekCurrent for go1.4 compatibility
035f9e8102 (HEAD) doc: update docs.html with new tour import path
ecccdccf3e misc/wasm: fix panic on os.Stdout.Sync() in the browser
2e88689168 go/types: temporarily disable a verification in Stdlib test
2578ac54eb cmd/compile: move argument stack construction to SSA generation
6c631ae227 (refs/bisect/good-6c631ae227cb6f15d5b0b8cc39c1715b8f4e9187) cmd/compile: in wasm, allocate approximately right number of locals for functions
At this point I'd probably guess at 2578ac54eb "cmd/compile: move argument stack construction to SSA generation".
Change https://golang.org/cl/168677 mentions this issue: cmd/compile: copy volatile values before emitting write barrier call
@ijc thanks for the bisects. I think CL https://golang.org/cl/168677 should fix this. You don't have to do more.
https://play.golang.org/p/5DpJ673vnSJ
On tip (700e969d5b), this fails quite reliably.
@gopherbot please backport this to Go 1.12.
Backport issue(s) opened: #30996 (for 1.12).
Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.
Change https://golang.org/cl/168817 mentions this issue: [release-branch.go1.12] cmd/compile: copy volatile values before emitting write barrier call
The bisect was automated so I may as well report the result which was that it blamed 2578ac54eb417488c70324e7f3fc25565ec3f03d "cmd/compile: move argument stack construction to SSA generation", which seems legit.
git bisect log
git bisect start # bad: [05e77d41914d247a1e7caf37d7125ccaa5a53505] [release-branch.go1.12] go1.12 git bisect bad 05e77d41914d247a1e7caf37d7125ccaa5a53505 # good: [41e62b8c49d21659b48a95216e3062032285250f] [release-branch.go1.11] runtime: mark sigInitIgnored nosplit git bisect good 41e62b8c49d21659b48a95216e3062032285250f # good: [e0faedbb5344eb6f8f704005fe88961cdc6cf5f8] cmd/go: add missing newlines in printf formats git bisect good e0faedbb5344eb6f8f704005fe88961cdc6cf5f8 # bad: [2dda040f19aa6a7551f090d8c5a3941e416b21df] cmd/compile/internal/gc: represent labels as bare Syms git bisect bad 2dda040f19aa6a7551f090d8c5a3941e416b21df # good: [9a033bf9d3f5f7485d82836ec95e51a3fa74a926] cmd/compile: optimize 386's assembly generator git bisect good 9a033bf9d3f5f7485d82836ec95e51a3fa74a926 # good: [29907b13db0455eded50263b4e37445045c82e6e] crypto: add AIX operating system git bisect good 29907b13db0455eded50263b4e37445045c82e6e # good: [2d6a7593b5cde33546be7bb69f420d32df060a47] cmd/doc: minor code simplification git bisect good 2d6a7593b5cde33546be7bb69f420d32df060a47 # bad: [6994731ec2babb6a4f2bbbb08dbe649767a25942] internal/bytealg: improve asm for memequal on ppc64x git bisect bad 6994731ec2babb6a4f2bbbb08dbe649767a25942 # skip: [e9b39417e4448d6001b2707d9cf42bba4673e9ab] go/types: copy embedded methods unchanged when completing interfaces git bisect skip e9b39417e4448d6001b2707d9cf42bba4673e9ab # bad: [ddf83eeb2359f28f81bce78a6d4521852d5e6dfe] cmd/compile: s/eqtype/types.Identical/ (fix build) git bisect bad ddf83eeb2359f28f81bce78a6d4521852d5e6dfe # good: [6c631ae227cb6f15d5b0b8cc39c1715b8f4e9187] cmd/compile: in wasm, allocate approximately right number of locals for functions git bisect good 6c631ae227cb6f15d5b0b8cc39c1715b8f4e9187 # bad: [3bf9b77c0c8f123728fcfc802599232e9bd95476] encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines git bisect bad 3bf9b77c0c8f123728fcfc802599232e9bd95476 # bad: [035f9e8102d3b46877b7462fcd365324272d1d0e] doc: update docs.html with new tour import path git bisect bad 035f9e8102d3b46877b7462fcd365324272d1d0e # bad: [2e88689168a57ae550ddae7ad0966fa14c877c5f] go/types: temporarily disable a verification in Stdlib test git bisect bad 2e88689168a57ae550ddae7ad0966fa14c877c5f # bad: [2578ac54eb417488c70324e7f3fc25565ec3f03d] cmd/compile: move argument stack construction to SSA generation git bisect bad 2578ac54eb417488c70324e7f3fc25565ec3f03d # first bad commit: [2578ac54eb417488c70324e7f3fc25565ec3f03d] cmd/compile: move argument stack construction to SSA generation
I also confirmed that at f23c601bf9f66ef4a0d9d2dcd003b95e78fb28f8 both my test case and my actual usecase are fixed. Thanks for the super fast turnaround!
Is this sort of miscompile something which might precipitate a quick turnaround on a go 1.12.2? We need to decide if we should roll back our projects to 1.11.x or wait a few days.
Is this sort of miscompile something which might precipitate a quick turnaround on a go 1.12.2? We need to decide if we should roll back our projects to 1.11.x or wait a few days.
@katiehockman and @andybons about the 1.12.2 release. (I don't know the answer.)
We try to do minor point releases on the first of the month, so the next one would be April 1 ideally.
Thanks @andybons that should help us figure out what to do.
@ALTree I see the milestone was changed from 1.12.2 to 1.13
Is a separate tracking-issue needed to track it for Go 1.12.2?
@thaJeztah That's #30996. It was opened a week ago, and that's the reason I moved this one to 1.13.
oh! I see now; I overlooked the bot's comment as it's usually commenting related to changes in Gerrit.
Apologies; still getting the hang of the bots and flow in this repository 🤗
@thaJeztah No problem. Feel free to
1) monitor that issue 2) ping Cherry on the linked CL (https://golang.org/cl/168817) in gerrit to merge the fix in the release branch
if you are worried we may forget it before the 1.12.2 release gets finalized. It happened once, unfortunately. The cherry-picked fix for an issues was ready, but it didn't get merged and the release scripts ignored it.
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
It reproduces with go 1.12.1 so I believe yes. I downloaded it from https://dl.google.com/go/go1.12.1.linux-amd64.tar.gz
I've also reproduced on
darwin/amd64
from https://golang.org/doc/install?download=go1.12.1.darwin-amd64.pkg:What operating system and processor architecture are you using (
go env
)?linux/amd64 (Debian).
go env
Output on linux/amd64I have also reproduced on darwin/amd64:
go env
Output on darwin/amd64What did you do?
I have a program which includes:
The types are
semver.Version
which is a struct with a handful of fields (Major
,Minor
etc).I am seeing corruption of
VersionLatest
where the major field is824638865200
(always that number, AFAICS) after init and not0
.Version0_2_0
is never corrupted.I have narrowed it down to this minimal case: go1.12-initvar-miscompile.zip
Note that the test program includes a massive bunch of anonymous imports, these seem to be necessary to trigger the runtime/gc load and cause the issue (they are derived from all the vendoring in my actual project which have
init()
functions). Trying to minimise the set of imports is non-linear (since the issue is non-deterministic AFAICT) and also reduces the probability of the issue occurring. I got it down to a dozen or so imports and a 1/10,000 incidence, since it is much easier to reproduce with the larger import set that is what I included here.What did you expect to see?
On success
VersionLatest
is"0.2.0"
.What did you see instead?
It is corrupted to
"824638766896.2.0"
.Analysis
After the call to
semver.MustParse
there is aruntime.writeBarrier
slow path:The issue occurs precisely when this jump is taken and is during this sequence:
On go1.11.4 the same code is compiled differently and the second copy is
memcpy(VersionLatest, Version0_2_0)
instead which does not use the corrupted stack copyBTW, the code pattern is the same in the fast pass, but that lacks the call to
runtime.typedmemmove
which overwrites the stack value, thus the issue only occurs if we manager to hit the slow path.Full
go tool objdump -s '^main.init.ializers'
Output for go 1.12.1 compilation