cmd/cgo/internal/test: consistently failing with `segmentation fault` on `linux-arm-aws` since CL 545277

[bcmills]

Go version

b2efd1de97402ec4b8fb4e9e0ec29c8e49e8e200

What operating system and processor architecture are you using (go env)?

linux-arm-aws at b2efd1de97402ec4b8fb4e9e0ec29c8e49e8e200

##### Test execution environment.
# GOARCH: arm
# CPU: ARMv8 Processor rev 1 (v8l)
# GOOS: linux
# OS Version: Linux 4.19.0-12-arm64 #1 SMP Debian 4.19.152-1 (2020-10-18) armv8l

What did you do?

https://build.golang.org/log/4023a6ee5a18667fe161b4b917a1a0c4579527d8

What did you expect to see?

Passing tests.

What did you see instead?

##### Testing cgo
ok      cmd/cgo/internal/test   0.916s
ok      cmd/cgo/internal/test   0.880s
ok      cmd/cgo/internal/testtls    0.005s
ok      cmd/cgo/internal/testtls    0.003s
ok      cmd/cgo/internal/testnocgo  0.004s
ok      cmd/cgo/internal/testnocgo  0.003s
signal: segmentation fault
FAIL    cmd/cgo/internal/test   0.521s

[dmitshur]

CC @mknyszek, @golang/compiler.

[mknyszek]

This doesn't make any sense to me... That CL should be totally unrelated to this test. I'll investigate.

[mknyszek]

I can't reproduce by running the tests cmd/cgo/internal/test tests directly via go tool dist test on a gomote. I tried all variants, including the ones that fall under the ##### Testing cgo heading. Trying run.bash.

[mknyszek]

Yep, I can't reproduce at all on a linux-arm-aws gomote. I do not know how to proceed here. :(

[cherrymui]

This reminds me #61514, which I also don't understand and it magically stopped...

[mknyszek]

Yeah, that seems like exactly the same issue to me.

[bcmills]

Possibly something relating to code or data alignment changing in the compiled binary? (I'm thinking of something along the general lines of #60825.)

[cherrymui]

It is possible that this triggers some bug that is very sensitive to how the binary is laid out. For #61514, I couldn't reproduce on gomote or anything other than pre-/post-submit builds, and the test doesn't print anything besides segmentation fault, which makes debugging very hard, though.

[bcmills]

@mknyszek, looking back at successful runs (https://build.golang.org/log/4cad61d2d6f9b3b16aa53609780318e36b984a8c) to work around #51050, I see some environment variables that might be relevant: "GOARM=6" "GOHOSTARCH=arm" "CGO_CFLAGS=-march=armv6" "CGO_LDFLAGS=-march=armv6" (set from here).

Maybe the difficultly reproducing the problem with go tool dist test comes down to an environment setting?

[mknyszek]

Explicitly setting the environment variables didn't help. IIRC, the buildlet does set them correctly for the gomote anyway.

Here was what I ran:

gomote run -e GOARM=6 -e GOHOSTARCH=arm -e CGO_CFLAGS=-march=armv6 -e CGO_LDFLAGS=-march=armv6 go/bin/go tool dist test cmd/cgo/internal/test:static

I also tried the other cmd/cgo/internal/test test variants and run.bash. I then made sure make.bash was run with these flags set and then tried the tests again. No dice. :(

[mknyszek]

Aaaaaand it magically stopped happening again...

[bcmills]

Ah! The failing invocation was the third invocation of cmd/cgo/internal/test, which I think makes it this one (it's also the first one after testnocgo): https://cs.opensource.google/go/go/+/master:src/cmd/dist/test.go;l=1228;drc=d7fcb5cf80953f1d63246f1ae9defa60c5ce2d76

That would be -buildmode=static -ldflags='-linkmode=external -extldflags "-static -pthread"', I think?

[bcmills]

Or go tool dist test cmd/cgo/internal/test:static.

[bcmills]

Ah, which I see is what you already tried. 😞

[cherrymui]

As it magically passes now, we still want to investigate it but it is not currently a release blocker.