tests (randomly?) crash with go 1.12 beta2

[decathorpe]

I've had no other issues with go 1.12, but the test suite for this package now fails on three out of six of fedora's supported architectures (ppc64le, aarch64, s390x). x86_64, i686, and armv7hl continue to work fine.

This is the crash output from the ppc64le run (from go test) for the 1.3.0 release:

unexpected fault address 0x7fffb449e94b
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7fffb449e94b pc=0x10c3f7590]
goroutine 52 [running]:
runtime.throw(0x10c40d701, 0x5)
    /usr/lib/golang/src/runtime/panic.go:617 +0x68 fp=0xc0000af6e0 sp=0xc0000af6a0 pc=0x10c1b80b8
runtime.sigpanic()
    /usr/lib/golang/src/runtime/signal_unix.go:397 +0x464 fp=0xc0000af720 sp=0xc0000af6e0 pc=0x10c1d0954
github.com/oschwald/maxminddb-golang.(*decoder).decodeCtrlData(0xc0000afc10, 0x3349, 0x10c550400, 0xc00044e000, 0xc000446000, 0xc000466000, 0x19)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:62 +0x50 fp=0xc0000af7b8 sp=0xc0000af740 pc=0x10c3f7590
github.com/oschwald/maxminddb-golang.(*decoder).decodeKey(0xc0000afc10, 0x3349, 0x15, 0x10c53f840, 0xc000446000, 0x98, 0x10c54ee40, 0xc000466000)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:673 +0x4c fp=0xc0000af840 sp=0xc0000af7b8 pc=0x10c3fc0fc
github.com/oschwald/maxminddb-golang.(*decoder).decodeMap(0xc0000afc10, 0x2, 0x3349, 0x10c550400, 0xc00044e000, 0x15, 0x1, 0x3325, 0x10c3f83cc, 0x0)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:484 +0x120 fp=0xc0000af908 sp=0xc0000af840 pc=0x10c3fb120
github.com/oschwald/maxminddb-golang.(*decoder).unmarshalMap(0xc0000afc10, 0x2, 0x3326, 0x10c54ee40, 0xc0004479a0, 0x194, 0x1, 0x2, 0x3326, 0x0)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:324 +0x320 fp=0xc0000afa20 sp=0xc0000af908 pc=0x10c3f9b30
github.com/oschwald/maxminddb-golang.(*decoder).decodeFromType(0xc0000afc10, 0x7, 0x2, 0x3326, 0x10c531700, 0xc0004479a0, 0x16, 0x1, 0x1, 0xd34263aa77ce097, ...)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:123 +0x4a0 fp=0xc0000afab0 sp=0xc0000afa20 pc=0x10c3f7d80
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc0000afc10, 0x3325, 0x10c531700, 0xc0004479a0, 0x16, 0x0, 0x3325, 0x0, 0x0)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:54 +0xd4 fp=0xc0000afb48 sp=0xc0000afab0 pc=0x10c3f7344
github.com/oschwald/maxminddb-golang.(*verifier).verifyDataSection(0xc0000afd48, 0xc000388f90, 0x0, 0x0)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:137 +0x178 fp=0xc0000afcc8 sp=0xc0000afb48 pc=0x10c3ff4c8
github.com/oschwald/maxminddb-golang.(*verifier).verifyDatabase(0xc0000afd48, 0x0, 0x0)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:94 +0xa0 fp=0xc0000afd10 sp=0xc0000afcc8 pc=0x10c3fef70
github.com/oschwald/maxminddb-golang.(*Reader).Verify(0xc0002d8aa0, 0xc0002c8200, 0x0)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:18 +0x80 fp=0xc0000afd50 sp=0xc0000afd10 pc=0x10c3fe670
github.com/oschwald/maxminddb-golang.TestVerifyOnGoodDatabases(0xc0002c8200)
    /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier_test.go:35 +0x114 fp=0xc0000aff70 sp=0xc0000afd50 pc=0x10c40a004
testing.tRunner(0xc0002c8200, 0x10c5ae540)
    /usr/lib/golang/src/testing/testing.go:862 +0xdc fp=0xc0000affb0 sp=0xc0000aff70 pc=0x10c26506c
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc0000affb0 sp=0xc0000affb0 pc=0x10c1eab24
created by testing.(*T).Run
    /usr/lib/golang/src/testing/testing.go:913 +0x304
goroutine 1 [chan receive]:
testing.(*T).Run(0xc0002c8200, 0x10c413d0f, 0x19, 0x10c5ae540, 0x10c777f01)
    /usr/lib/golang/src/testing/testing.go:914 +0x320
testing.runTests.func1(0xc000162000)
    /usr/lib/golang/src/testing/testing.go:1154 +0x8c
testing.tRunner(0xc000162000, 0xc0000addd8)
    /usr/lib/golang/src/testing/testing.go:862 +0xdc
testing.runTests(0xc00009c3a0, 0x10c777740, 0x23, 0x23, 0x0)
    /usr/lib/golang/src/testing/testing.go:1152 +0x2a0
testing.(*M).Run(0xc000160000, 0x0)
    /usr/lib/golang/src/testing/testing.go:1069 +0x174
main.main()
    _testmain.go:120 +0x150
exit status 2
FAIL    github.com/oschwald/maxminddb-golang    0.049s

If I can provide any other useful information, please let me know.

[oschwald]

That is very strange. The line in question seems to be accessing a slice. I could understand an index out of bounds error, but I am not sure how the above panic could be an issue with this code. There is no use of unsafe when running on those platforms. Unfortunately, I don't have access to those platforms. Do these tests work fine on 1.11? What about tip?

[decathorpe]

I agree, the errors are a bit strange. It might as well be a bug in the go 1.12 beta release. I only have limited access to those machines.

However, it looks like the crash doesn't happen consistently, as a recent run also crashed with this error on x86_64, so it's probably not dependent on the architecture.

[oschwald]

https://github.com/containerd/containerd/issues/3005 sounds similar. I suspect this is a bug in 1.12 and probably should be reported against Go itself.

[decathorpe]

Yeah, that might very well be the same issue, the crash looks really similar. Since people are already aware of it, I expect this to get fixed before go 1.12 stable is released.

I'll update this issue once I know more. Thanks for your help!

[oschwald]

It looks like 1.12rc1 is out. I'd be curious to know if you still see this there.

[oschwald]

Assuming this is the same issue seen in containerd, https://github.com/golang/go/issues/30283 was reported against Go.

[decathorpe]

Yep, looks like it's the same problem.

Thanks for keeping an eye on this issue!

[oschwald]

@decathorpe, are you able to reproduce this with Go 1.12 now that it is released? It looks like the issue with the compiler was fixed.

[decathorpe]

go 1.12 final isn't available on fedora yet, which means I can't test it on the problematic architectures - yet.

I'll report back once I can check if it fixes this issue. Thanks!

[oschwald]

It turns out that Travis supports ppc64le. I've added it to the build configuration and it looks like it is ok on 1.12. I'll close this issue but please let me know if you still see it.

[decathorpe]

Ah, great. Thanks! If there's anything else, I'll leave a comment.

[decathorpe]

This is strange. I've checked back since 1.12 is in fedora now, and it looks like it doesn't fix the issue. In fact, the segfault started to happen on every supported architecture (including amd64) at some point in time.

You can see the error at the bottom of the build log here: https://kojipkgs.fedoraproject.org/work/tasks/2729/33122729/build.log (this one is for amd64)

[oschwald]

That is strange. I am on 1.12 with amd64 locally and I haven't been able to reproduce it. I was able to reproduce it with a pre-1.12 release on ppc64le. Do you have specific steps I could take to reliably reproduce this?

[decathorpe]

That's the command line that's used for executing unit tests in fedora builds:

go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''

Immediately following this, the crash happens:

unexpected fault address 0x7f4cb0829cd6
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f4cb0829cd6 pc=0x55e0b7c1d211]

Since you can't reproduce this, I suspect it's due to outdated dependencies ...