fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

[ilinum]

Hi!

In our project, we use the gozstd library to compress/decompress records. We ran into the issue where Go completely crashes the program about once in 50 million calls to decompress. Here's one stack trace:

runtime: pointer 0xc007e64a76 to unallocated span span.base()=0xc007dd8000 span.limit=0xc007ed7e00 span.state=3
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x51453e9, 0x3e)
        /usr/local/go/src/runtime/panic.go:617 +0x72 fp=0x7f05f5bd5c88 sp=0x7f05f5bd5c58 pc=0x4e2e42
runtime.findObject(0xc007e64a76, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/runtime/mbitmap.go:397 +0x3bd fp=0x7f05f5bd5cd8 sp=0x7f05f5bd5c88 pc=0x4c9d5d
runtime.wbBufFlush1(0xc0000a4500)
        /usr/local/go/src/runtime/mwbbuf.go:288 +0xae fp=0x7f05f5bd5d30 sp=0x7f05f5bd5cd8 pc=0x4df32e
runtime.wbBufFlush.func1()
        /usr/local/go/src/runtime/mwbbuf.go:218 +0x3b fp=0x7f05f5bd5d48 sp=0x7f05f5bd5d30 pc=0x50e63b
runtime.systemstack(0xc000001380)
        /usr/local/go/src/runtime/asm_amd64.s:351 +0x66 fp=0x7f05f5bd5d50 sp=0x7f05f5bd5d48 pc=0x510486
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1153 fp=0x7f05f5bd5d58 sp=0x7f05f5bd5d50 pc=0x4e73f0

goroutine 547151 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc001a93020 sp=0xc001a93018 pc=0x510410
runtime.wbBufFlush(0xc00ce01290, 0xc00617d380)
        /usr/local/go/src/runtime/mwbbuf.go:206 +0x66 fp=0xc001a93050 sp=0xc001a93020 pc=0x4df1f6
runtime.gcWriteBarrier(0x4b168e0, 0xc00ce01290, 0x599a2b, 0xc00123b110, 0x0, 0x0, 0xc00123b110, 0x502028, 0x17d72e615fba, 0xc00ccdc0f3, ...)
        /usr/local/go/src/runtime/asm_amd64.s:1424 +0xb4 fp=0xc001a930d8 sp=0xc001a93050 pc=0x5125a4
bytes.NewBuffer(...)
        /usr/local/go/src/bytes/buffer.go:451

goroutine 546998 [runnable]:
runtime.cgocall(0x4089750, 0xc006645060, 0xc000096500)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc006645030 sp=0xc006644ff8 pc=0x4b826b
github.com/valyala/gozstd._Cfunc_ZSTD_initDStream_usingDDict(0x7f05e0000c80, 0x7f05d0001f80, 0x5)
        bazel-out/k8-opt/bin/go/src/github.com/valyala/gozstd/1.12/_cgo_gotypes.go:610 +0x4e fp=0xc006645060 sp=0xc006645030 pc=0x1832a7e
github.com/valyala/gozstd.initDStream.func1(0x7f05e0000c80, 0x7f05d0001f80, 0xc0066450c0)
        go/src/github.com/valyala/gozstd/reader.go:95 +0x8f fp=0xc006645098 sp=0xc006645060 pc=0x1835baf
github.com/valyala/gozstd.initDStream(0x7f05e0000c80, 0xc000384d08)
        go/src/github.com/valyala/gozstd/reader.go:95 +0x3d fp=0xc0066450c0 sp=0xc006645098 pc=0x1833d3d
github.com/valyala/gozstd.NewReaderDict(0x5736bc0, 0xc0142a2ab0, 0xc000384d08, 0xc00125c450)
        go/src/github.com/valyala/gozstd/reader.go:50 +0x3e fp=0xc006645110 sp=0xc0066450c0 pc=0x1833a2e

I got a few stack traces. The common theme with the stack traces is the decompression using the gozstd library as well as a call to runtime.gcWriteBarrier (can be in an unrelated goroutine).

I am kind of at a loss about how to debug this, happy to try something/add logs if you have suggestions.

Go version: 1.12 gozstd version: caebc4659becc90614b8d281a497fd08785b92ff Platform: linux

[valyala]

The caebc4659becc90614b8d281a497fd08785b92ff (v1.6.0) is quite old. Could you update to v1.6.4 and see whether this fixes the issue?

[ilinum]

The caebc46 (v1.6.0) is quite old. Could you update to v1.6.4 and see whether this fixes the issue?

Sounds good. Will try it this week or next week and let you know.

[ilinum]

Upgrading seems to have helped! Thanks!