sonic.Marshal Incidental panic msg "runtime error: growslice: cap out of range"

GoneWithWind1 commented 10 months ago

高并发场景下使用sonic出现growslice: cap out of range的panic，详细报错

github.com/bytedance/sonic/internal/encoder.encodeTypedPointer(0xc0872996b0, 0x18da580, 0xc087ed3eb8, 0x7fba581efa68?, 0x0)
    /root/go/pkg/mod/github.com/bytedance/sonic@v1.9.1/internal/encoder/primitives.go:83 +0x1c5
github.com/bytedance/sonic/internal/encoder.encodeInto(0xc0872996b0, {0x18da580, 0xc087edd680}, 0xc0867ac320?)
    /root/go/pkg/mod/github.com/bytedance/sonic@v1.9.1/internal/encoder/encoder.go:211 +0xdf
github.com/bytedance/sonic/internal/encoder.Encode({0x18da580, 0xc087edd680}, 0xc087edd680?)
    /root/go/pkg/mod/github.com/bytedance/sonic@v1.9.1/internal/encoder/encoder.go:168 +0xcc
github.com/bytedance/sonic.frozenConfig.Marshal(...)
    /root/go/pkg/mod/github.com/bytedance/sonic@v1.9.1/sonic.go:83
github.com/bytedance/sonic.Marshal(...)
    /root/go/pkg/mod/github.com/bytedance/sonic@v1.9.1/api.go:148

liuq19 commented 10 months ago

能否提供可以复现的完整代码和环境参数

GoneWithWind1 commented 10 months ago

能否提供可以复现的完整代码和环境参数 @liuq19 复现场景是高并发和超大结构体转换时出现问题，完整代码不太方便贴出来给出部分代码如下： func Map2Struct(mapData interface{}, data interface{}) error { if tools.IsEmpty(mapData) { return errors.New("mapData empty") } if reflect.Ptr != reflect.TypeOf(data).Kind() { return errors.New("data is not ptr") } strInfo, err := sonic.Marshal(mapData) if err != nil { return err } err = sonic.Unmarshal(strInfo, data) return err }，panic报错为sonic.Marshal这一行，定位到底层包encodeTypedPointer函数的err := fn(buf, unsafe.Pointer(vp), sb, fv)这行， go env参数： GO111MODULE="on" GOARCH="amd64" GOBIN="/home/work/go/bin" GOCACHE="/home/work/.cache/go-build" GOENV="/home/work/.config/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/work/go/pkg/mod" GONOSUMDB="*" GOOS="linux" GOPATH="/home/work/go" GOROOT="/home/work/sdk/go1.20.7" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/home/work/sdk/go1.20.7/pkg/tool/linux_amd64" GOVCS="" GOVERSION="go1.20.7" GCCGO="gccgo" GOAMD64="v1" AR="ar" CC="/opt/compiler/gcc-8.2/bin/gcc" CXX="/opt/compiler/gcc-8.2/bin/g++" CGO_ENABLED="1" GOMOD="/dev/null" GOWORK="" CGO_CFLAGS="-O2 -g" CGO_CPPFLAGS="" CGO_CXXFLAGS="-O2 -g" CGO_FFLAGS="-O2 -g" CGO_LDFLAGS="-O2 -g" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3757990476=/tmp/go-build -gno-record-gcc-switches" 个人猜测可能和内存管理有关，在结构体小的高并发场景下不复现

AsterDY commented 10 months ago

更新到1.10.2试试

GoneWithWind1 commented 10 months ago

更新到1.10.2试试

@AsterDY 请问1.9.1->1.10.2是有做这部分代码的优化么？

AsterDY commented 9 months ago

可能有gc相关的问题。请问有解决吗？@GoneWithWind1

bytedance / sonic

sonic.Marshal Incidental panic msg "runtime error: growslice: cap out of range" #536