search

pinpoint-apm / pinpoint-go-agent

Pinpoint Go agent
Apache License 2.0
46 stars 15 forks source link

pinpoint + plugin/sarama asyncProducer occurs panic #103

Open rlaxodbs0 opened 1 week ago

rlaxodbs0 commented 1 week ago

version: pinpoint v1.4.2, plugin/sarama v1.4.1

I use asyncProducer tracing of the sarama plugin. When large messages produce, panic occurs intermittently.

Has this case ever been mentioned? I wonder others have experienced this too

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x47d196]

goroutine 30541704 [running]:
google.golang.org/protobuf/encoding/protowire.AppendString(...)
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/encoding/protowire/wire.go:455
google.golang.org/protobuf/internal/impl.appendStringNoZeroValidateUTF8({0xc000e817a0?, 0x416c2b?, 0x7f2300b600f8?}, {0xc000341910?}, 0xc0003418f0?, {0x3?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:5113 +0x99
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc0002a02c0, {0xc000e817a0?, 0xc000e817a0?, 0x1418d00?}, {0xc0000f6f08?}, {0x52?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:145 +0x13f
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc0000f6f20?, {{}, {0xe9e8c0, 0xc000341910}, {0xc000e817a0, 0xde, 0x112}, 0x2})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:113 +0x85
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0xd8?, 0x5b?, 0xea?}, {0xc000e817a0, 0xde, 0x112}, {0xe9e8c0, 0xc000341910})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:194 +0x35d
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0xa0?, 0x17?, 0xe8?}, {0xc000e817a0, 0xde, 0x112}, {0xe8eac0?, 0xc000341910?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:158 +0x73
google.golang.org/protobuf/internal/impl.appendMessage({0xc000e817a0, 0xdc, 0x112}, {0xe8eac0, 0xc000341910}, 0xa, {0xd8?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:281 +0xb8
google.golang.org/protobuf/internal/impl.makeMessageFieldCoder.func2({0xc000e817a0, 0xdc, 0x112}, {0x416965?}, 0xc0003e2510, {0xc8?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:213 +0x73
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOneofFieldCoders.func4({0xc000e817a0, 0xdc, 0x112}, {0xc000341900?}, 0xc007801c80?, {0xc0?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:96 +0x5d
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc0002a0160, {0xc000e817a0?, 0xc0003418e0?, 0x1418d00?}, {0xc0000f7218?}, {0x52?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:145 +0x13f
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc0000f7230?, {{}, {0xe9e8c0, 0xc000341900}, {0xc000e817a0, 0xdc, 0x112}, 0x2})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:113 +0x85
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0xd8?, 0x5b?, 0xea?}, {0xc000e817a0, 0xdc, 0x112}, {0xe9e8c0, 0xc000341900})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:194 +0x35d
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0xa0?, 0x17?, 0xe8?}, {0xc000e817a0, 0xdc, 0x112}, {0xe8eac0?, 0xc000341900?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:158 +0x73
google.golang.org/protobuf/internal/impl.appendMessage({0xc000e817a0, 0xda, 0x112}, {0xe8eac0, 0xc000341900}, 0x62, {0x4?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:281 +0xb8
google.golang.org/protobuf/internal/impl.makeMessageFieldCoder.func2({0xc000e817a0, 0xda, 0x112}, {0x22?}, 0xc00020c548, {0x50?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:213 +0x73
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc0002a0000, {0xc000e817a0?, 0x0?, 0xc0003418d0?}, {0xc0000f74e0?}, {0x52?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:145 +0x13f
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc000213201?, {{}, {0xe9e8c0, 0xc0003418d0}, {0xc000e817a0, 0xb8, 0x112}, 0x2})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:113 +0x85
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x50?, 0x81?, 0xa5?}, {0xc000e817a0, 0xb8, 0x112}, {0xe9e8c0, 0xc0003418d0})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:194 +0x35d
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0xa0?, 0x17?, 0xe8?}, {0xc000e817a0, 0xb8, 0x112}, {0xe8eac0?, 0xc0003418d0?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:158 +0x73
google.golang.org/protobuf/internal/impl.appendMessageSlice({0xc000e817a0?, 0x42?, 0x112?}, {0xe8eac0?}, 0x6a, {0xea5bd8, 0xd2a7c0}, {0x59?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:561 +0x191
google.golang.org/protobuf/internal/impl.makeMessageSliceFieldCoder.func2({0xc000e817a0?, 0x42?, 0x112?}, {0xc000341890?}, 0xc00178ff48?, {0x40?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:470 +0x2a
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc000253760, {0xc000e817a0?, 0xc000341840?, 0x1418d00?}, {0xc0000f77e8?}, {0x52?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:145 +0x13f
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc0000f7800?, {{}, {0xe9e8c0, 0xc000341890}, {0xc000e817a0, 0x3, 0x112}, 0x2})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:113 +0x85
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0xd8?, 0x5b?, 0xea?}, {0xc000e817a0, 0x3, 0x112}, {0xe9e8c0, 0xc000341890})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:194 +0x35d
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0xa0?, 0x17?, 0xe8?}, {0xc000e817a0, 0x3, 0x112}, {0xe8eac0?, 0xc000341890?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:158 +0x73
google.golang.org/protobuf/internal/impl.appendMessage({0xc000e817a0, 0x0, 0x112}, {0xe8eac0, 0xc000341890}, 0xa, {0xd8?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:281 +0xb8
google.golang.org/protobuf/internal/impl.makeMessageFieldCoder.func2({0xc000e817a0, 0x0, 0x112}, {0xc71500?}, 0xc000775170, {0xd8?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:213 +0x73
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOneofFieldCoders.func4({0xc000e817a0, 0x0, 0x112}, {0xc00023faa8?}, 0x47913e?, {0x0?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:96 +0x5d
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc000253600, {0xc000e817a0?, 0x120?, 0xc00023fb50?}, {0xc0000f7af8?}, {0x8?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:145 +0x13f
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0x0?, {{}, {0xe9e8c0, 0xc0003417c0}, {0xc000e817a0, 0x0, 0x112}, 0x2})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:113 +0x85
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x40?, 0xfc?, 0x23?}, {0x0, 0x0, 0x0}, {0xe9e8c0, 0xc0003417c0})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:194 +0x35d
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0x0?, 0x32?, 0xd1?}, {0x0, 0x0, 0x0}, {0xe8eac0?, 0xc0003417c0?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:158 +0x73
github.com/golang/protobuf/proto.marshalAppend({0x0, 0x0, 0x0}, {0xe92750?, 0xc0021e0a50?}, 0x0)
        /home1/irteam/tuyy/goworkspace/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0x97
github.com/golang/protobuf/proto.Marshal(...)
        /home1/irteam/tuyy/goworkspace/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23
google.golang.org/grpc/encoding/proto.codec.Marshal({}, {0xd13200?, 0xc0021e0a50?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/grpc@v1.52.0/encoding/proto/proto.go:45 +0x52
google.golang.org/grpc.encode({0x7f230006b718?, 0x1495fa0?}, {0xd13200?, 0xc0021e0a50?})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/grpc@v1.52.0/rpc_util.go:595 +0x3e
google.golang.org/grpc.prepareMsg({0xd13200?, 0xc0021e0a50?}, {0x7f230006b718?, 0x1495fa0?}, {0x0, 0x0}, {0x0, 0x0})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/grpc@v1.52.0/stream.go:1708 +0xc7
google.golang.org/grpc.(*clientStream).SendMsg(0xc00528f320, {0xd13200, 0xc0021e0a50})
        /home1/irteam/tuyy/goworkspace/pkg/mod/google.golang.org/grpc@v1.52.0/stream.go:846 +0xd7
github.com/pinpoint-apm/pinpoint-go-agent/protobuf.(*spanSendSpanClient).Send(0x0?, 0xc000129398?)
        /home1/irteam/tuyy/goworkspace/pkg/mod/github.com/pinpoint-apm/pinpoint-go-agent@v1.4.2/protobuf/Service.pb.go:110 +0x25
github.com/pinpoint-apm/pinpoint-go-agent.(*spanStream).sendSpan.func1()
        /home1/irteam/tuyy/goworkspace/pkg/mod/github.com/pinpoint-apm/pinpoint-go-agent@v1.4.2/grpc.go:743 +0x23
github.com/pinpoint-apm/pinpoint-go-agent.sendStreamWithTimeout.func1()
        /home1/irteam/tuyy/goworkspace/pkg/mod/github.com/pinpoint-apm/pinpoint-go-agent@v1.4.2/grpc.go:525 +0x23
created by github.com/pinpoint-apm/pinpoint-go-agent.sendStreamWithTimeout in goroutine 44
        /home1/irteam/tuyy/goworkspace/pkg/mod/github.com/pinpoint-apm/pinpoint-go-agent@v1.4.2/grpc.go:524 +0xa6
tuyy commented 1 week ago

Same issue here. Any update?

dwkang commented 1 week ago

Recently, the concurrent issue of sarama asyncproducer was fixed in the v1.4.3 version. I hope you test this version. https://github.com/pinpoint-apm/pinpoint-go-agent/releases/tag/v1.4.3

And, the v1.4.2 version has a memory leak issue. I recommend you upgrade to the latest version of v1.4.4 version. https://github.com/pinpoint-apm/pinpoint-go-agent/releases/tag/v1.4.4