search

SkyAPM / go2sky

Distributed tracing and monitor SDK in Go for Apache SkyWalking APM
https://skywalking.apache.org/
Apache License 2.0
447 stars 122 forks source link

panic invalid memory address or nil pointer dereference #105

Closed Anthony-Dong closed 2 years ago

Anthony-Dong commented 3 years ago

Describe the bug 线上发现因为skywalking 上报trace出现频繁的重启

To Reproduce 程序panic信息，只有这些！确实不太好定位


panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x462c16]

goroutine 41 [running]:
google.golang.org/protobuf/encoding/protowire.AppendString(...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/encoding/protowire/wire.go:450
google.golang.org/protobuf/internal/impl.appendStringNoZeroValidateUTF8(0xc0017a4000, 0xa6, 0x12440, 0xc00152ab40, 0xc0004266c0, 0xc0014ff102, 0x458827, 0x1209760, 0x0, 0x2a46c0, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/codec_gen.go:5026 +0xf7
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc000125040, 0xc0017a4000, 0xa6, 0x12440, 0xc00152ab40, 0x1321402, 0xc0014ff170, 0xa34e3f, 0xc000125040, 0x46e989, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/encode.go:138 +0x10a
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc000125040, 0x164e700, 0xc001232480, 0xc0017a4000, 0xa6, 0x12440, 0x2, 0xc000125040, 0x164e700, 0xc001232480, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/encode.go:106 +0x8a
google.golang.org/protobuf/proto.MarshalOptions.marshal(0xc001010001, 0xc0017a4000, 0xa6, 0x12440, 0x164e700, 0xc001232480, 0xc0014ff2f8, 0x9cda7e, 0x164e700, 0xc001232480, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/proto/encode.go:163 +0x2b9
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend(0xc001010001, 0xc0017a4000, 0xa6, 0x12440, 0x16199c0, 0xc001232480, 0x12440, 0x2, 0xc0003625a0, 0xc0014ff380, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/proto/encode.go:125 +0x98
google.golang.org/protobuf/internal/impl.appendMessageSlice(0xc0017a4000, 0xa5, 0x12440, 0xc0001a4ea0, 0x62, 0x1658bc0, 0x133ada0, 0x12402, 0x2, 0xc0017a4000, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/codec_field.go:539 +0x219
google.golang.org/protobuf/internal/impl.makeMessageSliceFieldCoder.func2(0xc0017a4000, 0xa4, 0x12440, 0xc0001a4ea0, 0xc0000185d8, 0x9fef02, 0xc0017a4000, 0xa4, 0x12440, 0x0, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/codec_field.go:455 +0x85
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc0001eb2c0, 0xc0017a4000, 0x5e, 0x12440, 0xc0001a4e40, 0xc0000ddb02, 0xc0014ff510, 0xa34e3f, 0xc0001eb2c0, 0x1658bc0, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/encode.go:138 +0x10a
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc0001eb2c0, 0x164e700, 0xc001232440, 0xc0017a4000, 0x5e, 0x12440, 0x2, 0xc0001eb2c0, 0x164e700, 0xc001232440, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/encode.go:106 +0x8a
google.golang.org/protobuf/proto.MarshalOptions.marshal(0xc001010001, 0xc0017a4000, 0x5e, 0x12440, 0x164e700, 0xc001232440, 0xc0014ff698, 0x9cda7e, 0x164e700, 0xc001232440, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/proto/encode.go:163 +0x2b9
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend(0xc001010001, 0xc0017a4000, 0x5e, 0x12440, 0x16199c0, 0xc001232440, 0x12440, 0x20, 0xa49601, 0x7f6e800c84b0, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/proto/encode.go:125 +0x98
google.golang.org/protobuf/internal/impl.appendMessageSlice(0xc0017a4000, 0x5c, 0x12440, 0xc000374aa0, 0x1a, 0x1658bc0, 0x13ade40, 0x2, 0x12440, 0x20, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/codec_field.go:539 +0x219
google.golang.org/protobuf/internal/impl.makeMessageSliceFieldCoder.func2(0xc0017a4000, 0x5b, 0x12440, 0xc000374aa0, 0xc000126f10, 0x2, 0xc0017a4000, 0x5b, 0x12440, 0x0, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/codec_field.go:455 +0x85
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer(0xc0001eb040, 0xc0017a4000, 0x0, 0x12440, 0xc000374a80, 0xc00015e402, 0x10100c0014ff8b0, 0x0, 0x12440, 0xc0017a4000, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/encode.go:138 +0x10a
google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal(0xc0001eb040, 0x164e700, 0xc0012320b0, 0xc0017a4000, 0x0, 0x12440, 0x2, 0x44838c, 0x14000, 0x119e5a0, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/internal/impl/encode.go:106 +0x8a
google.golang.org/protobuf/proto.MarshalOptions.marshal(0xc001000001, 0xc000778a00, 0x0, 0x2446, 0x164e700, 0xc0012320b0, 0xc0012320b0, 0x164e700, 0xc0012320b0, 0x137cd80, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/proto/encode.go:163 +0x2b9
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend(0x1000001, 0xc000778a00, 0x0, 0x2446, 0x16199c0, 0xc0012320b0, 0x7f6eab5e3d98, 0x0, 0x0, 0xc0001de3b8, ...)
    /root/go/pkg/mod/google.golang.org/protobuf@v1.23.0/proto/encode.go:125 +0x98
github.com/golang/protobuf/proto.marshalAppend(0xc000778a00, 0x0, 0x2446, 0x1631060, 0xc000374a80, 0xc0014ffb00, 0x470b60, 0x221f700, 0xc000908100, 0x2, ...)
    /root/go/pkg/mod/github.com/golang/protobuf@v1.4.3/proto/wire.go:40 +0xbe
github.com/golang/protobuf/proto.(*Buffer).Marshal(...)
    /root/go/pkg/mod/github.com/golang/protobuf@v1.4.3/proto/buffer.go:111
google.golang.org/grpc/encoding/proto.marshal(0x137cd80, 0xc000374a80, 0xc001bb8000, 0x0, 0x0, 0xc0000dd800, 0xc0014ffc28, 0xae3d58)
    /root/go/pkg/mod/google.golang.org/grpc@v1.27.0/encoding/proto/proto.go:59 +0x111
google.golang.org/grpc/encoding/proto.codec.Marshal(0x137cd80, 0xc000374a80, 0x203000, 0xc000f181b0, 0x8d7b, 0x203000, 0x203000)
    /root/go/pkg/mod/google.golang.org/grpc@v1.27.0/encoding/proto/proto.go:74 +0xad
google.golang.org/grpc.encode(0x7f6eab5893e8, 0x233e688, 0x137cd80, 0xc000374a80, 0xc0008e03a0, 0xc00092e380, 0x7f6eab5e3d98, 0x0, 0x0)
    /root/go/pkg/mod/google.golang.org/grpc@v1.27.0/rpc_util.go:543 +0x52
google.golang.org/grpc.prepareMsg(0x137cd80, 0xc000374a80, 0x7f6eab5893e8, 0x233e688, 0x0, 0x0, 0x0, 0x0, 0xc000f181b0, 0x5, ...)
    /root/go/pkg/mod/google.golang.org/grpc@v1.27.0/stream.go:1519 +0x85
google.golang.org/grpc.(*clientStream).SendMsg(0xc00050ac60, 0x137cd80, 0xc000374a80, 0x0, 0x0)
    /root/go/pkg/mod/google.golang.org/grpc@v1.27.0/stream.go:699 +0x169
github.com/SkyAPM/go2sky/reporter/grpc/language-agent.(*traceSegmentReportServiceCollectClient).Send(0xc0002ca100, 0xc000374a80, 0x1, 0x0)
    /root/go/pkg/mod/github.com/!sky!a!p!m/go2sky@v0.5.0/reporter/grpc/language-agent/Tracing.pb.go:632 +0x49
github.com/SkyAPM/go2sky/reporter.(*gRPCReporter).initSendPipeline.func1(0xc00052c280)
    /root/go/pkg/mod/github.com/!sky!a!p!m/go2sky@v0.5.0/reporter/grpc.go:240 +0x206
created by github.com/SkyAPM/go2sky/reporter.(*gRPCReporter).initSendPipeline
    /root/go/pkg/mod/github.com/!sky!a!p!m/go2sky@v0.5.0/reporter/grpc.go:230 +0x46

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

wu-sheng commented 3 years ago

It looks like you put some null value into the span, which proto is not accepted.

Anthony-Dong commented 3 years ago

It looks like you put some null value into the span, which proto is not accepted.

但是目前使用的接口api来看，没有可能开发人员对于引入nil的操作

type Span interface {
    SetOperationName(string)
    GetOperationName() string
    SetPeer(string)
    SetSpanLayer(v3.SpanLayer)
    SetComponent(int32)
    Tag(Tag, string)
    Log(time.Time, ...string)
    Error(time.Time, ...string)
    End()
    IsEntry() bool
    IsExit() bool
}

所以不太清楚是啥原因造成的，有没有可能是因为 grpc/pb 包版本的问题

wu-sheng commented 3 years ago

Such as you put tage value as string, when it is serialized later, it could cause NULL error. But need to check proto go source codes, whether it is a panic. It is an exception only in Java.

mrproliu commented 3 years ago

Could you please use log reporter(go2sky.WithReporter(reporter.NewLogReporter())) to print segment data to the std? It uses json.Marshal to serialize the span data as JSON format, which will help you find out where has a null value. Also, I notice your protobuf version is not the same as go2sky@v.0.5.0, please have a check.

lipeiru0329 commented 2 years ago

Could I know how to solve it previously, since I meet the same problem and report the same error?

Anthony-Dong commented 2 years ago

我们当时出现问题的服务器把 skywalking 组件给下掉了，有可能是日志过大！因为我们当时会在apm中记录业务请求的日志，那个服务器日志量大且单次请求日志量也多