search

aws / aws-xray-sdk-go

AWS X-Ray SDK for the Go programming language.
Apache License 2.0
276 stars 117 forks source link

Suppress Panic in Emitter #419

Closed wangzlei closed 11 months ago

wangzlei commented 11 months ago

Issue #, if available:

The potential concurrency risk could lead to Default_Emitter.Emit() Panic during JSON marshaling. We would like to use recover() to prevent X-ray SDK crashes in the user's application before addressing the root cause.

panic: runtime error: index out of range [13] with length 13 [recovered]
    panic: runtime error: index out of range [13] with length 13

goroutine 41146499 [running]:
encoding/json.(*encodeState).marshal.func1()
    /usr/local/go/src/encoding/json/encode.go:291 +0x80
panic({0x1162f60?, 0x4001aa78a8?})
    /usr/local/go/src/runtime/panic.go:914 +0x218
encoding/json.mapEncoder.encode({0x40031e77a8?}, 0x40044c8340, {0x102d760?, 0x4001410478?, 0xffffffffffffffff?}, {0xe?, 0x0?})
    /usr/local/go/src/encoding/json/encode.go:745 +0x5b0
encoding/json.structEncoder.encode({{{0x4002bd6000, 0x19, 0x23}, 0x4000521d10, 0x4000521ec0}}, 0x40044c8340, {0x1205d60?, 0x4001410340?, 0x4002bd6000?}, {0x19?, ...})
    /usr/local/go/src/encoding/json/encode.go:706 +0x190
encoding/json.ptrEncoder.encode({0x40031e79c8?}, 0x40044c8340, {0x11f52e0?, 0x4001410340?, 0x40031e79d8?}, {0x60?, 0x72?})
    /usr/local/go/src/encoding/json/encode.go:878 +0x1c4
encoding/json.(*encodeState).reflectValue(0x40031e7a78?, {0x11f52e0?, 0x4001410340?, 0x1258e0?}, {0x40?, 0x65?})
    /usr/local/go/src/encoding/json/encode.go:323 +0x70
encoding/json.(*encodeState).marshal(0x40031e7b28?, {0x11f52e0?, 0x4001410340?}, {0xb4?, 0x8?})
    /usr/local/go/src/encoding/json/encode.go:295 +0xb8
encoding/json.Marshal({0x11f52e0, 0x4001410340})
    /usr/local/go/src/encoding/json/encode.go:162 +0x94
github.com/aws/aws-xray-sdk-go/xray.(*DefaultStreamingStrategy).StreamCompletedSubsegments(0x40031e7d38?, 0x4001410000)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_streaming_strategy.go:74 +0x2b4
github.com/aws/aws-xray-sdk-go/xray.packSegments.func1(0x4001410000)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_emitter.go:97 +0xb4
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x400281f040, {0x4002767500, 0x6, 0x8})
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_emitter.go:110 +0xcc
github.com/aws/aws-xray-sdk-go/xray.(*DefaultEmitter).Emit(0x40006bc168, 0x18a44?)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_emitter.go:66 +0x80
github.com/aws/aws-xray-sdk-go/xray.(*Segment).emit(0x4002f12fc0?)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:468 +0x84
github.com/aws/aws-xray-sdk-go/xray.(*Segment).flush(0x1e89f98?)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:510 +0x224
github.com/aws/aws-xray-sdk-go/xray.(*Segment).send(0x400281f040)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:490 +0x60
github.com/aws/aws-xray-sdk-go/xray.(*Segment).handleContextDone(0x400281f040)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:476 +0x54
github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling.func1()
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:152 +0x40
created by github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling in goroutine 40912008
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:150 +0xd2c
fatal error: concurrent map iteration and map write

goroutine 25931611 [running]:
reflect.mapiterinit(0x200?, 0x10e40e0?, 0x40021cd578?)
    /usr/local/go/src/runtime/map.go:1392 +0x1c
reflect.(*MapIter).Next(0x102d760?)
    /usr/local/go/src/reflect/value.go:1935 +0x60
encoding/json.mapEncoder.encode({0x40021cd7e8?}, 0x40025153c0, {0x102d760?, 0x4003cd2138?, 0x40021cd8a8?}, {0xe?, 0x0?})
    /usr/local/go/src/encoding/json/encode.go:744 +0x28c
encoding/json.structEncoder.encode({{{0x4000772000, 0x19, 0x23}, 0x4000512de0, 0x4000512e10}}, 0x40025153c0, {0x1205d60?, 0x4003cd2000?, 0x40021cd908?}, {0x28?, ...})
    /usr/local/go/src/encoding/json/encode.go:706 +0x190
encoding/json.ptrEncoder.encode({0x40021cda08?}, 0x40025153c0, {0x11f52e0?, 0x4003cd2000?, 0x40021cda18?}, {0x7c?, 0xf0?})
    /usr/local/go/src/encoding/json/encode.go:878 +0x1c4
encoding/json.(*encodeState).reflectValue(0x40021cdab8?, {0x11f52e0?, 0x4003cd2000?, 0x1258e0?}, {0x40?, 0x65?})
    /usr/local/go/src/encoding/json/encode.go:323 +0x70
encoding/json.(*encodeState).marshal(0x0?, {0x11f52e0?, 0x4003cd2000?}, {0xc8?, 0xf6?})
    /usr/local/go/src/encoding/json/encode.go:295 +0xb8
encoding/json.Marshal({0x11f52e0, 0x4003cd2000})
    /usr/local/go/src/encoding/json/encode.go:162 +0x94
github.com/aws/aws-xray-sdk-go/xray.packSegments.func1(0x4003cd2000)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_emitter.go:100 +0x1a4
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x4000e03860, {0x400361f200, 0x5, 0x8})
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_emitter.go:110 +0xcc
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x400276d040, {0x400361f200, 0x5, 0x8})
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_emitter.go:109 +0xb0
github.com/aws/aws-xray-sdk-go/xray.(*DefaultEmitter).Emit(0x40007021c8, 0x18a44?)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/default_emitter.go:66 +0x80
github.com/aws/aws-xray-sdk-go/xray.(*Segment).emit(0x40029fe480?)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:468 +0x84
github.com/aws/aws-xray-sdk-go/xray.(*Segment).flush(0x50cf98?)
    /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.1/xray/segment.go:510 +0x224

Description of changes:

Adding defer recover() in Default_Emitter.Emit() function.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

hxy1991 commented 5 months ago

@wangzlei @srprash

hi all, fatal error 2 still occurs in 1.8.2

fatal error: concurrent map iteration and map write

goroutine 3216017 [running]:
reflect.mapiterinit(0x40?, 0x14ae160?, 0x4003537568?)
 /usr/local/go/src/runtime/map.go:1392 +0x1c
reflect.(*MapIter).Next(0x13ad9e0?)
 /usr/local/go/src/reflect/value.go:1935 +0x60
encoding/json.mapEncoder.encode({0x40035377d8?}, 0x400379a700, {0x13ad9e0?, 0x4001770600?, 0x165acea?}, {0x6?, 0x0?})
 /usr/local/go/src/encoding/json/encode.go:744 +0x28c
encoding/json.structEncoder.encode({{{0x40008fa000, 0x19, 0x23}, 0x4000e400c0, 0x4000e400f0}}, 0x400379a700, {0x1628840?, 0x40017704e0?, 0x12c67350000000f?}, {0x5c?, ...})
 /usr/local/go/src/encoding/json/encode.go:706 +0x190
encoding/json.ptrEncoder.encode({0x40035379f8?}, 0x400379a700, {0x1614ca0?, 0x40017704e0?, 0x1000003537a38?}, {0x1c?, 0xd4?})
 /usr/local/go/src/encoding/json/encode.go:878 +0x1c4
encoding/json.(*encodeState).reflectValue(0x4003537aa8?, {0x1614ca0?, 0x40017704e0?, 0x12d2e0?}, {0xb8?, 0x7a?})
 /usr/local/go/src/encoding/json/encode.go:323 +0x70
encoding/json.(*encodeState).marshal(0x4003537b18?, {0x1614ca0?, 0x40017704e0?}, {0xd8?, 0x5e?})
 /usr/local/go/src/encoding/json/encode.go:295 +0xb8
encoding/json.Marshal({0x1614ca0, 0x40017704e0})
 /usr/local/go/src/encoding/json/encode.go:162 +0x94
github.com/aws/aws-xray-sdk-go/xray.packSegments.func1(0x40017704e0)
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/default_emitter.go:106 +0x1a4
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x4004535ba0, {0x400373f920, 0x2, 0x2})
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/default_emitter.go:116 +0xcc
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x4000103860, {0x400373f920, 0x2, 0x2})
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/default_emitter.go:115 +0xb0
github.com/aws/aws-xray-sdk-go/xray.(*DefaultEmitter).Emit(0x4000a141b0, 0x18a44?)
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/default_emitter.go:72 +0x9c
github.com/aws/aws-xray-sdk-go/xray.(*Segment).emit(0x4003d90600?)
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/segment.go:468 +0x84
github.com/aws/aws-xray-sdk-go/xray.(*Segment).flush(0x1cf4f98?)
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/segment.go:510 +0x224
github.com/aws/aws-xray-sdk-go/xray.(*Segment).send(0x4000103860)
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/segment.go:490 +0x60
github.com/aws/aws-xray-sdk-go/xray.(*Segment).handleContextDone(0x4000103860)
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/segment.go:476 +0x54
github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling.func1()
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/segment.go:152 +0x40
created by github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling in goroutine 3030832
 /go/pkg/mod/github.com/aws/aws-xray-sdk-go@v1.8.2/xray/segment.go:150 +0xd2c