search

s12v / awsbeats

AWS Kinesis plugin for Filebeat
Apache License 2.0
61 stars 26 forks source link

streams: Panic on collecting failed records #41

Closed mumoshu closed 6 years ago

mumoshu commented 6 years ago

I started to see persistent panics like the below in my production deployment:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7fda4c8ce9b1]

goroutine 83 [running]:
github.com/s12v/awsbeats/streams.collectFailedEvents(0xc420ad1dd0, 0xc420f4b300, 0x32, 0x3b4, 0xc420ad1dd0, 0x0, 0x0)
        /go/src/github.com/s12v/awsbeats/streams/client.go:170 +0xd1
github.com/s12v/awsbeats/streams.(*client).publishEvents(0xc4200da930, 0xc420f4b300, 0x32, 0x3b4, 0x1d14c30, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/s12v/awsbeats/streams/client.go:92 +0x2be
github.com/s12v/awsbeats/streams.(*client).Publish(0xc4200da930, 0x14b9860, 0xc422b27940, 0xc42008ce40, 0xc420f69f78)
        /go/src/github.com/s12v/awsbeats/streams/client.go:64 +0x45
github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc42000d8a0, 0x14b9860, 0xc422b27940, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:43 +0x4b
github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc4200af280)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:90 +0x1a9
created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:31 +0xf0

This indicates that we're receiving records with unexpected structures from kinesis streams. I have no reference to the concrete specification of kinesis records that I'm unable to "fix" it.

In the meantime, I'd make awsbeats gracefully degrade, that is to give up retrying failed records with unexpected structures but just leave some log messsage to help further investigation.