After #28, it seems like filebeat+awsbeats+kinesis data streams occasionally fail while retrying batch send to Kinesis:
2018-05-28T03:35:56.266Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":97},"total":{"ticks":710,"time":723,"value":710},"user":{"ticks":620,"time":626}},"info":{"ephemeral_id":"6703632b-8559-4104-8e5a-5b9d64489e08","uptime":{"ms":60027}},"memstats":{"gc_next":18773184,"memory_alloc":13742984,"memory_total":136054552,"rss":27189248}},"filebeat":{"events":{"active":4568,"added":4578,"done":10},"harvester":{"open_files":30,"running":30,"started":30}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":2},"output":{"events":{"acked":4146,"batches":83,"total":4146}},"pipeline":{"clients":4,"events":{"active":4067,"filtered":404,"published":4166,"retry":50,"total":4570},"queue":{"acked":100}}},"registrar":{"states":{"current":70,"update":10},"writes":10},"system":{"load":{"1":8.43,"15":7.16,"5":7.46,"norm":{"1":4.215,"15":3.58,"5":3.73}}}}}}
2018-05-28T03:35:58.686Z INFO log/harvester.go:216 Harvester started for file: /var/lib/docker/containers/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf-json.log
2018-05-28T03:36:12.918Z INFO log/harvester.go:216 Harvester started for file: /var/lib/docker/containers/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf/26bbe1f1660ab24647e5d0342d700fe820776312d3fb4603e07831a6eceb1dbf-json.log
2018-05-28T03:36:17.215Z ERROR registrar/registrar.go:246 Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:17.465Z ERROR registrar/registrar.go:246 Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:21.155Z ERROR registrar/registrar.go:246 Writing of registry returned error: rename /usr/share/filebeat/data/registry.new /usr/share/filebeat/data/registry: no such file or directory. Continuing...
2018-05-28T03:36:22.237Z WARN streams/client.go:144 Retrying 50 events
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xd74489]
goroutine 71 [running]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*Batch).Drop(0xc42168a000)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/batch.go:59 +0x29
github.com/elastic/beats/libbeat/publisher/pipeline.(*retryer).loop(0xc420139fc0)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:140 +0x4ff
created by github.com/elastic/beats/libbeat/publisher/pipeline.newRetryer
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:70 +0x138
After #28, it seems like filebeat+awsbeats+kinesis data streams occasionally fail while retrying batch send to Kinesis: