s12v / awsbeats

AWS Kinesis plugin for Filebeat
Apache License 2.0
61 stars 26 forks source link

fix(streams): Various fixes to correctly retry only on Kinesis failure, plus Stream+Firehose+S3 support #31

Closed mumoshu closed 6 years ago

mumoshu commented 6 years ago

Turns out awsbeats was calling batch.RetryEvents on different data and timing. Revise the implementation according to the official beat outputs

Fixes #29

I've also added a new-line per each kinesis record data.

Firehose doesn't automatically add trailing new-line on after each record. This ends up a stream->firehose->s3 pipeline to produce useless s3 objects. Not ndjson, but a sequence of json objects without separators... Fix it just adding a new-line.

mumoshu commented 6 years ago

This PR depends on #28

codecov-io commented 6 years ago

Codecov Report

Merging #31 into master will increase coverage by 29.73%. The diff coverage is 73.68%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master      #31       +/-   ##
===========================================
+ Coverage   33.89%   63.63%   +29.73%     
===========================================
  Files           4        4               
  Lines         118      132       +14     
===========================================
+ Hits           40       84       +44     
+ Misses         71       42       -29     
+ Partials        7        6        -1
Impacted Files Coverage Δ
streams/client.go 68.81% <73.68%> (+43.5%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 028b7c1...a4967d5. Read the comment docs.

mumoshu commented 6 years ago

This addresses a few issues I had encountered in my production testing. As I'm satisfied with the implementation and tests are added for additional coverage, I'll be happily merging this tomorrow :)

mumoshu commented 6 years ago

@s12v Thanks for reviewing 👍