s12v / awsbeats

AWS Kinesis plugin for Filebeat
Apache License 2.0
61 stars 26 forks source link

feat: Running metricbeat+awsbeats on Kubernetes #24

Closed mumoshu closed 6 years ago

mumoshu commented 6 years ago

See the last two commits for changes relevant to this PR.

You can basically build a metricbeat+awsbeats docker image and deploy it to your Kubernetes cluster. I tried hard to not fork the upstream metricbeat deployment manifests. So, it is matter of just git-cloning the upstream elastic/beats repo, replace some parts of the manifests, run kubectl -f apply.

I have also added an experimental feature to streams output plugin. It is enabled via partition_key_provider: xid. This is useful to evenly distribute metricbeat events to Kinesis Data Stream shards for maximum scalability.

The first commit is #23, hence this PR Depends on #23.

codecov-io commented 6 years ago

Codecov Report

Merging #24 into master will increase coverage by 9.15%. The diff coverage is 54.54%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #24      +/-   ##
==========================================
+ Coverage   24.74%   33.89%   +9.15%     
==========================================
  Files           3        4       +1     
  Lines          97      118      +21     
==========================================
+ Hits           24       40      +16     
- Misses         68       71       +3     
- Partials        5        7       +2
Impacted Files Coverage Δ
streams/config.go 80% <0%> (-20%) :arrow_down:
streams/client.go 25.31% <40%> (+4.26%) :arrow_up:
streams/partition_key.go 75% <75%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0175ee7...1aad605. Read the comment docs.

mumoshu commented 6 years ago

Ah, finally Codecov is happy ☺️