search

awslabs / aws-fluent-plugin-kinesis

Amazon Kinesis output plugin for Fluentd
Apache License 2.0
293 stars 96 forks source link

Send data to different Kinesis stream #165

Closed jenishshah671993 closed 5 years ago

jenishshah671993 commented 6 years ago

Hi, I have JSON data. On the basis of JSON key value I need to send data to different kinesis stream. So can you suggest me if it's possible

cosmo0920 commented 5 years ago

It is possible to implement with built-in placeholders, which is the part of brand new Fluentd v1 API, for https://github.com/awslabs/aws-fluent-plugin-kinesis/blob/master/lib/fluent/plugin/out_kinesis_streams.rb#L49 Currently, it is not implemented.

cosmo0920 commented 5 years ago

I've registered a PR to fix this issue. Does anyone want to try out this?

simukappu commented 5 years ago

Thank you for your contributing. I think we need to separate buffer for each destination streams for plugin durability. What do you think?

cosmo0920 commented 5 years ago

@simukappu Fluentd v1 separates buffer for each destination streams: https://docs.fluentd.org/v1.0/articles/buffer-section#other-keys

With the testing configuration, Fluentd handles buffer per $.key.nested.

simukappu commented 5 years ago

Sounds good. Does anybody have any feedback for this issue or PR?

amitdhawan commented 5 years ago

Is this done? I want to use this feature.

amitdhawan commented 5 years ago

Sounds good. Does anybody have any feedback for this issue or PR?

@simukappu Lets merge it.

cosmo0920 commented 5 years ago

@amitdhawan Can #174 patch work under your created AWS kinesis environment? Testing on a few users are required.

amitdhawan commented 5 years ago

Should i install this plugin from this branch on my fluentd config? How can i do that, could you please share the steps.I can test that on my AWs environment then.

amitdhawan commented 5 years ago

Let me know the steps how can install this plugin on Fluentd. I can test it on my environment then.

cosmo0920 commented 5 years ago 
$ git clone git@github.com:awslabs/aws-fluent-plugin-kinesis.git
$ git fetch origin pull/174/head:pr-174
$ git checkout pr-174
$ gem build fluent-plugin-kinesis.gemspec

And then, 

$ td-agent-gem install fluent-plugin-kinesis-*.gem
# Or
$ fluent-gem install fluent-plugin-kinesis-*.gem
amitdhawan commented 5 years ago

ok..let me give it a shot.

simukappu commented 5 years ago

Released as v3.2.0. https://rubygems.org/gems/fluent-plugin-kinesis/versions/3.2.0

If you have any feedback, feel free to share with us.