search

logstash-plugins / logstash-input-kinesis

Logstash Plugin for AWS Kinesis Input
Apache License 2.0
45 stars 57 forks source link

Upgrade to KCL v2 #55

Closed w4 closed 2 years ago

w4 commented 5 years ago

Resolves #46.

The way KCL is configured has changed between v1 and v2, configuration values are stored in separate POJOs now so there can't just be a single additional_options block that allows everything to be changed.

Since the consumer now has to instantiate the clients it wants to use itself there will have to be separate configuration values for setting things like Kinesis endpoint (so there has been a regression on items like #49 since #51) - endpointOverride takes a URI and unsure how we'd handle that without adding kinesis_endpoint, cloudwatch_endpoint, etc configuration values separately, open to any suggestions if that should be in the scope of this PR.

Note: this is a breaking change as it requires the following extra permissions to be added to the IAM role being assumed by logstash:

w4 commented 5 years ago

Looks like we're still running into #35 with this. We're still currently having to use a cronjob to restart Logstash every 24 hours, we're looking into this hopefully we can come up with a fix.

w4 commented 5 years ago

Just as an update, we've started implementing a Kinesis input using the new Java plugin API. I've heard from one of the guys working on the KCL that generally when you use the KCL you'd also implement a "stuck shard detector" so I think that's the missing piece of the puzzle here.

If I manage to get a stable version of this working I'll see what I can do about getting it out to you all

stefankirov commented 4 years ago

Hi, do you know if this update to KCL 2.0 will be released? AWS has released support for Enhanced Fan Out which can be consumed only with the KCL 2.0. Is there a way for us to fork your solution and complete the missing pieces so I can use/test kinesis enhanced fan out feature locally?

lobsterdore commented 4 years ago

I'm keen for this update as well, is there any news on when this might be finished off and merged?

mik973 commented 2 years ago

Do you know if this update to KCL 2.0 will be released in the near future?

w4 commented 2 years ago

I'm no longer working at Amazon nor using Kinesis and so have no intention of keeping this MR up to date, apologies.