search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.44k stars 1.07k forks source link

Allow two Graylog clusters to read from same Kinesis steam #18149

Open stepkirk opened 10 months ago

stepkirk commented 10 months ago

What?

Allow two different Graylog clusters to use the AWS Kinesis CloudWatch Input to read from the same Kinesis stream simultaneously.

Why?

I am starting a project to migrate from Graylog V3.x to Graylog 5.x clusters. My plan is to run both clusters in parallel so users can use both while they transition all of their Alerts/Dashboards/etc from old to new.

I would like the ability to have an Input on both clusters that read from the same Kinesis stream. As it sits today, it appears that I would have to disable the Input on the old cluster in order for the new cluster to see new messages on the stream as they both use the same DynamoDB table to track where they are in reading data in the stream.

For example, the stream name is eventflow and the DynamoDB table name (for both) is graylog-aws-plugin-eventflow. Seems like the solution would be to make the DynamoDB table name unique for each Graylog cluster?

Please advise if there is already a way to do this and I have overlooked how to do so.

Your Environment

kroepke commented 9 months ago

Yes, you are correct. For Kinesis to support multiple consumers the application name used in the client needs to be different, which right now it isn't.

I suppose adding the cluster id to the name will solve the issue. For reference https://github.com/Graylog2/graylog2-server/blob/3afdedfc60015e9dda8a73a00c78f3844ddc0818/graylog2-server/src/main/java/org/graylog/integrations/aws/transports/KinesisConsumer.java#L129 is using a fixed name, and we need to make this unique per cluster.

kroepke commented 9 months ago

Please also backport this to 5.2 to aid in migration.

stepkirk commented 7 months ago

Any idea when this will be "fixed". Will it be in 5.2.x or maybe 6.0?

stepkirk commented 5 months ago

@kroepke , any updates on this? Fixed in one of the 5.2.x releases or in the new 6.0.x?

sethgraylog commented 5 months ago

Hi @stepkirk, this feature has not been completed as of yet. It is in a short list to be completed but has run into higher priorities that have delayed our ability to get to it. The Product Owner will definitely update the status when this does move forward.