search

lukewaite / logstash-input-cloudwatch-logs

Input plugin for Logstash to stream events from CloudWatch Logs
Other
142 stars 74 forks source link

Get all log groups matching log group prefix #7

Closed lukewaite closed 7 years ago

lukewaite commented 8 years ago

Contribution received via patchfile.

Fixes #2

lukewaite commented 8 years ago

My primary concern with this patch as currently implemented is that this changes the behaviour of the log_group setting. Assuming it is always a prefix, and searching against it may be undesirable for some users.

Feedback is welcome as to how to implement. We could possibly add a log_group_prefix setting, and make log_group and log_group_prefix not required, but depend on at least one of them being set.

lukewaite commented 8 years ago

Optionally required inputs are handled during register by the beats plugin, so I think that would probably be the best way to go about it here. https://github.com/logstash-plugins/logstash-input-beats/blob/master/lib/logstash/inputs/beats.rb#L108-L113

Perhaps not ideal, since I don't think it's caught during a --configtest, but perhaps the best we can accomplish without a specific api being made available for a plugin to do more advanced configuration testing. If something like this exists, I'm missing it.

Chintanvpatel commented 8 years ago

Hello @lukewaite ,

This will be very useful in case if we have multiple log groups with same pattern. So we don't required to run multiple logstash services with same configuration.

It will be good instead of giving prefix option, you can make "log_group" as array so we can give some specific log group names.

Thanks.

lukewaite commented 8 years ago

Thanks for the input @Chintanvpatel.

I think that there would be a use case for both prefix and array based log-group selection.

dayglojesus commented 8 years ago

First, this plugin saved me a ton of work, so thanks for sharing!

Second, 👍 on the patch -- this would be extremely useful. I'm trying to source CloudWatch logs for API Gateway and the naming convention for these logs is really odd and there doesn't appear to be anything one can do to control it...

Example: API-Gateway-Execution-Logs_3e1cj9xz8i/test

Each stage of the API gets its own "log-group" which makes the logstash input configuration... challenging.

lukewaite commented 7 years ago

These changes were included in the MR from @davidwestlund https://github.com/lukewaite/logstash-input-cloudwatch-logs/pull/9 which has just been merged.

JoaoRXFernandes commented 7 years ago

+1 for this one! Thanks!