search

splunk / splunk-aws-project-trumpet

MIT License
95 stars 31 forks source link

Receive *all* CloudwatchLog streams? #9

Open atlgeek007 opened 4 years ago

atlgeek007 commented 4 years ago

Is there a way to specify a wildcard for all cloudwatch log streams and groups? Or would I need to update the template every time we decide to add a new group/stream?

spensireli commented 4 years ago

+1 on this. I would prefer to opt for collecting all log files in CloudWatch if possible.

nstonesplunk commented 4 years ago

Thanks for bringing this up. This one is a little tricky because the ingestion path relies on CloudWatch subscription filters which are 1-1 with each CloudWatch log group. That said we could probably use a Lambda shim that finds all log groups matching the wildcard and creates a subscription filter for each of them. If someone has a better idea please jump in!

I'll look into this.

atlgeek007 commented 4 years ago

Thank you very much, this tool has come in very handy for my Splunk deployment and that would be icing on the cake!

atlgeek007 commented 4 years ago

We are working around it by having our codepipelines create the subscription filter when they create a new log stream, though we haven't yet gotten that to a testing phase and it may not work as intended.

atlgeek007 commented 4 years ago

Another feature I'd want is a way to stream an s3 bucket containing ELB/ALB access logs to the Firehose with the sourcetype aws:elb:accesslogs since reading those in from S3 is prone to throttling.

nstonesplunk commented 4 years ago

So far this has been working with a log group prefix for existing log groups, but only evaluates log groups when the template is run. Any new log groups that are created after the template is run that match the prefix would not be added without rerunning the template. What is nice about this approach is that removing the Cloudformation stack will remove all the subscription filters as well. Obviously the downside is that it is not a continuous process, the other downside is it requires two templates, one that creates a Cloudformation macro and another that uses the macro to discover log groups and add them back to the template before running

Another option is to deploy a Lambda with the Cloudformation template that looks for log groups matching a provided prefix that do not have a subscription filter applied, and applies one that routes events to Firehose. The main downside of this approach is that the Cloudformation stack has no knowledge of newly created subscription filters, so if the stack is deleted, the subscription filters added by the Lambda will not be deleted and will send to a deleted Firehose.

For now we'll work on the first option and think about ways to make it continuous.

The access logs can probably be read from Lambda and sent directly to HEC, similarly to config snapshots. May not be too much work - i'll look into this as well.

atlgeek007 commented 4 years ago

I would find updating the stack on creation of new log groups to be an acceptable compromise if that's how it has to work. I like being able to clear the subscription filters if I remove the stack.

AlDawoode commented 3 years ago

I think you can achieve that by creating lambda function that runs every day or week once ... Lambda will check all cloudwatch log groups and see if any group doesn't has a subscription to firehose, it will create one.