search

s12v / awsbeats

AWS Kinesis plugin for Filebeat
Apache License 2.0
61 stars 26 forks source link

CloudWatch Logs output plugin #20

Open mumoshu opened 6 years ago

mumoshu commented 6 years ago

As per me and @s12v's discussion in #8, probably we can include CloudWatch Logs output plugin within awsbeats. Anyone has a specific use-case for that? Any concrete feature you'd like to have?

jpuskar commented 5 years ago

I'd love for filebeat to output to cloudwatch logs! The cloudwatch logs agent doesn't have a 'processors' feature, or really any concept of fields. This makes adding cloud metadata to events really difficult.

My current experiment is to try filebeat -> kinesis -> lambda -> cwlogs -> kinesis -> logstash -> es. A CWLogs output plugin would skip the first kinesis -> lambda.

mumoshu commented 5 years ago

@jpuskar Hey, thanks for the feedback!

I'm also looking forward to this feature, especially after the arrival of CloudWatch Logs Insight which gives us the serverless alternative to kibana/es query :)

Your experiment looks good. The pipeline from kinesis to lambda should be used whenever you can't afford the cloudwatch logs' cost per ingested GB - $0.4~0.5/GB oh my - and the ingestion rate linit.

But it doesn't negate the usefulness of this feature for the smaller scale.