Open mumoshu opened 6 years ago
I'd love for filebeat to output to cloudwatch logs! The cloudwatch logs agent doesn't have a 'processors' feature, or really any concept of fields. This makes adding cloud metadata to events really difficult.
My current experiment is to try filebeat -> kinesis -> lambda -> cwlogs -> kinesis -> logstash -> es. A CWLogs output plugin would skip the first kinesis -> lambda.
@jpuskar Hey, thanks for the feedback!
I'm also looking forward to this feature, especially after the arrival of CloudWatch Logs Insight which gives us the serverless alternative to kibana/es query :)
Your experiment looks good. The pipeline from kinesis to lambda should be used whenever you can't afford the cloudwatch logs' cost per ingested GB - $0.4~0.5/GB oh my - and the ingestion rate linit.
But it doesn't negate the usefulness of this feature for the smaller scale.
As per me and @s12v's discussion in #8, probably we can include CloudWatch Logs output plugin within awsbeats. Anyone has a specific use-case for that? Any concrete feature you'd like to have?