New `aws_cloudwatch_logs` source

[binarylogic]

The goal of this source is to be a simple way to start consuming logs from AWS in low volume environments by pulling logs from exposed AWS CloudWatch APIs (FilterLogEvents and GetLogEvents).

There will be a relatively low volume limit to this source (< 5 MBps / 50k events in the case of FilterLogEvents) which we should document and take extra care around surfacing when we are hitting AWS rate limits.

We've also observed that FilterLogEvents does not perform well for groups with a large number of streams. If we go with this API, we should document this restriction (and provide knobs for specificity specific streams or stream prefix).

I'd probably recommend an RFC just for this source to suss out any decision points and to provide the answers to open questions, including:

• Do we want to allow users to scrape to scrape multiple log groups with a single source? Do we want to allow wildcards for the group name if so?
• Decide on FilterLogEvents vs. GetLogEvents (or do we want to support both?)
• Do we want to allow consuming a log group from its beginning? Or just newly published events?
• Do we want to persist checkpoints on disk to allow for restarts? Or just always start from "now"?
• If we go with FilterLogEvents do we want to allow users to specify a filter?
• Should we rely on the AWS pagination token (nextToken) for checkpoints or store our own serialized cursor that we can use to pick back up. nextToken is only valid for 24 hours and, I think, is tied to the filter pattern for FilterLogEvents (though we can verify)

Examples of other implementations:

• https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs
• https://github.com/lukewaite/logstash-input-cloudwatch-logs

External refs:

• https://github.com/aws/containers-roadmap/issues/66

[jszwedko]

@binarylogic Rather than creating a new issue, I updated this issue description to be for the naive aws_cloudwatch_logs source. I'm not sure about how we want to prioritize it vs. the AWS Firehose -> Vector source (https://github.com/timberio/vector/issues/3566)

[trevorstr]

We've got logs being emitted to CloudWatch Logs from other AWS services (ie. Lambda). We would like to move these logs to Grafana Loki. It looks like Amazon CloudWatch Logs isn't currently supported as a data source.

[jszwedko]

The currently recommended approach for CloudWatch Logs is to use Firehose to forward them.