matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
https://matano.dev
Apache License 2.0
1.44k stars 97 forks source link

Managed log sources for CloudWatch Metric Streams #120

Open timoguin opened 1 year ago

timoguin commented 1 year ago

Add managed log source for CloudWatch metric streams.

Considerations

Currently, the main way to get CW metrics streams is through integration with Kinesis Firehose. If we supported the metrics stream output format, the user would only need to point their Firehose(s) at an S3 bucket for Matano to ingest the data. The log source will need to handle decoding from base64 since that's how Firehose will deliver it (unless configured with Lambda transforms).

JSONL output format seems to be the one that is most supported. There is also an OTEL output format, but it has less features and seems to be a bit more experimental.

References