search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.68k stars 1.56k forks source link

New `aws_cloudtrail_logs` source #3077

Open binarylogic opened 4 years ago

binarylogic commented 4 years ago

This source will collect events from AWS CloudTrail and ingest them into Vector as log events. We need to investigate what this integration will look like, and therefore a spec/RFC is in order.

jszwedko commented 4 years ago

I discovered you can forward CloudTrail logs to CloudWatch Logs so #3566 could be tractable for getting CloudTrail logs out too; however it has a pretty big limitation:

Because CloudWatch Logs has an event size limitation of 256 KB, CloudTrail does not send events larger than 256 KB to CloudWatch Logs. For example, a call to the EC2 RunInstances API to launch 500 instances will exceed the 256 KB limit. CloudTrail does not send the event to CloudWatch Logs. To ensure that CloudTrail sends events to CloudWatch Logs, break large requests into smaller batches.

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/send-cloudtrail-events-to-cloudwatch-logs.html

I'm imagining we'll prefer the traditional route of CloudTrail -> S3 with a new Vector S3 source.