Managed log source for AWS S3 access logs

timoguin commented 1 year ago

Add support for managing AWS S3 access logs.

Considerations

S3 access logs are an odd man out for a few reasons:

KMS encryption is NOT supported (must use SSE-S3 encryption, if any)
Cross-account access requires roles assumption due to object ownership weirdness

Tasks

[x] ECS normalization for S3 access logs
[x] VRL transforms
[ ] UX design if role assumption is required (add role ARN to config?)

References

Samrose-Ahmed commented 1 year ago

Just shipped the parser.🚀

Can you elaborate on the cross account/role note?

From what I can tell, S3 access logs doesn't support cross account access, in that the target and source bucket have to be in the same account. They seem to reccomend CloudTrail data logs for that case.

https://repost.aws/questions/QU5QIi-GQSTwiWz1UzY2MnWw/s-3-server-access-logging-another-account
https://docs.aws.amazon.com/AmazonS3/latest/userguide/logging-with-S3.html (Cross-account log delivery)

timoguin commented 1 year ago

Just shipped the parser.🚀

Nice!

Can you elaborate on the cross account/role note?

Yes, S3 access logs can only be delivered when the source and destination bucket exist within the same account and the same region.

I'm talking about what you have to do if you want to process logs in a bucket that exists in another account.

S3 access logs are unique in the sense that you need to assume a cross account role and then copy the objects while modifying the object ownership.

I'd need to look back into the specifics to elaborate further. It's been several years since I've done it. This is all from memory.

The main point is that if we want to allow Matano to process these logs from a bucket in another account, a cross-account S3 bucket policy isn't enough. We would at least need to do some UX planning to make it easy.

For now I would say ship with initial support for buckets in the same account Matano is running in. Then we can follow up to add mechanisms for processing buckets in another account.

From what I can tell, S3 access logs doesn't support cross account access, in that the target and source bucket have to be in the same account. They seem to reccomend CloudTrail data logs for that case.

Logging S3 Data Events with CloudTrail gets really expensive quickly at scale. You will have pain if you enable them on high-traffic buckets. They'll blow the CloudTrail budget out of the water fast.

At this point I would only ever turn them on for small, high risk buckets that don't see any serious production traffic.

Normal S3 access logs are free though, and they still contain a ton of valuable data.

Samrose-Ahmed commented 1 year ago

matanolabs / matano