Dedicated S3 bucket per log_source

[MadsRC]

At the moment, only a single S3 bucket seem to be supported as the ingestion bucket.

It would make integration more frictionless if it was possible to define an ingestion bucket per defined log_source. Perhaps make it optional, so that if it isn't specified, the default ingestion bucket will be used.

One use-case is CloudTrail logging, where the organisation already has a S3 bucket with CloudTrail. Being able to point matano at that bucket would avoid the cost of either having CloudTrail send to multiple buckets, or the delay/cost associated with reading from the main bucket and copying to the ingestion bucket.

[shaeqahmed]

If I understand correctly, this is already supported :)

https://www.matano.dev/docs/log-sources/ingestion#bringing-your-own-bucket

We call it "bring your own bucket" and it is configured via the ingest.s3_source parameters for bucket_name and key_prefix.

The idea is that we support a Matano managed bucket based ingestion by default for all log sources, but for use cases where there is a pre-existing one you can onboard it via this configuration (e.g. onboard Cloudtrail).

[MadsRC]

I read that part of the docs, and the way that I read it is that you can bring exactly one bucket. So all your log sources sourced through an S3 bucket would have to be from that bucket?

Imagine an org that has one bucket for CloudTrail logs, and one bucket per region for Load Balancer logs (since they have to be in the same region as the load balancer).

Or am I misinterpreting the docs, and it's already supported to bring a dedicated bucket to each individual log source?

[Samrose-Ahmed]

Yes, it is supported to bring an individual bucket per log source if you use BYOB. Each log source can use a different bucket in the ingest.s3_source configuration.

[MadsRC]

In that case, I'll close this ticket! Thank you!