OOTB support AWS Eventbridge

mjmbischoff commented 4 days ago

Proposed commit message

Adding OOTB support AWS Eventbridge generated events for S3 changes, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html

Checklist

[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] ~I have made corresponding change to the default configuration files~
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

The logic is triggered as a fall-through and is best effort.

Author's Checklist

[ ] Check for idiomatic golang constructs not being used, with golang not being my primary lang there's a good chance for non-idiomatic constructs to sneak in.

How to test this PR locally

Since it relies on an AWS setup, testing locally beyond units tests is not possible.

Related issues

As it was discovered as part of support ticket there should be an ER linked.

Use cases

AWS has added a new way to notify on S3 changes, next to the existing plain SQS and SNS->SQS, AWS now also support using AWS EventBridge -> SQS

This PR achieves two goals:

no longer need a custom script (which is the current workaround
Improve user experience by allowing it to 'work out of the box'

cla-checker-service[bot] commented 4 days ago

💚 CLA has been signed

mergify[bot] commented 4 days ago

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @mjmbischoff? 🙏. For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine commented 4 days ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mjmbischoff commented 3 days ago

As a general comment; it looks like aws security-lake uses the same format for notifications but omits the detail-type field: https://docs.aws.amazon.com/security-lake/latest/userguide/subscriber-data-access.html#sample-notification

Also really liking the part where non of these protocols look like their are versioned or anything. :disappointed:

mjmbischoff commented 3 days ago

As a general comment; it looks like aws security-lake uses the same format for notifications but omits the detail-type field: https://docs.aws.amazon.com/security-lake/latest/userguide/subscriber-data-access.html#sample-notification

Also really liking the part where non of these protocols look like their are versioned or anything. 😞

We have an integration for Amazon Security Lake which currently reads:

The Amazon Security Lake integration currently supports only one mode of log collection:

AWS S3 polling mode: Amazon Security Lake writes data to S3, and Elastic Agent polls the S3 bucket by listing its contents and reading new files.

This change would also allow us to improve that integration, again the detail-type field is in question. I guess we should spin up a AWS security lake setup to verify. I hope it's there and the documentation is off as don't think it's safe to assume every event is object created when the field is not available.

The integration is now quite limited as a lot of users have a retention long enough that there's enough objects in the bucket that polling becomes prohibitively expensive.

mjmbischoff commented 3 days ago

https://github.com/elastic/beats/pull/40006/commits/02cae8e1f77d7c3785b0cda6fe3b6a6c7528fa4c to cover 'I have made corresponding changes to the documentation'

mergify[bot] commented 3 days ago

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ootb-aws-eventbridge upstream/ootb-aws-eventbridge
git merge upstream/main
git push upstream ootb-aws-eventbridge

mjmbischoff commented 3 days ago

and https://github.com/elastic/beats/pull/40006/commits/a8eb0747be4d37f363e3336cba4c63efe842e6f9 to cover changelog item.

elasticmachine commented 2 days ago

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

elasticmachine commented 2 days ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elasticmachine commented 2 days ago

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

elasticmachine commented 2 days ago

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

mergify[bot] commented 1 day ago

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ootb-aws-eventbridge upstream/ootb-aws-eventbridge
git merge upstream/main
git push upstream ootb-aws-eventbridge

elastic / beats