Aws-s3 Wodle bugs and improvements

hitman28594 commented 2 years ago

Affected integration
AWS, GCloud, Azure...

Description

[ ] Test in a manager.
[ ] Test in an agent.
[ ] Unit tests without failures. Updated if there are any relevant changes.
[ ] Integration tests without failures. Updated if there are any relevant changes.
[ ] Update the documentation if necessary.
[ ] Add entry to the changelog if necessary.

Hello team,

I am writing to highlight several issues and improvements opportunities with the aws-s3 Wodle.

Custom bucket type in aws-s3 Wodle doesnt really support custom types, the documentation doesn’t clarify this very well:

Log collection Improvements:

add support for a new bucket_type to ingest any JSON logs.
add support for a new bucket_type to ingest CSV formatted logs, with option to specify the csv separator/delimiter.
The s3 wodles cloudtrail.db only stores “1000” entries per bucket type. When monitoring large buckets or multiple buckets of the same “bucket_type” this causes the monitoring to stop working for some of the configured buckets as the Wodle forgets it ever monitored certain paths/files.
there should be a check in the aws-s3 Wodle to confirm that the number of events ingested by the Wodle matches the number of raw events in the bucket/logfile. These inconsistencies should be monitored and reported even if they are ignored or skipped.
support for monitoring multi-line logs stored within an s3 bucket… basically any option available to localfile, should be available to the aws-s3 Wodle.

Other non log-collection improvements:

Wazuh has FIM capabilities via agents.
S3 creates a “digest” file for the logs in an s3 bucket, which is basically a FIM check per logfile.
if these digest files change, this is a FIM “file modified” event.
It would be nice if the aws-s3 Wodle could support using these digest files as part of the FIM analysis checks.

davidjiglesias commented 2 years ago

Thanks @hitman28594 for your time to contribute. We are actively working in improvements for the aws-s3 wodle and the other cloud integrations.

davidjiglesias commented 1 year ago