improvement: Refactor CloudTrail log processing, support batching(com… - Githubissues

newrelic / aws_s3_log_ingestion_lambda

AWS Lambda that sends log data from S3 to New Relic Logging

Apache License 2.0

13 stars 44 forks source link

improvement: Refactor CloudTrail log processing, support batching(com… #34

Closed haihongren closed 2 years ago

haihongren commented 2 years ago

Fix #32
Fix #35

Support batch send for CloudTrail log

Refactored CloudTrail log processing logic to take advantage of existing batch send function
Externalized BATCH_SIZE_FACTOR, added S3_CLOUDTRAIL_LOG_PATTERN, S3_IGNORE_PATTERN env variables

BATCH_SIZE_FACTOR: affects uncompressed batch size, default: 1.5 S3_CLOUDTRAIL_LOG_PATTERN: regex pattern to match CloudTrail log file, default: .CloudTrail..json.gz$ S3_IGNORE_PATTERN: regex pattern to match log file to be ignored/skipped, default: $^

haihongren commented 2 years ago

@William-Hill Could you please review? thanks!