Incorrect metadata on LogArchive Cloudwatch logs

[edhull]

Describe the bug

Cloudwatch logs are being compressed and consolidated within the aws-accelerator-central-logs-<account>-<region>/CloudWatchLogs S3 bucket/path in the LogArchive account via the Landing Zone Accelerator. However, these logs are not being assigned a metadata/mimetype of gzip when stored. Other logs in the same bucket (such as CloudTrail logs, load balancer logs) are correctly being assigned this metadata.

The impact of this bug is that when attempting to ingest these logs (e.g. https://github.com/aws-samples/siem-on-amazon-opensearch-service) Cloudwatch logs cannot correctly be ingested as there no metadata or file extension to indicate that they should first be decompressed.

To Reproduce

• LZA deployed
• Access LogArchive account
• Access log object stored within S3 aws-accelerator-central-logs-<acount>-<region>/CloudWatchLogs/
• View object metadata
• Observe that no Content-Encoding is set, and the Content-Type is set to application/octet

Expected behavior Compressed Cloudwatch logs which are consolidated from Organization accounts should be stored with the correct metadata/mimetype. This should include a Content-Encoding of gzip and Content-Type of application/json.

Please complete the following information about the solution:

• [x] Version: v.1.6.1
• [x] Region: eu-west-2
• [x] Was the solution modified from the version published on this repository? No
• [x] If the answer to the previous question was yes, are the changes available on GitHub?
• [x] Have you checked your service quotas for the services this solution uses? Yes
• [x] Were there any errors in the CloudWatch Logs? N/A

[nagmesh]

Hello,

Thank you for your interest in LZA.

Firehose by default puts object with in s3 with Content-Type application/octet-stream.

Currently within s3 destination settings we have to leave it as uncompressed as CloudWatch sends the files with gzip level8 compression. Firehose service released a new feature to add file extension where we can try and add .gz extension. However, we will need to run a full test to see the behavior is same on all regions and partitions before it can be released.

I'll leave this ticket to track the firehose file extension ask. The other alterantives are to

• Have firehose write to opensearch which is being tracked here
• use the log s3 prefix as input to data lake that runs jobs which will pre-process and stage it in the right format for the siem tool