While the setup appears to be correctly ingesting various types of logs, I'm encountering issues specifically with VPC Flow Logs. Despite adjusting the s3_key to point to the appropriate location of the VPC Flow Logs in S3, the logs either fail to be ingested, or I encounter errors. The most telling feedback I've received is a warning message indicating that no entries were successfully loaded:
{
"level": "WARNING",
"message": "No entries were successed to load",
"location": "process_records:346",
"timestamp": "2024-02-21 18:05:00,054+0000",
"service": "os-loader",
"cold_start": false,
"function_name": "OpenSearchSiemStack-SiemProcessorB1FDF325-OFFrKfdLmfiP",
"function_memory_size": "512",
"function_arn": "arn:aws:lambda:Region-1:Account ID:function:OpenSearchSiemStack-SiemProcessor",
"s3_key": "CloudWatchLogs/vpcflowlogs/2024/02/21/18/PBMMAccel-Firehose-Delivery-Stream-Partition-1-2024-02-21-18-03-12-359e52ca"
}
Attempts to Resolve & Questions:
I've tried multiple configurations for the s3_key to ensure it points to the correct VPC Flow Logs, without success.
Is there a specific format or preprocessing required for VPC Flow Logs to be compatible with this SIEM solution?
I've recently deployed a SIEM solution on OpenSearch using the AWS Secure Environment Accelerator, specifically following the guidance and resources provided in this repository https://github.com/aws-samples/aws-secure-environment-accelerator/tree/32ee10c50d0489a418888a5bddda7af0e2b9a3c8/reference-artifacts/Add-ons/opensiem .
Problem:
While the setup appears to be correctly ingesting various types of logs, I'm encountering issues specifically with VPC Flow Logs. Despite adjusting the s3_key to point to the appropriate location of the VPC Flow Logs in S3, the logs either fail to be ingested, or I encounter errors. The most telling feedback I've received is a warning message indicating that no entries were successfully loaded:
{ "level": "WARNING", "message": "No entries were successed to load", "location": "process_records:346", "timestamp": "2024-02-21 18:05:00,054+0000", "service": "os-loader", "cold_start": false, "function_name": "OpenSearchSiemStack-SiemProcessorB1FDF325-OFFrKfdLmfiP", "function_memory_size": "512", "function_arn": "arn:aws:lambda:Region-1:Account ID:function:OpenSearchSiemStack-SiemProcessor", "s3_key": "CloudWatchLogs/vpcflowlogs/2024/02/21/18/PBMMAccel-Firehose-Delivery-Stream-Partition-1-2024-02-21-18-03-12-359e52ca" } Attempts to Resolve & Questions:
I've tried multiple configurations for the s3_key to ensure it points to the correct VPC Flow Logs, without success. Is there a specific format or preprocessing required for VPC Flow Logs to be compatible with this SIEM solution?