The AWS Secure Environment Accelerator is a tool designed to help deploy and operate secure multi-account, multi-region AWS environments on an ongoing basis. The power of the solution is the configuration file which enables the completely automated deployment of customizable architectures within AWS without changing a single line of code.
Apache License 2.0
725
stars
233
forks
source link
[QUESTION] Issues Ingesting VPC Flow Logs into OpenSearch SIEM with AWS Secure Environment Accelerator #1211
While the setup appears to be correctly ingesting various types of logs, I'm encountering issues specifically with VPC Flow Logs. Despite adjusting the s3_key to point to the appropriate location of the VPC Flow Logs in S3, the logs either fail to be ingested, or I encounter errors. The most telling feedback I've received is a warning message indicating that no entries were successfully loaded:
{
"level": "WARNING",
"message": "No entries were successed to load",
"location": "process_records:346",
"timestamp": "2024-02-21 18:05:00,054+0000",
"service": "os-loader",
"cold_start": false,
"function_name": "OpenSearchSiemStack-SiemProcessorB1FDF325-OFFrKfdLmfiP",
"function_memory_size": "512",
"function_arn": "arn:aws:lambda:Region-1:Account ID:function:OpenSearchSiemStack-SiemProcessor",
"s3_key": "CloudWatchLogs/vpcflowlogs/2024/02/21/18/PBMMAccel-Firehose-Delivery-Stream-Partition-1-2024-02-21-18-03-12-359e52ca"
}
Attempts to Resolve & Questions:
What specific configurations should be applied to the s3_key to accurately reference VPC Flow Logs within the system?
In adjusting the aws.ini file to accommodate various log types for s3_key, are there particular considerations or parameters to be mindful of?
Could you clarify if there's a prescribed format or preprocessing methodology essential for ensuring compatibility of VPC Flow Logs with this SIEM solution?
I've recently deployed a SIEM solution on OpenSearch using the AWS Secure Environment Accelerator, specifically following the guidance and resources provided in this repository https://github.com/aws-samples/aws-secure-environment-accelerator/tree/32ee10c50d0489a418888a5bddda7af0e2b9a3c8/reference-artifacts/Add-ons/opensiem .
Problem:
While the setup appears to be correctly ingesting various types of logs, I'm encountering issues specifically with VPC Flow Logs. Despite adjusting the s3_key to point to the appropriate location of the VPC Flow Logs in S3, the logs either fail to be ingested, or I encounter errors. The most telling feedback I've received is a warning message indicating that no entries were successfully loaded:
{ "level": "WARNING", "message": "No entries were successed to load", "location": "process_records:346", "timestamp": "2024-02-21 18:05:00,054+0000", "service": "os-loader", "cold_start": false, "function_name": "OpenSearchSiemStack-SiemProcessorB1FDF325-OFFrKfdLmfiP", "function_memory_size": "512", "function_arn": "arn:aws:lambda:Region-1:Account ID:function:OpenSearchSiemStack-SiemProcessor", "s3_key": "CloudWatchLogs/vpcflowlogs/2024/02/21/18/PBMMAccel-Firehose-Delivery-Stream-Partition-1-2024-02-21-18-03-12-359e52ca" } Attempts to Resolve & Questions: What specific configurations should be applied to the s3_key to accurately reference VPC Flow Logs within the system? In adjusting the aws.ini file to accommodate various log types for s3_key, are there particular considerations or parameters to be mindful of? Could you clarify if there's a prescribed format or preprocessing methodology essential for ensuring compatibility of VPC Flow Logs with this SIEM solution?