We found aes-siem-es-loader giving below error message when loading some of the WAF log records to OpenSearch:
{
"level": "ERROR",
"message": "1 of logs were NOT loaded into OpenSearch Service",
"location": "process_record:545",
"timestamp": "2024-02-28 08:04:48,438+0000",
"service": "es-loader",
"cold_start": false,
"function_name": "aes-siem-es-loader",
"function_memory_size": "2048",
"function_arn": "arn:aws:lambda:us-east-1:1234567890:function:aes-siem-es-loader",
"function_request_id": "72a18220-fdaa-4222-8968-a602fca37f6d",
"s3_key": "AWSLogs/1234567890/WAF/ap-east-1/2024/02/28/08/aws-waf-logs-xxx-1-2024-02-28-08-03-40-5996b31a-9556-4bc1-9730-8983ecf1690a.gz",
"s3_bucket": "aes-siem-1234567890-log",
"message_error": [
{
"type": "mapper_parsing_exception",
"reason": "object mapping for [ruleGroupList.nonTerminatingMatchingRules.ruleMatchDetails.matchedData] tried to parse field [null] as object, but found a concrete value",
"log_number": 42
}
],
"xray_trace_id": "1-65dee91f-6087a22c081977d222e15070"
}
Content of log number 42 is as below:
{"timestamp":1709108635066,"formatVersion":1,"webaclId":"arn:aws:wafv2:ap-east-1:1234567890:regional/webacl/xxx/b954fa85-5178-4201-8948-221e9bf0d032","terminatingRuleId":"AWS-AWSManagedRulesCommonRuleSet","terminatingRuleType":"MANAGED_RULE_GROUP","action":"BLOCK","terminatingRuleMatchDetails":[{"conditionType":"XSS","location":"BODY","matchedData":["<?","xml"],"matchedFieldName":""}],"httpSourceName":"ALB","httpSourceId":"1234567890-app/qa-xxx-alb-new/72afb40a799b993a","ruleGroupList":[{"ruleGroupId":"AWS#AWSManagedRulesAmazonIpReputationList","terminatingRule":null,"nonTerminatingMatchingRules":[],"excludedRules":null,"customerConfig":null},{"ruleGroupId":"AWS#AWSManagedRulesCommonRuleSet","terminatingRule":{"ruleId":"CrossSiteScripting_BODY","action":"BLOCK","ruleMatchDetails":null},"nonTerminatingMatchingRules":[{"ruleId":"CrossSiteScripting_BODY_RC_COUNT","action":"COUNT","ruleMatchDetails":[{"conditionType":"XSS","location":"BODY","matchedData":["<?","xml"],"matchedFieldName":""}]},{"ruleId":"SizeRestrictions_BODY","action":"COUNT","overriddenAction":"BLOCK","ruleMatchDetails":[]}],"excludedRules":null,"customerConfig":null}],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":null,"httpRequest":{"clientIp":"[103.60.248.229](http://103.60.248.229/)","country":"HK","headers":[{"name":"Cache-control","value":"no-cache"},{"name":"Cache-store","value":"no-store"},{"name":"Pragma","value":"no-cache"},{"name":"User-Agent","value":"Apache-Maven/3.6.3 (Java 1.8.0_181; Mac OS X 10.16)"},{"name":"Content-Length","value":"10492"},{"name":"Host","value":"xxx.xxx.global"},{"name":"Connection","value":"Keep-Alive"},{"name":"Expect","value":"100-continue"},{"name":"Accept-Encoding","value":"gzip,deflate"},{"name":"Authorization","value":"Basic YWRtaW46b3ZlcnNlYXM="}],"uri":"/repository/maven-snapshots/com/kun/kun-dependencies/1.0-SNAPSHOT/kun-dependencies-1.0-20240228.082354-59.pom","args":"","httpVersion":"HTTP/1.1","httpMethod":"PUT","requestId":"1-65deed9a-4d373a463405c2ed763a5638"},"labels":[{"name":"awswaf:managed:aws:core-rule-set:CrossSiteScripting_Body_RC_COUNT"},{"name":"awswaf:managed:aws:core-rule-set:CrossSiteScripting_Body"},{"name":"awswaf:managed:aws:core-rule-set:SizeRestrictions_Body"}],"oversizeFields":["REQUEST_BODY"],"requestBodySize":10492,"requestBodySizeInspectedByWAF":8192}
We found aes-siem-es-loader giving below error message when loading some of the WAF log records to OpenSearch:
Content of log number 42 is as below:
This seems to be a bug.