[AWS] Network Firewall logs ingest pipeline duplicate field error

[ottramst]

Hey, guys!

Seeing an issue with AWS Network Firewall ingest pipeline specifically on the JSON processor. Using aws package version 1.29.0.

Example AWS Network Firewall log

{"firewall_name":"outbound","availability_zone":"eu-central-1a","event_timestamp":"1674225318","event":{"app_proto":"tls","src_ip":"10.125.94.186","src_port":5302,"event_type":"alert","alert":{"severity":3,"signature_id":4,"rev":0,"signature":"aws:alert_established action","action":"allowed","category":""},"flow_id":1829676028160453,"dest_ip":"123.123.123.123","proto":"TCP","tls":{"subject":"CN=sqs.eu-central-1.amazonaws.com","issuerdn":"C=US, O=Amazon, OU=Server CA 1B, CN=Amazon","subject":"CN=sqs.eu-central-1.amazonaws.com","issuerdn":"C=US, O=Amazon, OU=Server CA 1B, CN=Amazon","serial":"08:C9:76:68:CB:30:31:1C:B2:24:A4:B3:22:F0:16:29","fingerprint":"34:36:32:fb:05:65:83:55:a0:8e:24:7b:41:52:bf:98:88:3b:bf:9b","sni":"sqs.eu-central-1.amazonaws.com","version":"TLS 1.2","notbefore":"2022-11-03T00:00:00","notafter":"2023-10-16T23:59:59","ja3":{},"ja3s":{}},"dest_port":443,"timestamp":"2023-01-20T14:35:18.307095+0000"}}

Ingest pipeline fails with the next error:

{
  "root_cause": [
    {
      "type": "x_content_parse_exception",
      "reason": "[1:502] Duplicate field 'subject'\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 502]"
    }
  ],
  "type": "x_content_parse_exception",
  "reason": "[1:502] Duplicate field 'subject'\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 502]",
  "caused_by": {
    "type": "json_parse_exception",
    "reason": "Duplicate field 'subject'\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 502]"
  }
}

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!