AWS Cloudtrail parsing failure

[alemairebe]

Hello, Lambda fail to process Cloudtrail event from inspector :

  "level": "ERROR",
  "message": "3 of logs were NOT loaded into OpenSearch Service",
  "location": "process_record:542",
  "timestamp": "2023-10-03 14:14:58,357+0000",
  "service": "service_undefined",
  "s3_key": "cloudtrail-AccountTrail/AWSLogs/ACCOUNT_ID/CloudTrail/us-east-1/2023/10/02/561107922129_CloudTrail_us-east-1_20231002T0015Z_pWygjyZu9PB0tJ26.json.gz",
  "s3_bucket": "my-log-bucket",
  "message_error": [
    {
      "type": "mapper_parsing_exception",
      "reason": "failed to parse field [apiVersion] of type [date] in document with id 'dd32ef67-aa73-405d-9a43-e0f2c171dcc7'. Preview of field's value: 'v20160216'",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "failed to parse date field [v20160216] with format [strict_date_optional_time||epoch_millis]",
        "caused_by": {
          "type": "date_time_parse_exception",
          "reason": "Failed to parse with all enclosed parsers"
        }
      },
      "log_number": 133
    },

the Cloudtrail event is the following :

    {
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "AssumedRole",
        "principalId": "MY_PRINCIPAL",
        "arn": "arn:aws:sts::SOME_ROLE",
        "accountId": "MY_ACCOUNT_ID",
        "accessKeyId": "REDACTED",
        "sessionContext": {
          "sessionIssuer": {
            "type": "Role",
            "principalId": "MY_PRINCIPAL",
            "arn": "arn:aws:iam::SOME_ROLE",
            "accountId": "MY_ACCOUNT_ID",
            "userName": "auditor"
          },
          "webIdFederationData": {},
          "attributes": {
            "creationDate": "2023-10-02T00:12:39Z",
            "mfaAuthenticated": "false"
          }
        }
      },
      "eventTime": "2023-10-02T00:12:40Z",
      "eventSource": "inspector.amazonaws.com",
      "eventName": "ListRulesPackages",
      "awsRegion": "us-east-1",
      "sourceIPAddress": "MY_IP",
      "userAgent": "aws-sdk-js/3.370.0 ua/2.0 os/linux#5.10.192-182.736.amzn2.x86_64 lang/js md/nodejs#18.17.1 api/inspector#3.370.0 exec-env/AWS_ECS_FARGATE",
      "requestParameters": {
        "maxResults": 50
      },
      "responseElements": null,
      "requestID": "3b83c9ec-f7bb-4286-a34e-aae8ce4fed98",
      "eventID": "dd32ef67-aa73-405d-9a43-e0f2c171dcc7",
      "readOnly": true,
      "eventType": "AwsApiCall",
      "apiVersion": "v20160216",
      "managementEvent": true,
      "recipientAccountId": "MY_ACCOUNT_ID",
      "eventCategory": "Management",
      "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "inspector.us-east-1.amazonaws.com"
      }
    },

Could there be a way to accept apiVersion to be prefixed by v ? Thanks!

[nakajiak]

Thanks for the feedback. We will investigate and fix it

[nakajiak]

Hi, what SIEM version do you use? In the error message, the cause is that apiVersion is defined as a date field. That field is defined as keyword from v2.1.1. If you are using an older version, please update. https://github.com/aws-samples/siem-on-amazon-opensearch-service/blob/v2.1.1/source/lambda/deploy_es/data.ini#L85

[alemairebe]

Hi, I use main branch , git commit d395e705f3b85e19fc38245b07424d628da40f5d ( after v2.10.1 )

[nakajiak]

Hi, I got it. Please let me know the result of this command in DevTool.

GET _index_template/log-aws-cloudtrail_aws

Do you see "component_template_log-aws"?

if yes, how about this command?

GET _component_template/component_template_log-aws

Then, can you find '"apiVersion": {"type": "keyword"}'?

Next,

GET log-aws-cloudtrail-2023-10

What values does apiVersion key have?

[alemairebe]

Hi, Sorry, I didn't knew I had to setup OpenSearch index templates before running the lambda. I added the index template and work as it should. Sorry to have wasted some of your time

[nakajiak]

Thanks for the feedback. I'm happy that the issue is solved. If you have any issue let me know again