aws-samples / siem-on-amazon-opensearch-service

A solution for collecting, correlating and visualizing multiple types of logs to help investigate security incidents.
MIT No Attribution
558 stars 185 forks source link

Object mapping for [requestParameters.map.groupBy] tried to parse field [groupBy] as object, but found a concrete value #447

Open FerFabbiano opened 3 months ago

FerFabbiano commented 3 months ago

Hi!

Found two issues related to the same problem.

  1. Object mapping for [requestParameters.map.groupBy] tried to parse field [groupBy] as object, but found a concrete value
  2. Object mapping for [requestParameters.tagSpecificationSet] tried to parse field [tagSpecificationSet] as object, but found a concrete value

In case number one, the error makes sense to me, because it's trying to map String as an object. Ok. In case number two, tagSpecificationSet is actually an object, so the error does not make sense to me. Why i this?

Following some of the issues you solved related to similar errors, i modified aes-siem-es-loader Lambda Function, adding the following lines to the file /siem/sf_cloudtrail.py (line 142). With this fix, error messages seems to dissapear, although i seriously doubt this is the correct fix because i can't see logs with neither requestParameters.tagSpecificationSet or requestParameters.map.groupBy fields.

# fix object mapping for [requestParameters.tagSpecificationSet] tried to parse field [tagSpecificationSet] as object, but found a concrete value
    elif event_source in ('ec2.amazonaws.com'):
        try:
            params = logdata['requestParameters']['tagSpecificationSet']
        except (KeyError, TypeError):
            params = None
        if params and isinstance(params, str):
            logdata['requestParameters']['tagSpecificationSet'] = {'value': params}

# fix object mapping for [requestParameters.map.groupBy] tried to parse field [groupBy] as object, but found a concrete value
    elif event_source in ('billingconsole.amazonaws.com'):
        try:
            params = logdata['requestParameters']['map']['groupBy']
        except (KeyError, TypeError):
            params = None
        if params and isinstance(params, str):
            logdata['requestParameters']['map']['groupBy'] = {'value': params}

Log examples are shown below, thanks in advance!

"tagSpecificationSet": {
                    "items": [
                        {
                            "resourceType": "instance",
                            "tags": [
                                {
                                    "key": "key",
                                    "value": "value"
                                },
                                {
                                    "key": "key",
                                    "value": "value"
                                },
                                {
                                    "key": "key",
                                    "value": "value"
                                },
                                {
                                    "key": "key",
                                    "value": "value"
                                },
                                {
                                    "key": "key",
                                    "value": "value"
                                },
                                {
                                    "key": "key",
                                    "value": "value"
                                },
                                {
                                    "key": "key",
                                    "value": "value"
                                }
                            ]
                        }
                    ]
                },
          "requestParameters": {
                "map": {
                    "month": "4",
                    "year": "2024",
                    "groupBy": "ServiceProvider"
                }
            },
FerFabbiano commented 3 months ago

Add new error

Object mapping for [requestParameters.operations] tried to parse field [null] as object, but found a concrete value

Log example

"requestParameters": {
                "constraints": {
                    "encryptionContextEquals": {
                        "aws:rds:src-db-arn": "...",
                        "aws:rds:snap-id": "...",
                        "aws:ebs:id": "..."
                    }
                },
                "operations": [
                    "Decrypt"
                ],