[AWS][Enhancement] Review AWS `request_parameters` and `response_elements` Values for Improvements

[terrancedejesus]

Overview

In the AWS integration, most of our detections, rely on the following fields.

• event.dataset
• event.provider
• event.action
• event.outcome
• aws.cloudtrail.request_parameters
• aws.cloudtrail.response_elements

The creation of this issue is to focus on the aws.cloudtrail.request_parameters and aws.cloudtrail.response_elements field. Most of the activity we actively capture with the AWS integrations are API requests, where event.provider represents the AWS service (i.e. EC2, Lambda, S3, etc.) and event.action contains the API request (i.e. CreateRole, PutBucketPolicy, etc.). In the aws.cloudtrail.request_parameters and aws.cloudtrail.response_elements fields we have more context about the request and response. However, this field is currently a keyword field type or flattened (aws.cloudtrail.flattened.request_parameters).

Challenges

• These fields have a length max that does not allow us to search in some circumstances as shown in the screenshot below.
• Since the value of these fields are nested JSON objects, querying the data requires us to rely on wildcard searches which is not ideal for efficacy or performance

There are a ideal outcomes that would enable better threat detection rules and their respective queries:

• For the purpose of EQL and ES|QL, it would be ideal to somehow be able to query on any of the key:value pairs in this field. Aggregations and sequences have to be on specific field names. The difficult situation we often run into is that we could have multiple docs/events that detail API requests and responses between services with unique EC2 instance IDs, ARNs, Session IDs, etc., however these are buried in request_parameters.
• At least increase the max length of aws.cloudtrail.request_parameters. It is not a consistent problem that I have ran into, but depends on the request itself. For example, when PEM file contents are sent with other parameters. Note that often we wildcard for values in here simply because they are not parsed out.

The challenging part of this is the randomness of the values in request_parameters. There is no consistent data structure, but I believe it is highly dependent on the event.provider and the AWS resource the API request is being sent to. For example, for ec2.amazonaws.com we can expect to have instanceId in this field nearly on every request.

I understand that this may be a difficult hurdle, but overcoming it would improve the efficacy and performance of our rules for AWS. I am happy to give access to my environment or anything else required for investigation into this. There may be some custom analyzer options here, but I will be honest in stating my lack of understanding when it comes to data ingestion and the analyzers for each integration.

Thank you in advance!

[terrancedejesus]

cc @jamiehynds

[tinnytintin10]

cc @eyalkraft @tehilashn

[imays11]

missing role.name

I'd like to follow-up with a specific ask here:

For events that include a role.name as part of the request_parameters, it would be hugely beneficial to pull this out into a separate field. Similar to how target.user_name and related.user are created as separate fields for events targeting another user. I've come across this with 'sts:assumeRole' and 'iam:AttachRolePolicy' event actions. There is no easy way to track what role.name is being targeted for those actions. This is necessary for threat hunting and alert triage purposes so that the targeted or referenced role.name can be compared against subsequent events with the same user.name. This will allow analyst to track the behaviors following potential compromise of an IAM role. I've attached a screenshot and events for a scenario where a highly privileged IAM poicy was attached to a role (captured as roleName in request_parameters), then that same role was assumed (captured in the role.arn of request_parameters), then that same assumed role did a follow-up action (captured under user.name field). It would be nice to be able to follow these behaviors by comparing a role.name field to that user.name field.

AttachRolePolicy Missing role.name

``` { "_index": ".ds-logs-aws.cloudtrail-default-2024.05.03-000001", "_id": "9fb2e514b6-000000000950", "_version": 1, "_score": 0, "_source": { "agent": { "name": "ip-172-31-19-10.us-east-2.compute.internal", "id": "62553910-c542-4dfb-8359-8269137b6a29", "ephemeral_id": "45b1dc0d-f05c-4091-8457-4535d26e3b48", "type": "filebeat", "version": "8.13.4" }, "log": { "file": { "path": "https://aws-cloudtrail-logs-us-east1-211125303337-f09b7768.s3.us-east-1.amazonaws.com/AWSLogs/o-f6ztlip47u/211125303337/CloudTrail/us-east-1/2024/05/31/211125303337_CloudTrail_us-east-1_20240531T1830Z_W43wFkZEBqHBUYaN.json.gz" }, "offset": 950 }, "elastic_agent": { "id": "62553910-c542-4dfb-8359-8269137b6a29", "version": "8.13.4", "snapshot": false }, "source": { "geo": { "continent_name": "North America", "region_iso_code": "US-GA", "city_name": "...", "country_iso_code": "US", "country_name": "United States", "region_name": "Georgia", "location": { "lon": ..., "lat": ... } }, "as": { "number": 7922, "organization": { "name": "COMCAST-7922" } }, "address": "...", "ip": "..." }, "tags": [ "preserve_original_event", "forwarded", "aws-cloudtrail" ], "cloud": { "region": "us-east-1", "account": { "id": "211125303337" } }, "input": { "type": "aws-s3" }, "@timestamp": "2024-05-31T18:27:41.000Z", "ecs": { "version": "8.0.0" }, "related": { "user": [ "admin-user" ] }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "aws.cloudtrail" }, "tls": { "cipher": "TLS_AES_128_GCM_SHA256", "client": { "server_name": "iam.amazonaws.com" }, "version": "1.3", "version_protocol": "tls" }, "event": { "agent_id_status": "verified", "ingested": "2024-05-31T18:31:19Z", "original": "{\"eventVersion\":\"1.09\",\"userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDATCKAMPAU636XQYGO4\",\"arn\":\"arn:aws:iam::211125303337:user/admin-user\",\"accountId\":\"211125303337\",\"accessKeyId\":\"AKIATCKAMPAU5NENWB5J\",\"userName\":\"admin-user\"},\"eventTime\":\"2024-05-31T18:27:41Z\",\"eventSource\":\"iam.amazonaws.com\",\"eventName\":\"AttachRolePolicy\",\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"...\",\"userAgent\":\"aws-cli/2.15.57 md/awscrt#0.19.19 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#partial md/command#iam.attach-role-policy\",\"requestParameters\":{\"roleName\":\"dev-role\",\"policyArn\":\"arn:aws:iam::aws:policy/AdministratorAccess\"},\"responseElements\":null,\"requestID\":\"ae4dc98b-6280-4069-bc8c-87b8ca6d0ab6\",\"eventID\":\"dbe2b919-53d6-42db-9d7f-96815b49447d\",\"readOnly\":false,\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"211125303337\",\"eventCategory\":\"Management\",\"tlsDetails\":{\"tlsVersion\":\"TLSv1.3\",\"cipherSuite\":\"TLS_AES_128_GCM_SHA256\",\"clientProvidedHostHeader\":\"iam.amazonaws.com\"}}", "provider": "iam.amazonaws.com", "created": "2024-05-31T18:31:09.352Z", "kind": "event", "action": "AttachRolePolicy", "id": "dbe2b919-53d6-42db-9d7f-96815b49447d", "type": [ "info" ], "dataset": "aws.cloudtrail", "outcome": "success" }, "aws": { "s3": { "bucket": { "name": "aws-cloudtrail-logs-us-east1-211125303337-f09b7768", "arn": "arn:aws:s3:::aws-cloudtrail-logs-us-east1-211125303337-f09b7768" }, "object": { "key": "AWSLogs/o-f6ztlip47u/211125303337/CloudTrail/us-east-1/2024/05/31/211125303337_CloudTrail_us-east-1_20240531T1830Z_W43wFkZEBqHBUYaN.json.gz" } }, "cloudtrail": { "event_version": "1.09", "flattened": { "request_parameters": { "policyArn": "arn:aws:iam::aws:policy/AdministratorAccess", "roleName": "dev-role" } }, "event_type": "AwsApiCall", "read_only": false, "user_identity": { "access_key_id": "AKIATCKAMPAU5NENWB5J", "type": "IAMUser", "arn": "arn:aws:iam::211125303337:user/admin-user" }, "recipient_account_id": "211125303337", "event_category": "Management", "request_parameters": "{policyArn=arn:aws:iam::aws:policy/AdministratorAccess, roleName=dev-role}", "request_id": "ae4dc98b-6280-4069-bc8c-87b8ca6d0ab6", "management_event": true } }, "user": { "name": "admin-user", "id": "AIDATCKAMPAU636XQYGO4" }, "user_agent": { "original": "aws-cli/2.15.57 md/awscrt#0.19.19 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#partial md/command#iam.attach-role-policy", "name": "aws-cli", "device": { "name": "Other" }, "version": "2.15.57" } }, "fields": { "@timestamp": [ "2024-05-31T18:27:41.000Z" ], "event.created": [ "2024-05-31T18:31:09.352Z" ] } } ```

AssumeRole Missing role.name

``` { "_index": ".ds-logs-aws.cloudtrail-default-2024.05.03-000001", "_id": "f40f6eab7a-000000000950", "_version": 1, "_score": 0, "_source": { "agent": { "name": "ip-172-31-19-10.us-east-2.compute.internal", "id": "62553910-c542-4dfb-8359-8269137b6a29", "type": "filebeat", "ephemeral_id": "45b1dc0d-f05c-4091-8457-4535d26e3b48", "version": "8.13.4" }, "log": { "file": { "path": "https://aws-cloudtrail-logs-us-east1-211125303337-f09b7768.s3.us-east-1.amazonaws.com/AWSLogs/o-f6ztlip47u/211125303337/CloudTrail/us-east-1/2024/05/31/211125303337_CloudTrail_us-east-1_20240531T1830Z_qHtAe3jTVmOAXsfX.json.gz" }, "offset": 950 }, "elastic_agent": { "id": "62553910-c542-4dfb-8359-8269137b6a29", "version": "8.13.4", "snapshot": false }, "source": { "geo": { "continent_name": "North America", "region_iso_code": "US-GA", "city_name": "...", "country_iso_code": "US", "country_name": "United States", "region_name": "Georgia", "location": { "lon": ..., "lat": ... } }, "as": { "number": 7922, "organization": { "name": "COMCAST-7922" } }, "address": "...", "ip": "..." }, "tags": [ "preserve_original_event", "forwarded", "aws-cloudtrail" ], "cloud": { "region": "us-east-1", "account": { "id": "211125303337" } }, "input": { "type": "aws-s3" }, "@timestamp": "2024-05-31T18:29:13.000Z", "ecs": { "version": "8.0.0" }, "related": { "user": [ "admin-user" ] }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "aws.cloudtrail" }, "tls": { "cipher": "TLS_AES_128_GCM_SHA256", "client": { "server_name": "sts.us-east-1.amazonaws.com" }, "version": "1.3", "version_protocol": "tls" }, "aws": { "s3": { "bucket": { "name": "aws-cloudtrail-logs-us-east1-211125303337-f09b7768", "arn": "arn:aws:s3:::aws-cloudtrail-logs-us-east1-211125303337-f09b7768" }, "object": { "key": "AWSLogs/o-f6ztlip47u/211125303337/CloudTrail/us-east-1/2024/05/31/211125303337_CloudTrail_us-east-1_20240531T1830Z_qHtAe3jTVmOAXsfX.json.gz" } }, "cloudtrail": { "event_version": "1.08", "flattened": { "request_parameters": { "roleArn": "arn:aws:iam::211125303337:role/dev-role", "roleSessionName": "priv_esc" }, "response_elements": { "assumedRoleUser": { "assumedRoleId": "AROATCKAMPAUSLKMVY7XX:priv_esc", "arn": "arn:aws:sts::211125303337:assumed-role/dev-role/priv_esc" }, "credentials": { "accessKeyId": "ASIATCKAMPAU3QJITX3L", "sessionToken": "IQoJb3JpZ2luX2VjEMP//////////wEaCXVzLWVhc3QtMSJGMEQCIAm69035MmE8NEFrMkbalA0Xde0WPWYJHjWoBLGlALrJAiAuO7YuIxDO6CyV8XwtpZfkq/Kv1AfrCAYzU1AaM/GViyqVAghMEAAaDDIxMTEyNTMwMzMzNyIMan2H7sXVscvr/U7vKvIBR1sW6y/BGV3X/AxiKsdFw7Jyc1oGHD3UOBEy0HZKPamJFglgB2nvR3mlQs/GucnmGRb+B5ry6T57htY2WWsrheigqj5AH1naygaapI9GWQ2W3fFBKIcgHUXXEhIbnfU+UAvkrApzUrpZ/eSrztL+VPdXel6YsqLpRi5RN51cttl8z/WDQizkz8KghqCCiL4z73dZ3YWiSqbNd5s/LTaBP4DSJ0Z8WeMyoDdyuPNS+s9wPzn26/P5LdmLSr6FpphxIJafrF95tnsN05iDaM0v790UmabgBbZkd00fAO9yFVSk0tW2f0isizX1W4IIcvFXHOEw+a3osgY6ngH8UNh7yH5CLx2Ddbir3kPjCR8njYOBgI1HKbWDfg3p9DslRMOLrgD8HUpGGm8Wb6avLtsnfoymwvrhLTZKoxUlSjLZev+iHpn/F8zcsLmEH/7oGnrUeIp9gC52h34tBH3uk7unXq6zxlDwWwCo5l+wPUg8CEbGpsHUH90V8eSzhQUQ4fbjrCs0lalXM2fZUbX74CKeJoSesV3S4pLgAQ==", "expiration": "May 31, 2024, 7:29:13 PM" } } }, "event_type": "AwsApiCall", "read_only": true, "user_identity": { "access_key_id": "AKIATCKAMPAU5NENWB5J", "type": "IAMUser", "arn": "arn:aws:iam::211125303337:user/admin-user" }, "recipient_account_id": "211125303337", "event_category": "Management", "request_parameters": "{roleArn=arn:aws:iam::211125303337:role/dev-role, roleSessionName=priv_esc}", "request_id": "763edb9c-a4b1-4b17-a85e-9aa7b27f38ab", "response_elements": "{assumedRoleUser={assumedRoleId=AROATCKAMPAUSLKMVY7XX:priv_esc, arn=arn:aws:sts::211125303337:assumed-role/dev-role/priv_esc}, credentials={accessKeyId=ASIATCKAMPAU3QJITX3L, sessionToken=IQoJb3JpZ2luX2VjEMP//////////wEaCXVzLWVhc3QtMSJGMEQCIAm69035MmE8NEFrMkbalA0Xde0WPWYJHjWoBLGlALrJAiAuO7YuIxDO6CyV8XwtpZfkq/Kv1AfrCAYzU1AaM/GViyqVAghMEAAaDDIxMTEyNTMwMzMzNyIMan2H7sXVscvr/U7vKvIBR1sW6y/BGV3X/AxiKsdFw7Jyc1oGHD3UOBEy0HZKPamJFglgB2nvR3mlQs/GucnmGRb+B5ry6T57htY2WWsrheigqj5AH1naygaapI9GWQ2W3fFBKIcgHUXXEhIbnfU+UAvkrApzUrpZ/eSrztL+VPdXel6YsqLpRi5RN51cttl8z/WDQizkz8KghqCCiL4z73dZ3YWiSqbNd5s/LTaBP4DSJ0Z8WeMyoDdyuPNS+s9wPzn26/P5LdmLSr6FpphxIJafrF95tnsN05iDaM0v790UmabgBbZkd00fAO9yFVSk0tW2f0isizX1W4IIcvFXHOEw+a3osgY6ngH8UNh7yH5CLx2Ddbir3kPjCR8njYOBgI1HKbWDfg3p9DslRMOLrgD8HUpGGm8Wb6avLtsnfoymwvrhLTZKoxUlSjLZev+iHpn/F8zcsLmEH/7oGnrUeIp9gC52h34tBH3uk7unXq6zxlDwWwCo5l+wPUg8CEbGpsHUH90V8eSzhQUQ4fbjrCs0lalXM2fZUbX74CKeJoSesV3S4pLgAQ==, expiration=May 31, 2024, 7:29:13 PM}}", "management_event": true } }, "event": { "agent_id_status": "verified", "ingested": "2024-05-31T18:31:19Z", "original": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDATCKAMPAU636XQYGO4\",\"arn\":\"arn:aws:iam::211125303337:user/admin-user\",\"accountId\":\"211125303337\",\"accessKeyId\":\"AKIATCKAMPAU5NENWB5J\",\"userName\":\"admin-user\"},\"eventTime\":\"2024-05-31T18:29:13Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"...\",\"userAgent\":\"aws-cli/2.15.57 md/awscrt#0.19.19 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#partial md/command#sts.assume-role\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::211125303337:role/dev-role\",\"roleSessionName\":\"priv_esc\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"ASIATCKAMPAU3QJITX3L\",\"sessionToken\":\"IQoJb3JpZ2luX2VjEMP//////////wEaCXVzLWVhc3QtMSJGMEQCIAm69035MmE8NEFrMkbalA0Xde0WPWYJHjWoBLGlALrJAiAuO7YuIxDO6CyV8XwtpZfkq/Kv1AfrCAYzU1AaM/GViyqVAghMEAAaDDIxMTEyNTMwMzMzNyIMan2H7sXVscvr/U7vKvIBR1sW6y/BGV3X/AxiKsdFw7Jyc1oGHD3UOBEy0HZKPamJFglgB2nvR3mlQs/GucnmGRb+B5ry6T57htY2WWsrheigqj5AH1naygaapI9GWQ2W3fFBKIcgHUXXEhIbnfU+UAvkrApzUrpZ/eSrztL+VPdXel6YsqLpRi5RN51cttl8z/WDQizkz8KghqCCiL4z73dZ3YWiSqbNd5s/LTaBP4DSJ0Z8WeMyoDdyuPNS+s9wPzn26/P5LdmLSr6FpphxIJafrF95tnsN05iDaM0v790UmabgBbZkd00fAO9yFVSk0tW2f0isizX1W4IIcvFXHOEw+a3osgY6ngH8UNh7yH5CLx2Ddbir3kPjCR8njYOBgI1HKbWDfg3p9DslRMOLrgD8HUpGGm8Wb6avLtsnfoymwvrhLTZKoxUlSjLZev+iHpn/F8zcsLmEH/7oGnrUeIp9gC52h34tBH3uk7unXq6zxlDwWwCo5l+wPUg8CEbGpsHUH90V8eSzhQUQ4fbjrCs0lalXM2fZUbX74CKeJoSesV3S4pLgAQ==\",\"expiration\":\"May 31, 2024, 7:29:13 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROATCKAMPAUSLKMVY7XX:priv_esc\",\"arn\":\"arn:aws:sts::211125303337:assumed-role/dev-role/priv_esc\"}},\"requestID\":\"763edb9c-a4b1-4b17-a85e-9aa7b27f38ab\",\"eventID\":\"4a34f87a-da45-4a3d-a8ba-e40f5bc96bb7\",\"readOnly\":true,\"resources\":[{\"accountId\":\"211125303337\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::211125303337:role/dev-role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"211125303337\",\"eventCategory\":\"Management\",\"tlsDetails\":{\"tlsVersion\":\"TLSv1.3\",\"cipherSuite\":\"TLS_AES_128_GCM_SHA256\",\"clientProvidedHostHeader\":\"sts.us-east-1.amazonaws.com\"}}", "provider": "sts.amazonaws.com", "created": "2024-05-31T18:31:16.492Z", "kind": "event", "action": "AssumeRole", "id": "4a34f87a-da45-4a3d-a8ba-e40f5bc96bb7", "type": [ "info" ], "category": [ "authentication" ], "dataset": "aws.cloudtrail", "outcome": "success" }, "user": { "name": "admin-user", "id": "AIDATCKAMPAU636XQYGO4" }, "user_agent": { "original": "aws-cli/2.15.57 md/awscrt#0.19.19 ua/2.0 os/macos#23.5.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#partial md/command#sts.assume-role", "name": "aws-cli", "device": { "name": "Other" }, "version": "2.15.57" } }, "fields": { "@timestamp": [ "2024-05-31T18:29:13.000Z" ], "event.created": [ "2024-05-31T18:31:16.492Z" ] } } ```

[romulets]

Hey @terrancedejesus, I'm looking into the cloudtrail integration (from another perspective) and I saw this issue.

Regarding this point:

Since the value of these fields are nested JSON objects, querying the data requires us to rely on wildcard searches which is not ideal for efficacy or performance

Don't the flattened fields aws.cloudtrail.flattened.response_elements and aws.cloudtrail.flattened.request_parameters enable you to query without relying on string processing and wildcards?

I've been doing it and it seems to suffice my needs. I wonder if it also fulfils your needs.

Below you can see a query I've done:

event.dataset: "aws.cloudtrail" and event.action: "AssumeRole"  and aws.cloudtrail.flattened.response_elements.assumedRoleUser.assumedRoleId: "AROA2IBR2EZTCIK7YJG3E:TrustedAdvisor_704479110758_65cf4c6f-b374-4c58-8ea8-2285b3af904e"