GuardDuty fails deserialization when ProcessRuntime euid contains a string

liranms commented 9 months ago

Describe the bug

GetFindings() client tries to deserialize a finding that contains the following path: service.runtimeDetails.process.euid (or userId) when the value of that path is a string "1000" instead of an integer, it fails with the following error:

SerializationError: failed decoding JSON RPC response
    status code: 200, request id: effa89f1-8ec6-458d-8c8f-dfea13c589ca
caused by: unsupported value: <nil> (*int64)

Expected Behavior

successfully deserialize the guardduty.Finding

Current Behavior

fails to serialize output, the following error is returned:

SerializationError: failed decoding JSON RPC response
    status code: 200, request id: effa89f1-8ec6-458d-8c8f-dfea13c589ca
caused by: unsupported value: <nil> (*int64)

Reproduction Steps

Just call GetFindings() on a detector ID that returns a finding with "runtimeDetails" that contains a "process" node, for example: "Trojan:Runtime/BlackholeTraffic!DNS" sample

Here's a sample JSON finding as returned by the server:

{
            "schemaVersion": "2.0",
            "accountId": "123123123123",
            "region": "eu-north-1",
            "partition": "aws",
            "id": "11111111111111111111111111111111",
            "arn": "arn:aws:guardduty:eu-north-1:123123123123:detector/22222222222222222222222222222222/finding/33333333333333333333333333333333",
            "type": "Trojan:Runtime/BlackholeTraffic!DNS",
            "resource": {
                "resourceType": "EKSCluster",
                "eksClusterDetails": {
                    "name": "GeneratedFindingEKSClusterName",
                    "arn": "GeneratedFindingEKSClusterArn",
                    "createdAt": 1636625755.218,
                    "vpcId": "GeneratedFindingEKSClusterVpcId",
                    "status": "ACTIVE",
                    "tags": [
                        {
                            "value": "GeneratedFindingEKSClusterTagValue1",
                            "key": "GeneratedFindingEKSClusterTag1"
                        },
                        {
                            "value": "GeneratedFindingEKSClusterTagValue2",
                            "key": "GeneratedFindingEKSClusterTag2"
                        },
                        {
                            "value": "GeneratedFindingEKSClusterTagValue3",
                            "key": "GeneratedFindingEKSClusterTag3"
                        }
                    ]
                },
                "kubernetesDetails": {
                    "kubernetesWorkloadDetails": {
                        "name": "GeneratedFindingKubernetesWorkloadName",
                        "namespace": "GeneratedFindingKubernetesWorkloadNamespace",
                        "type": "GeneratedFindingKubernetesWorkloadType",
                        "uid": "GeneratedFindingKubernetesWorkloadID"
                    }
                },
                "containerDetails": {
                    "id": "GeneratedFindingContainerId",
                    "name": "GeneratedFindingContainerName",
                    "image": "GeneratedFindingContainerImage"
                },
                "instanceDetails": {
                    "instanceId": "i-99999999",
                    "instanceType": "m3.xlarge",
                    "outpostArn": "arn:aws:outposts:us-west-2:123123123123:outpost/op-0fffffffffffffffff",
                    "launchTime": "2016-08-02T02:05:06.000Z",
                    "platform": null,
                    "productCodes": [
                        {
                            "productCodeId": "GeneratedFindingProductCodeId",
                            "productCodeType": "GeneratedFindingProductCodeType"
                        }
                    ],
                    "iamInstanceProfile": {
                        "arn": "GeneratedFindingInstanceProfileArn",
                        "id": "GeneratedFindingInstanceProfileId"
                    },
                    "networkInterfaces": [
                        {
                            "networkInterfaceId": "eni-ffffffff",
                            "privateIpAddresses": [
                                {
                                    "privateDnsName": "GeneratedFindingPrivateName",
                                    "privateIpAddress": "10.0.0.1"
                                }
                            ],
                            "subnetId": "GeneratedFindingSubnetId",
                            "vpcId": "GeneratedFindingVPCId",
                            "privateDnsName": "GeneratedFindingPrivateDnsName",
                            "securityGroups": [
                                {
                                    "groupName": "GeneratedFindingSecurityGroupName",
                                    "groupId": "GeneratedFindingSecurityId"
                                }
                            ],
                            "publicIp": "123.123.123.123",
                            "ipv6Addresses": [],
                            "publicDnsName": "GeneratedFindingPublicDNSName",
                            "privateIpAddress": "10.0.0.1"
                        }
                    ],
                    "tags": [
                        {
                            "value": "GeneratedFindingInstanceValue1",
                            "key": "GeneratedFindingInstanceTag1"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue2",
                            "key": "GeneratedFindingInstanceTag2"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue3",
                            "key": "GeneratedFindingInstanceTag3"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue4",
                            "key": "GeneratedFindingInstanceTag4"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue5",
                            "key": "GeneratedFindingInstanceTag5"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue6",
                            "key": "GeneratedFindingInstanceTag6"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue7",
                            "key": "GeneratedFindingInstanceTag7"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue8",
                            "key": "GeneratedFindingInstanceTag8"
                        },
                        {
                            "value": "GeneratedFindingInstanceTagValue9",
                            "key": "GeneratedFindingInstanceTag9"
                        }
                    ],
                    "instanceState": "running",
                    "availabilityZone": "GeneratedFindingInstanceAvailabilityZone",
                    "imageId": "ami-99999999",
                    "imageDescription": "GeneratedFindingInstanceImageDescription"
                }
            },
            "service": {
                "serviceName": "guardduty",
                "detectorId": "2ac55c1ccdd5502a6beb81c4a80742e8",
                "action": {
                    "actionType": "DNS_REQUEST",
                    "dnsRequestAction": {
                        "domain": "GeneratedFindingDomainName",
                        "protocol": "UDP",
                        "blocked": false
                    }
                },
                "runtimeDetails": {
                    "process": {
                        "pid": 1234,
                        "name": "GeneratedFindingProcessName",
                        "uuid": "GeneratedFindingUUId",
                        "executablePath": "GeneratedFindingPath",
                        "executableSha256": "GeneratedFindingHash",
                        "cmdLine": "GeneratedFindingCommandLine",
                        "user": "ec2-user",
                        "euid": "1000",
                        "userId": "1000",
                        "pwd": "GeneratedFindingPath",
                        "startTime": 1637012294,
                        "parentUuid": "GeneratedFindingUUId",
                        "lineage": [
                            {
                                "executablePath": "GeneratedFindingPath",
                                "euid": 1000,
                                "pid": 1233,
                                "uuid": "GeneratedFindingUUId",
                                "parentUuid": "GeneratedFindingUUId"
                            },
                            {
                                "executablePath": "GeneratedFindingPath",
                                "euid": 1000,
                                "pid": 1230,
                                "uuid": "GeneratedFindingUUId",
                                "parentUuid": "GeneratedFindingUUId"
                            }
                        ]
                    }
                },
                "featureName": "RuntimeMonitoring",
                "resourceRole": "TARGET",
                "additionalInfo": {
                    "threatListName": "GeneratedFindingThreatListName",
                    "sample": true,
                    "agentDetails": {
                        "agentVersion": "1",
                        "agentId": "GeneratedFindingAgentId"
                    },
                    "value": "{\"threatListName\":\"GeneratedFindingThreatListName\",\"sample\":true,\"agentDetails\":{\"agentVersion\":\"1\",\"agentId\":\"GeneratedFindingAgentId\"}}",
                    "type": "default"
                },
                "evidence": {
                    "threatIntelligenceDetails": [
                        {
                            "threatNames": [
                                "GeneratedFindingThreatName"
                            ],
                            "threatListName": "GeneratedFindingThreatListName"
                        }
                    ]
                },
                "eventFirstSeen": "2023-09-21T15:05:16.000Z",
                "eventLastSeen": "2023-09-21T15:21:50.000Z",
                "archived": false,
                "count": 2
            },
            "severity": 5,
            "createdAt": "2023-09-21T15:05:16.502Z",
            "updatedAt": "2023-09-21T15:21:50.375Z",
            "title": "Blackholed domain name queried by EC2 instance i-99999999.",
            "description": "Process GeneratedFindingProcessName from EC2 instance i-99999999 is querying a blackholed domain name."
        }

Possible Solution

Allow deserializing strings as integers, or solve it with the API owner to never return a string instead of int

Additional Information/Context

when type ProcessDetails struct was introduced to the SDK on v1.44.233 on this PR aws/aws-sdk-go#4782 the problem seem to have surfaced, (although the API should return a json integer/number and not a string representation of it) It is possible to solve it by allowing numbers to be deserialized from strings on private/protocol/json/jsonutil/unmarshal.go

Downgrading toe SDK v1.44.232 "solves" the problem as well (as there is no attempt to deserialize to int64 from string)

SDK version used

1.45.16 (latest) down to 1.44.233 (including)

Environment details (Version of Go (`go version`)? OS name and version, etc.)

go version go1.20.5 windows/amd64

RanVaknin commented 9 months ago

Hi @liranms ,

If you did not know, all the AWS SDKs are code-generated from the API model definition of each AWS service.

In this case, AWS GuardDuty's userId and euid are modeled as an Integer, but it seems like the guard duty is returning back a string, causing the protocol deserializer to break.

So in simple terms, the service guarantees one thing, and then sends back a different thing. We cannot accept your PR because the whole point of protocol deserializers is to provide robust type checking.

I have created an internal ticket with the service team and there's a fix pending. Please stand by for updates.

Thanks, Ran~

github-actions[bot] commented 9 months ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

bentzy commented 9 months ago

Hi @RanVaknin Thanks for helping How can we follow up on this issue? Specifically we need to know when a version with a fix for this is released. Thanks!

RanVaknin commented 9 months ago

Hi @bentzy ,

The service team has reported a rollout on the backend, so it's not tied to an SDK version, its a change in server side behavior. I suggest you test your code again.

Thanks, Ran~

aws / aws-sdk