GuardDuty S3BucketDetail.createdAt unmarshal error time.Time expected but epoch timestamp received

max-frank commented 1 year ago

Describe the bug

Unmarshalling GuardDuty S3Bucket evens with the SDK fails since the s3BucketDetail.createdAt field is set as time.Time, but actual GuardDuty events have this field set as an number epoch timestamp. The golang default JSON unmarshaler for time.Time can only handle string timestamps.

https://github.com/aws/aws-sdk-go-v2/blob/main/service/guardduty/types/types.go#L2297

As far as I can tell this affects at least all S3Bucket related events though I am not sure if

This is a bug in the SDK due to the struct using the wrong type/unmarshaling or if parts of the GuardDuty events publish incorrect timestamp formats. As most other timestamps are in ISO format.
If any other resource type is afflicted by a similar bug

Expected Behavior

Able to unmarshal GurdDuty S3Bucket findings

Current Behavior

Trying to unmarshal will give you due to encountering number timestamps were string timestamps are expected

unable to unmarshal: Time.UnmarshalJSON: input is not a JSON string

Reproduction Steps

// You can edit this code!
// Click here and start typing.
package main

import (
    "encoding/json"

    guardduty "github.com/aws/aws-sdk-go-v2/service/guardduty/types"
)

func main() {

    event := []byte(`
{
    "schemaVersion": "2.0",
    "accountId": "redacted",
    "region": "redacted",
    "partition": "aws",
    "id": "redacted",
    "arn": "arn:aws:guardduty:redacted",
    "type": "Impact:S3/AnomalousBehavior.Write",
    "resource": {
        "resourceType": "S3Bucket",
        "accessKeyDetails": {
            "accessKeyId": "redacted",
            "principalId": "redacted",
            "userType": "IAMUser",
            "userName": "redacted"
        },
        "s3BucketDetails": [
            {
                "arn": "redacted",
                "name": "redacted",
                "defaultServerSideEncryption": {
                    "encryptionType": "AES256",
                    "kmsMasterKeyArn": null
                },
                "createdAt": 1.592602041E9,
                "tags": [],
                "owner": {
                    "id": "redacted"
                },
                "publicAccess": {
                    "permissionConfiguration": {
                        "bucketLevelPermissions": {
                            "accessControlList": {
                                "allowsPublicReadAccess": false,
                                "allowsPublicWriteAccess": false
                            },
                            "bucketPolicy": {
                                "allowsPublicReadAccess": false,
                                "allowsPublicWriteAccess": false
                            },
                            "blockPublicAccess": {
                                "ignorePublicAcls": true,
                                "restrictPublicBuckets": true,
                                "blockPublicAcls": true,
                                "blockPublicPolicy": true
                            }
                        },
                        "accountLevelPermissions": {
                            "blockPublicAccess": {
                                "ignorePublicAcls": false,
                                "restrictPublicBuckets": false,
                                "blockPublicAcls": false,
                                "blockPublicPolicy": false
                            }
                        }
                    },
                    "effectivePermission": "NOT_PUBLIC"
                },
                "type": "Destination"
            }
        ]
    },
    "service": {
        "serviceName": "guardduty",
        "detectorId": "redacted",
        "action": {
            "actionType": "AWS_API_CALL",
            "awsApiCallAction": {
                "api": "CreateMultipartUpload",
                "serviceName": "s3.amazonaws.com",
                "callerType": "Remote IP",
                "remoteIpDetails": {
                    "ipAddressV4": "1.2.3.4",
                    "organization": {
                        "asn": "redacted",
                        "asnOrg": "redacted",
                        "isp": "Amazon.com",
                        "org": "Amazon.com"
                    },
                    "country": {
                        "countryName": "redacted"
                    },
                    "city": {
                        "cityName": "redacted"
                    },
                    "geoLocation": {
                        "lat": 0,
                        "lon": 0
                    }
                },
                "affectedResources": {
                    "AWS::S3::Bucket": "redacted"
                }
            }
        },
        "resourceRole": "TARGET",
        "additionalInfo": {
            "userAgent": {
                "fullUserAgent": "[redacted]",
                "userAgentCategory": "redacted"
            },
            "authenticationMethod": "AuthHeader",
            "anomalies": {
                "anomalousAPIs": "s3.amazonaws.com:[CreateMultipartUpload:success]"
            },
            "profiledBehavior": {
            },
            "unusualBehavior": {
                "unusualAPIsAccountProfiling": "",
                "unusualUserTypesAccountProfiling": "",
                "unusualUserNamesAccountProfiling": "",
                "unusualASNsAccountProfiling": "",
                "unusualUserAgentsAccountProfiling": "aws-sdk-ruby3",
                "unusualBucketsAccountProfiling": "",
                "unusualAPIsUserIdentityProfiling": "",
                "unusualUserNamesBucketProfiling": "",
                "unusualASNsUserIdentityProfiling": "",
                "unusualASNsBucketProfiling": "",
                "unusualUserAgentsUserIdentityProfiling": "aws-sdk-ruby3",
                "unusualBucketsUserIdentityProfiling": "",
                "isUnusualUserIdentity": "false"
            },
            "value": "",
            "type": "default"
        },
        "eventFirstSeen": "2023-06-01T02:47:27.000Z",
        "eventLastSeen": "2023-06-01T02:47:27.000Z",
        "archived": false,
        "count": 1
    },
    "severity": 5,
    "createdAt": "2023-06-01T02:57:05.015Z",
    "updatedAt": "2023-06-01T02:57:05.015Z",
    "title": "An IAM entity invoked an S3 API that attempts exporting data in a suspicious way.",
    "description": "Principal redacted wrote objects to S3 bucket redacted in an unusual way."
}`)
    var f guardduty.Finding
    if err := json.Unmarshal(event, &f); err != nil {
        panic(err)
    }
}

Possible Solution

Use custom timestamp format that can either unmarshal just epoch time or both ISO and epoch time e.g.,

func (t Timestamp) MarshalJSON() ([]byte, error) {
    if t.rfc3339 {
        return t.Time.MarshalJSON()
    }
    return t.formatUnix()
}

func (t *Timestamp) UnmarshalJSON(data []byte) error {
    err := t.Time.UnmarshalJSON(data)
    if err != nil {
        return t.parseUnix(data)
    }
    t.rfc3339 = true
    return nil
}

or simply a number type (must support floats since float seem to also be returned)

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2/service/guardduty v1.18.1

Compiler and Version used

go version go1.19.5 darwin/arm64

Operating System and version

macOS

aajtodd commented 1 year ago

The model shows S3BucketDetail#CreatedAt is a regular timestamp shape with no custom formatting which means it will default to whatever the protocol uses. Guardduty uses restJson1 which defaults to epoch seconds.

This is all deserialized correctly.

Taking a step back. What service operation doesn't work? Your example just shows you trying to unmarshal some raw json through the stdlib json APIs. How is this a bug in the SDK?

max-frank commented 1 year ago

I see I was consuming GuardDuty events from the SNS feed and working under the assumption that the SDK was using standard golang JSON. The above raw JSON is a slighlty redacted example of the GuardDuty finding data being received via SNS.

I'll switch to using smithy for deserialization then instead of the standard JSON library. Thanks for the quick response.

Looking at the smithy stuff though I can't figure out how to do this nicely. It seems this is more meant for talking to the API directly but with the SNS notification I already have the raw JSON in hand and just want to de-serialize this. To clarify further my use case involves one a collector that collects the findings from SNS subscription and then a separate processor that retrieves single findings as raw JSON from another messaging queue. I would need to de-serialize the JSON in that processor to do some processing. What would be the correct way to do this with smithy?

Also feel free to close this issue/change the label from bug to question.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aajtodd commented 1 year ago

What would be the correct way to do this with smithy?

Serializers and deserializers are not publicly exposed and we don't have plans to, they are not appropriate for arbitrary serialization/deserialization as not every field is necessarily bound to the document (e.g. json/xml). I'd recommend wrapping/aliasing the SDK types appropriately to enable using the stdlib json marshal/unmarshal APIs.

Closing as this isn't an actionable issue for the SDK.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / aws-sdk-go-v2