Duplicate data types and inconsistencies in data types from AWS SDK

[mikeparker]

Is your feature request related to a problem? Please describe. I'm writing a library to use across fargate containers and lambda to deal with SQS messages and there are some annoying inconsistencies in the duplicate structs which represent the same objects. Why not just use the structs from the main SDK?

For example, the SQS event in this library:

type SQSEvent struct {
    Records []SQSMessage `json:"Records"`
}

type SQSMessage struct {
    MessageId              string                         `json:"messageId"`
    ReceiptHandle          string                         `json:"receiptHandle"`
    Body                   string                         `json:"body"`
    Md5OfBody              string                         `json:"md5OfBody"`
    Md5OfMessageAttributes string                         `json:"md5OfMessageAttributes"`
    Attributes             map[string]string              `json:"attributes"`
    MessageAttributes      map[string]SQSMessageAttribute `json:"messageAttributes"`
    EventSourceARN         string                         `json:"eventSourceARN"`
    EventSource            string                         `json:"eventSource"`
    AWSRegion              string                         `json:"awsRegion"`
}

And the main AWS sdk has similar:

// A list of received messages.
type ReceiveMessageOutput struct {
    _ struct{} `type:"structure"`

    // A list of messages.
    Messages []*Message `locationNameList:"Message" type:"list" flattened:"true"`
}

// An Amazon SQS message.
type Message struct {
    _ struct{} `type:"structure"`

    // A map of the attributes requested in ReceiveMessage to their respective values.
    // Supported attributes:
    //
    //    * ApproximateReceiveCount
    //
    //    * ApproximateFirstReceiveTimestamp
    //
    //    * MessageDeduplicationId
    //
    //    * MessageGroupId
    //
    //    * SenderId
    //
    //    * SentTimestamp
    //
    //    * SequenceNumber
    //
    // ApproximateFirstReceiveTimestamp and SentTimestamp are each returned as an
    // integer representing the epoch time (http://en.wikipedia.org/wiki/Unix_time)
    // in milliseconds.
    Attributes map[string]*string `locationName:"Attribute" locationNameKey:"Name" locationNameValue:"Value" type:"map" flattened:"true"`

    // The message's contents (not URL-encoded).
    Body *string `type:"string"`

    // An MD5 digest of the non-URL-encoded message body string.
    MD5OfBody *string `type:"string"`

    // An MD5 digest of the non-URL-encoded message attribute string. You can use
    // this attribute to verify that Amazon SQS received the message correctly.
    // Amazon SQS URL-decodes the message before creating the MD5 digest. For information
    // about MD5, see RFC1321 (https://www.ietf.org/rfc/rfc1321.txt).
    MD5OfMessageAttributes *string `type:"string"`

    // Each message attribute consists of a Name, Type, and Value. For more information,
    // see Amazon SQS Message Attributes (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes)
    // in the Amazon Simple Queue Service Developer Guide.
    MessageAttributes map[string]*MessageAttributeValue `locationName:"MessageAttribute" locationNameKey:"Name" locationNameValue:"Value" type:"map" flattened:"true"`

    // A unique identifier for the message. A MessageIdis considered unique across
    // all AWS accounts for an extended period of time.
    MessageId *string `type:"string"`

    // An identifier associated with the act of receiving the message. A new receipt
    // handle is returned every time you receive a message. When deleting a message,
    // you provide the last received receipt handle to delete the message.
    ReceiptHandle *string `type:"string"`
}

Even ignoring the top level inconsistencies, if I want to pass around the attributes map from an SQS message, the sdk uses map[string]*string where as this library uses map[string]string.. arg! So I have to have 2 nearly identical functions depending on if the sqs message came from the aws golang sdk or the lambda golang sdk.

https://github.com/aws/aws-lambda-go/blob/master/events/sqs.go#L15 https://github.com/aws/aws-sdk-go/blob/main/service/sqs/api.go#L3688

Describe the solution you'd like Please either a) Use the same structs for both libraries b) At least use the same base types for the attributes map and similar.. largely agreeing when to use pointers..

[harrisonhjones]

disclaimer: I work at Amazon but don't work for the Lambda team

Why not just use the structs from the main SDK?

I believe the answer to this one is: because we didn't want to take a dependency on the aws sdk. This massively reduces the sizes of your Lambda functions assuming you don't need the aws sdk for some other reason. I think that's probably unlikely so I would be in favor of a fork of this library that just used the aws sdk directly for struct definitions.

That said, there is nothing stopping you from NOT using the events defined in this package and instead define your own, eg:

import "github.com/aws/aws-sdk-go/service/sqs" // not sure if this is the 100% correct import.

type SQSEvent struct {
    Records []sqs.Message
}

I believe that'll work and give you the interop you want.