aws / aws-lambda-go

Libraries, samples and tools to help Go developers develop AWS Lambda functions.
Apache License 2.0
3.64k stars 554 forks source link

Kinesis Firehose lambda api doesn't support routing to multiple table for iceberg #575

Open amalakar opened 1 week ago

amalakar commented 1 week ago

Is your feature request related to a problem? Please describe. As per: https://docs.aws.amazon.com/firehose/latest/dev/apache-iceberg-format-input-record-different.html In order to set the target table while processing via lambda needs to set metadata as:

    "metadata":{
        "otfMetadata":{
            "destinationTableName":"gameevents",
            "destinationDatabaseName":"PlayerSpawn",
            "operation":"insert"
        }
    }

But the response metadata struct only supports partition keys https://github.com/aws/aws-lambda-go/blob/1ba09afd7523e0d5a8a2e2fb58ddb8ec8b92ebb7/events/firehose.go#L40

type KinesisFirehoseResponseRecord struct {
.....
    Metadata KinesisFirehoseResponseRecordMetadata `json:"metadata"`
}

type KinesisFirehoseResponseRecordMetadata struct {
    PartitionKeys map[string]string `json:"partitionKeys"`
}

Describe the solution you'd like Please support otfMetadata as part of metadata for lambda response object.