search

awslabs / aws-sdk-rust

AWS SDK for the Rust Programming Language
https://awslabs.github.io/aws-sdk-rust/
Apache License 2.0
3.03k stars 248 forks source link

[request]: Serialize/Deserialize of models for Lambda events #269

Open nmoutschen opened 3 years ago

nmoutschen commented 3 years ago

Community Note

Tell us about your request

Add support for deserialization of structs for use with Lambda events. This is similar to https://github.com/awslabs/aws-sdk-rust/issues/211, but for event sources that integrate with Lambda.

Alternatively, we could provide a official set of Lambda events, but this would be hard to do without an authoritative repository of event schemas for Lambda.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

I am currently writing a Lambda function in Rust for a GitHub sample that's invoked from a DynamoDB streams to showcase how to do parallel processing within a Lambda function.

Lambda's Event Source Mapping takes care of polling events from the DynamoDB stream and invoking the Lambda function with a batch. Lambda sends a JSON objects resembling aws_sdk_dynamodbstreams::output::GetRecordsOutput without NextShardIterator.

Are you currently working around this issue?

Right now, I've recreated most of it by making my own structs copying the Go Lambda SDK's DynamoDBEvent structure.

Additional context

N/A

Attachments

Velfi commented 3 years ago

Thanks for submitting this, I'll ask the rest of the team what they think about it.

calavera commented 3 years ago

You might want to take a look at this project: https://github.com/LegNeato/aws-lambda-events

spf5000 commented 2 years ago

I wanted to piggy-back off this as I'm not sure the best place to put generic requests for serde serialize/deserialize support. I'm currently looking for ways to deserialize the items (aka HashMap<String, AttributeValue>) from DynamoDB after exporting the table to s3 and downloading the files locally. I don't currently see an easy way to deserialize these items from S3 into items/structs using the existing serde_json or serde_dynamo crates today.

Maybe there is something I'm missing here, but this would be extremely valuable to be able to deserialize the DynamoDB table exports for scripts in Rust while learning the language and experimenting with the SDK.

liberwang1013 commented 2 years ago

I have same problems. My situation is that I want to parse cloudwatch event detail field into aws config ConfigurationItemChangeNotification, ConfigurationItem is really complicated struct, serde remote type is not a good way to do it

Velfi commented 2 years ago

The biggest issue is what this will do to crate sizes and compile times. I wrote about this in an FAQ entry.

Why don't the SDK service crates implement serde::Serialize or serde::Deserialize for any types?

1. Compilation time: serde makes heavy use of several crates (proc-macro2, quote, and syn) that are very expensive to compile. Several service crates are already quite large and adding a serde dependency would increase compile times beyond what we consider acceptable. When we last checked, adding serde derives made compilation 23% slower.

2. Misleading results: We can't use serde for serializing requests to AWS or deserializing responses from AWS because both sides of that process would require too much customization. Adding serialize/deserialize impls for operations has the potential to confuse users when they find it doesn't actually capture all the necessary information (like headers and trailers) sent in a request or received in a response.

In the future, we may add serde support behind a feature gate. However, we would only support this for operation Input and Output structs with the aim of making SDK-related tests easier to set up and run.

It's possibly that in a future version of Rust that supports "poll-based dependency resolution" we could add support for serde without affecting compile times for those that don't use it. We're not compiler experts and are definitely open to suggestions, but it's important that any proposed solution weighs the needs of all users.

related topics: https://internals.rust-lang.org/t/towards-a-second-edition-of-the-compiler/5582 https://users.rust-lang.org/t/why-cant-rustc-only-compile-code-that-is-actually-used/64601

dcormier commented 2 years ago

It would be nice to have a crate feature to enable deserialization for those of us who need it. It's a fairly common way to do that in the Rust community. We are currently impacted by its absence, and are resorting to using types from the aws_lambda_events crate.

MathieuDuponchelle commented 1 week ago

I have a similar need for serializing transcripts received using the transcribestreaming API. If I'm not mistaken those events are received as JSON in the first place, so my ideal solution would be a hook for direct access to the received messages.

landonxjames commented 1 week ago

You should be able to get access directly to the not-yet-deserialized JSON with an Interceptor using the read_before_deserialization hook: https://docs.rs/aws-smithy-runtime-api/latest/aws_smithy_runtime_api/client/interceptors/trait.Intercept.html#method.read_before_deserialization

MathieuDuponchelle commented 6 days ago

You should be able to get access directly to the not-yet-deserialized JSON with an Interceptor using the read_before_deserialization hook: https://docs.rs/aws-smithy-runtime-api/latest/aws_smithy_runtime_api/client/interceptors/trait.Intercept.html#method.read_before_deserialization

Oh interesting thanks :)

MathieuDuponchelle commented 6 days ago

@landonxjames this was promising but unfortunately I can't get anything useful out of it, I filed https://github.com/awslabs/aws-sdk-rust/issues/1212 with what I tried :)

MathieuDuponchelle commented 6 days ago

In my understanding the interceptor code can only be used to inspect the initial response, not follow up messages

MathieuDuponchelle commented 3 days ago

@landonxjames I ended up using serde_with to copy the relevant structures over to types I can implement Serialize for, this is of course not optimal, if I missed a way to actually intercept the events in the event stream please let me know.