[WIP] Add Data Fixtures for Lambda Event Types

[naftulikay]

Tests and documentation forthcoming. I am using these data structures for my serverless event bus in Rust using Crowbar.

My lambda handler looks something like this:

lambda!(|event, _context| {
    // dispatch event based on its type
    match serde_json::from_value::<Event>(event) {
        Ok(Event::Auth(event)) => Ok(to_value(&service::auth::route(&context, &event)).unwrap()),
        Ok(Event::CloudWatch(event)) => Ok(service::cloudwatch::route(&context, &event)),
        Ok(Event::Http(event)) => Ok(to_value(&service::http::route(&context, &event)).unwrap()),
        Ok(Event::Records(records)) => Ok(service::multi::route_all(&context, records.entries)),
        Ok(Event::Unknown(event)) => Ok(service::unknown::route(&context, &event)),
        Err(_) => {
            error!("Unable to convert event.");
            Ok(json!({ "statusCode": 400 ,"message": "Unable to convert event." }))
        }
    }
});

Thus, I do routing based on event type. For example, SNS:

pub fn route(context: &Context, record: &sns::Record) {
    info!("Received SNS Event: {}", record.event.message_id);

    match from_str(&record.event.message) {
        Ok(SnsEventKind::Ses(message)) => service::ses::receive(&context, &message),
        Ok(SnsEventKind::Unknown(value)) => warn!("Unknown JSON Event: {}", to_string_pretty(&value).unwrap()),
        Err(_) => error!("Unknown non-JSON event: {}", record.event.message),
    };
}

My multi-event router:

/// Dispatch multiple events.
pub fn route_all(context: &Context, events: Vec<Record>) -> Value {
    for event in events {
        match event {
            Record::S3(event_record) => s3::route(&context, &event_record),
            Record::Ses(event_record) => {
                // if it's an SES request/response, a response may be required, so only process one
                // of them if the type is RequestResponse
                match event_record.event.receipt.action {
                    Action::Lambda(ref action) if action.invocation_type == RequestResponse => {
                        return to_value(ses::filter(&context, &event_record)).unwrap();
                    },
                    _ => ses::route(&context, &event_record.event),
                };
            },
            Record::Sns(event_record) => sns::route(&context, &event_record),
            Record::Unknown(value) => {
                error!("Unknown event record type: {}", to_string_pretty(&value).unwrap())
            }
        };
    }

    Value::Null
}

This dispatching occurs in around 400-500 microseconds and works extremely well to make things type safe.

Obviously there's a lot here and it's not done. If @ilianaw or others have any input, RFC is open to comments :smiley:

[iliana]

I like this. :)

Can you add what has been done and what remains as checkboxes to the pull request description so we can see the path to this leaving a WIP state?

[naftulikay]

:tada: so happy that you're welcoming this PR. I will create a separate issue and track things there. There are so many different data types to support, so this PR will maybe only cover a subset with future on the way, hope that's okay.

[iliana]

Yeah, I don't have an issue with incremental development over time.

I'm hoping I'll have a chance to do a good read through of all of this soon.

[naftulikay]

@ilianaw can you please go turn on Travis CI for this project? I have updated with a build badge and Travis configuration for building and testing.

EDIT: Actually, I will submit Travis support in a separate PR.

[naftulikay]

Latest adds support for CloudWatch AutoScaling events.

[softprops]

I'd love to see this branch ship fast with additional followups to add other events types. Commented with extra thoughts on the checkbox ticket

[naftulikay]

I will start breaking this apart into a PR per service supported, I think that's probably the most prudent course of action.

One thing that is probably a really good idea is to find a way to efficiently nest CloudWatch style events and deserialize them more efficiently than I'm currently doing.

[softprops]

One thing that is probably a really good idea is to find a way to efficiently nest CloudWatch style events and deserialize them more efficiently than I'm currently doing.

Can you link to an example in this pull or sketch out in a comment what you mean?

[iliana]

Overall I'm happy with this, and thanks to @softprops for the additional review.

I'm fine with continuing with this PR, or breaking it apart as a PR per service (the latter would be great as its easier for others to help write the structs out).

I have two thoughts:

• What happens when AWS inevitably adds another field to these objects? Does serde correctly ignore extra fields?
• Would it be reasonable to make all of these an enabled-by-default feature to allow folks who aren't using this to disable it and speed up compile times?

[softprops]

What happens when AWS inevitably adds another field to these objects? Does serde correctly ignore extra fields?

Yes. This used to not be true but is now a default of serde.

[naftulikay]

I apologize for my lateness, just getting back from vacation. I will get caught up this week.

[iliana]

No worries! ^_^;

[naftulikay]

TL;DR the events received from Lambda can come via a few sources:

• SNS: a fairly consistent data type; you'd basically just deserialize the message property into the struct(s) that you expect.
• CloudWatch: there are a lot (#32) of different CloudWatch data types.
• API Gateway: requests, auth requests, and responses.

Therefore, our enums for representing all kinds of incoming events should basically include these three, and then users can disambiguate between the implementations (ie cloudwatch::BatchEvent, cloudwatch::ApiCallEvent, apigateway::Request, apigateway::AuthRequest, etc.)

I will probably break this PR down into tiny pieces and make one PR for each service to add. I wholeheartedly agree with removing Option<T> wherever possible and using the default for things like Vec<T> and BTreeMap<K, V>.

[softprops]

Is there any way to help move this forward? I'd love to help