search

iliana / rust-crowbar

Wrapper to simplify writing AWS Lambda functions in Rust (using the Python execution environment)
https://docs.rs/crowbar
Apache License 2.0
195 stars 16 forks source link

Create Data Mappings for Lambda Input and Output Events #32

Open naftulikay opened 6 years ago

naftulikay commented 6 years ago

TODO

JFC CloudWatch :skull:

Pull #31 is going to get some of this.

naftulikay commented 6 years ago

Please be advised that Amazon has since added more metadata to their calls, so the difficulty here is that things will be masked by serde only deserializing the fields it knows about.

softprops commented 6 years ago

This is awesome! I was actually motivated enough to start a bridge project https://github.com/softprops/lando to add event types ( api gateway is all I needed ) . What is the likely hood we could iterate and cut releases quickly with this. I would love to just start using this today. That's a big list of checkboxes and I'd love to just get going with a release version.

Aside. I took a slightly different approach that users might appreciate. When deploying a lambdas its atypical for a single function to service multiple types of events. Rather is more common for a single lambda to target a single type of trigger.

A slightly different approach I was taking was to have a named macro per event type, in this case 

gateway!(
  |request, context| Ok(lando::Response::default())  
);

So that a lambda targeting a certain trigger would only have to focus on a function types that are relevant vs having to on one extra level of pattern matching. I think something like that could be added in addition to this approach separately.

Either way this looks great and I don't want to duplicate effort. How likely would it be that we could break up this large list into smaller lists and release more often?

naftulikay commented 6 years ago

Yeah, let's break things into pieces. Packages should be nested as represented above.

FWIW my use case is a single Lambda function dispatching all the events.

softprops commented 6 years ago

I think thing the single lambda is a really good default. I'm excited to see where this goes!

softprops commented 6 years ago

I wasn't sure where the best place to post this would be this seemed reasonable. I just announced a crate release that builds on on crowbar that adds bindings for the standard http crate to crowbar https://twitter.com/softprops/status/1003458005852196870. I can likely simplify my implementation when some of these the changes in this issue get addressed.

My workflows for aws lambda mostly revolve around the serverless framework so I made a serverless plugin to help facilitate rustlang applications https://github.com/softprops/serverless-rust. This should also by extension work well with any crowbar application.

LegNeato commented 6 years ago

Ya'll may be interested in https://github.com/srijs/rust-aws-lambda/tree/master/aws_lambda_events. I generate the event structs from Go. I'm going to publish it as its own crate soon.

softprops commented 6 years ago

What's the status on this and how I help push it forward?

naftulikay commented 6 years ago

@softprops I think we need to consolidate around a singular place for these definitions and I'm not sure that this is the best repository for it, we should probably put it in rusoto or something like that. FWIW here is some sampling of what I'm doing with my own Lambda API:

lambda!(|event, _context| {
    INITIALIZE.call_once(|| {
        logging::initialize();
        threading::initialize();
        debug!("Finished one-time initialization.");
    });

    if env::var("DEBUG").map(|s| s == "true").unwrap_or(false) {
        // if we're in debug mode, dump
        debug!("Event: {}", to_string_pretty(&event).unwrap());
    }

    // dispatch event based on its type
    match serde_json::from_value::<Event>(event) {
        Ok(Event::Auth(event)) => Ok(to_value(&service::auth::route(&event)).unwrap()),
        Ok(Event::CloudWatch(event)) => Ok(service::cloudwatch::route(&event)),
        Ok(Event::Http(event)) => Ok(to_value(&service::http::route(&event)).unwrap()),
        Ok(Event::Records(records)) => Ok(service::multi::route_all(records.entries)),
        Ok(Event::Unknown(event)) => Ok(service::unknown::route(&event)),
        Err(_) => {
            error!("Unable to convert event.");
            Ok(json!({ "statusCode": 400 ,"message": "Unable to convert event." }))
        }
    }
});

Here's my Event enum:

#[derive(Deserialize)]
#[serde(untagged)]
pub enum Event {
    CloudWatch(cloudwatch::Event),
    Auth(auth::Event),
    Http(HttpEvent),
    Records(Records),
    Unknown(Value),
}

The ergonomics of Rust are amazing for this purpose. Static routing like this gives me 500 microsecond response times, which is almost unthinkably fast, Amazon bills by 100ms intervals, I'm 200x faster than that :100:

I'd love to see a hierarchy expressed like this, perhaps using the underlying data objects from rusoto in enums for the Lambda entry point. It's really hard to get things working for Rust + Lambda (largely because the underlying Lambda system image is really old, no fault of crowbar), but once it's working, it's a joy to use!

I'd love to see more forward progress on this, but I'm currently pretty busy and haven't been able to devote time to my API as much as I'd like.

softprops commented 6 years ago

Rusoto is a boto client for rust. Boto only describes aws services. If it also describes lambda event data models :thumbsup: if not I feel like a stand alone crate on its own release schedule would do. I'd go so far as to say especially for the release schedule. I think rusoto is a bit reserved in releasing often because it rereleases all of its crates in batches bumping versions when nothing's changed. For something like a crate of lambda event datastuctures I'd imagine the change cycle to release time to be super short.

naftulikay commented 6 years ago

Yup, we're on the same page! I thought that somebody had done it already, but I could be mistaken. If anyone wants to start that, please do and I'll contribute some work at it.

LegNeato commented 6 years ago

I did: https://github.com/srijs/rust-aws-lambda/tree/master/aws_lambda_events. It doesn't have any deps on a particular rust framework and is auto-generated from the official go typedefs. It even autogenerates tests and uses the example json from the official go lib.

The generator is https://github.com/srijs/rust-aws-lambda/tree/master/aws_lambda_events_codegen.

That being said, all those cloudwatch events are out of scope....I'm only focusing on Lambda events.

softprops commented 6 years ago

@naftulikay would you be open to publishing what you have now in a stand alone crate. I'm using crowbar at work in a growing number of places. I would use said crate as soon as you'd publish it :)

softprops commented 6 years ago

@LegNeato ha I started typing before you posted. Would you be able to put that in a repo and on its own release schedule? I fine is convenient for maintainers but less so for users to have projects that release multiple artifacts because of the release coordination involved ( publishing new versions of things when they haven't changed as an artifact :) ). It also makes it less daunting for outside contributors. If you published that today I'd use it.. tomorrow :)

softprops commented 6 years ago

Knowing this exists it'd be great to consolidate efforts. A separate create would keep crowbar super small for cases when you may not gain much from the structures, scheduled crons ECT.

softprops commented 6 years ago

@LegNeato this looks great! I could shed some fur I've grown in another crate with your data structs

https://github.com/softprops/lando/blob/master/src/request.rs

What dependencies does your data structures crate have on its parent project?

LegNeato commented 6 years ago

@softprops Zero deps on the parent project. The parent project consumes it and reexports. I was doing my own runtime with gob encoding and decided to join forces. It also helps to have a project that is testing it while developing, hence why it is part of the workspace. It was always envisioned as a crate that can be used with all rust lambda runtimes even though it lives in a specific one's repo...which is why I posted https://github.com/ilianaw/rust-crowbar/issues/32#issuecomment-394627582 😄

I added a README and do intend to publish it soon. I wanted to get https://github.com/srijs/rust-aws-lambda/issues/8 in first though.

softprops commented 6 years ago

@LegNeato sounds great! Having it embedded in another crate without actually depending on that the other crates makes it harder to discover and potentially more tricky with ci since itd be coupled to its parent projects ci. I'm not sure if how much value you're getting from the embedding now but I assume it would actually make it easier to manage independently from a contributor/ci/release cycle stand point in its own repo. Either way I look forward to its release!

softprops commented 6 years ago

https://github.com/srijs/rust-aws-lambda/issues/8 is an great example. It seems this would be easier to manage changes specific to the data structures in its own crate which you can manage/release independently

naftulikay commented 6 years ago

I will shortly post my data types that I have internally defined. Unfortunately they're in a non-public repository that I don't plan on making public.

naftulikay commented 6 years ago

Here it is everybody, I apologize it's in a Gist, but it's what I can ship for now: https://gist.github.com/naftulikay/99f4ab201c7efddb2210930f411e60e4

I have a series of test cases which should make things clear. Basically mod.rs/Event is the root value that the Lambda function will receive. It disambiguates into HTTP events (from API Gateway), CloudWatch events directly, API Gateway Authorization Events, or to a series of mod.rs/Records which are mod.rs/Record objects.

Records can be S3, SES, SNS, and other types that I haven't implemented yet.

Some of the more complicated flows are found in SES, where it's necessary to define types for full messages (basically SES -> SNS -> Lambda is required to get a full message) versus a "filter" action which allows bouncing emails, but only gives you access to basic metadata about the message.

I have unit tests stashed away and I have the Terraform required to create all of this, but it is likewise not open source.

LegNeato commented 6 years ago

@softprops FWIW I published an initial version of the events package: https://crates.io/crates/aws_lambda_events

softprops commented 6 years ago

@LegNeato that's great news!