search

awslabs / aws-lambda-web-adapter

Run web applications on AWS Lambda
Apache License 2.0
1.8k stars 107 forks source link

Idea: built-in middleware/transformations #216

Closed mkdynamic closed 7 months ago

mkdynamic commented 1 year ago

One scenario for this would be serving web requests and also (for example) processing SQS messages.

For SQS messages, a middleware could wrap those into the request body and send to a configured path (specified via an environment variable).

This would enable having a single Lambda function to handle multiple types of event, using a common HTTP based backend, which is appealing for simplicity.

Thoughts?

bnusunny commented 1 year ago

Could you tell me a bit more about the use cases? Are you looking for other event sources besides SQS? What's the schema you would expect?

We are thinking about expending the event sources and convert the events to CloudEvents with HTTP Protocol Binding. Do you think this would cover your use cases?

mkdynamic commented 1 year ago

We are thinking about expending the event sources and convert the events to CloudEvents with HTTP Protocol Binding. Do you think this would cover your use cases?

Yes! This would be amazing. Big fan of CloudEvents and would love to see more AWS services adopt it too.

For some more context around the specific use case, I want to have Lambda process events from SQS via event source mappings (with a batch size of 10, say). If aws-lambda-web-adapter could transform this into an HTTP request, by setting the relevant HTTP CloudEvents using the binary format, and placing the event/context into the request body, my app code can process these events like a regular HTTP request.

One specific thing for SQS is that ideally I would want a way to signal back to SQS when items in the batch failed (https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting). It seems like for this, the aws-lambda-web-adapter could just return the response body (as JSON) to the caller.

mgoodings commented 1 year ago

We also have a use for this. Currently using a bunch of SQS FIFO driving python lambdas. Would love to replace awslambdaric with aws-lambda-web-adapter for all of these projects, it's much easier and simpler to package into lightweight containers.

If it could handle SQS batches into single/multiple HTTP requests and let you resolve with a HTTP response would be damn near perfect.

lijok commented 1 year ago

Similar usecase here. We're building services which have an API component and some workers that consume SQS, SNS, etc messages. Because of awslambdaric and the Lambda extensions API, for each of the services we need to build at least two docker images. Building for receiving events over an HTTP endpoint would also mean our developers don't have to learn about lambda entrypoints.

kueckermann commented 8 months ago

Also looking for this kind of functionality. We used to do this with @vendia/serverless-express to manage incoming SQS messages as part of our API.

In our case we'd be completely happy if the payload can just be received by our application and our application can decide how to route it accordingly. The issue is that currently even tho an SQS message will run the application when it hits a lambda running web adapter, the body and any other metadata are empty, so we can't process the request.

I'd opened an issue here https://github.com/awslabs/aws-lambda-rust-runtime/issues/765 for this, which was deemed more relevant to this already open issue.

bnusunny commented 8 months ago

@kueckermann for your use case, do you need to use one function to handle both SQS messages and HTTP requests?

kueckermann commented 8 months ago

@kueckermann for your use case, do you need to use one function to handle both SQS messages and HTTP requests?

In our use case we have a nest.js application running in lambda. It would be completely fine for us if the request was somehow able to provide the message as a body or even just provide the original body as a header so we can parse is. In the nest application we'd create an interceptor that would detect the request is from sqs and then based on our own sqs message attributes decide what route path to use to process the request.

Main issue for us at the moment is that the body is not available in the request when sqs triggers the lambda.

But to answer your question more directly. In our case it's better if we can handle sqs message and http requests to one function. That function is running our API, so essentially sqs is triggering an API endpoints. SNS of course has the ability to trigger http endpoints but the issue we have there is that SNS has a hard timeout, so for our long running batch processes they get triggered via an sqs queue.

bnusunny commented 7 months ago

Got it. We need to update lambda-http crate to pass through the non-http payloads. I'm working on a prototype for this.

kueckermann commented 7 months ago

Thanks @bnusunny for getting on this. I saw a PR was merged to the rust runtime. So great to see this moving forward, and looking forward to having this feature in a future version of web adapter.

bnusunny commented 7 months ago

Yes, we are working on it. Hopefully, we could submit a PR next week, if not sooner.