Lambda-based workers

[Fodoj]

I am looking into migrating background job worker to be fully serverless, with Lambda+SQS - one of reasons is to avoid paying for compute if no jobs are scheduled. At the moment primary web application is deployed with Copilot. To get similar great developer experience for Lambda function, I would need to use both Copilot and SAM: https://github.com/aws/serverless-application-model - let Copilot build the container image and deploy to ECS, and then trigger SAM to deploy this image for the lambda function. Are there any plans to make Lamda part of Copilot, or is the idea to have Copilot focused on ECS+AppRunner and SAM on Lambda?

[Lou1415926]

Hello @Fodoj! Thank you very much for the feature request - yes we are open to the idea of inviting lambda into Copilot!

Just for me to gather more information on your use case - is this set up similar to what you are looking for? Is there any resources within the environment that your lambda potentially want to talk with (e.g. a database, an ECS service, etc.)?

Another follow-up question would be -

let Copilot build the container image and deploy to ECS, and then trigger SAM to deploy this image for the lambda function.

It seems like for the lambda you are going to use the exact same image as the ECS service (LBWS/Backend/Worker) - I wonder if your lambda is going to have a different CMD or ENTRYPOINT?

[Fodoj]

Reading from SQS with Lambda is exactly what I want to implement - basically replacing ECS-based Worker service with incoming lambda function.

Container image is the same, but indeed different command will be invoked (so that instead of launching main web app, it will process SQS events).

[rmarapp-dio]

Yes - this would be a much-needed addition. We currently have a lot of use cases where it makes more sense to have a lambda listen for messages within SQS - ones where the volume is not too high and does not make sense to have an always-running worker service those are not high volume.

[praneetap]

@Fodoj @rmarapp-dio Hello! I am the Product Manager for AWS SAM and SAM CLI. I would love to chat with folks who are interested in giving feedback on their dev experience around hybrid workflows similar to the one mentioned in this issue. If you would like to give feedback and help CoPilot and SAM teams, please reach out to me on twitter DM and I will schedule a call! thanks!

[Fodoj]

I have successfully integrated Copilot with SAM with a GitHub Actions pipeline, by using container images for Lambdas. We have a simple deploy script like this:

if [[ $1 == 'ecs' ]]; then
  copilot deploy -a mkdev -e $2 -n web
else
  sam deploy --template-file sam.yaml --parameter-overrides ParameterKey=Env,ParameterValue=$2 ParameterKey=TAG,ParameterValue=$IMAGE_TAG --config-env $2 --resolve-image-repos
fi

We build container image just once, and use it for both runtimes. And then we run in parallel deployment to ECS and to Lambda. Works pretty well so far!

[dannyrandall]

Hey all! Wanted to share an example Lambda based worker I created using the new upload local artifacts feature released in Copilot v1.21.0. Hopefully this helps simplify some deployment scenarios! https://github.com/dannyrandall/copilot-lambda-worker!

[mreferre]

I have successfully integrated Copilot with SAM with a GitHub Actions pipeline, by using container images for Lambdas. We have a simple deploy script like this:

if [[ $1 == 'ecs' ]]; then
  copilot deploy -a mkdev -e $2 -n web
else
  sam deploy --template-file sam.yaml --parameter-overrides ParameterKey=Env,ParameterValue=$2 ParameterKey=TAG,ParameterValue=$IMAGE_TAG --config-env $2 --resolve-image-repos
fi

We build container image just once, and use it for both runtimes. And then we run in parallel deployment to ECS and to Lambda. Works pretty well so far!

@Fodoj What is the structure of your code in your image? Do you just change the ENTRYPOINT/COMMAND depending on where you deploy it?

[Fodoj]

@mreferre image is exactly the same for both ECS and Lamda. Only difference is another command set inside sam.yaml :)

[paragbhingre]

Related to https://github.com/aws/copilot-cli/discussions/4484

[fade2black]

I've solved it using a separate SAM project with manual configuration (Nat Gateway, RDS, RDS Proxy, Secret Manager, VPC Endpoint, SQS, SQS Poll permissions) :-)