(aws-lambda-python): standard pattern for Lambda unit tests

SamStephens commented 2 years ago

Description

It'd be great to have official guidance/patterns as to how to do unit tests of the code a Python lambda contains. I've failed to find any official guidance for either aws-lambda-python or aws-lambda in general. It's hard to find bloggers who have solved this challenge also.

Use Case

So I can solve this problem in a standard way other CDK developers will know how to work with. So that the CDK has parity with other frameworks that make unit testing Lambda code easy.

Proposed Solution

An example application in an examples repository with a link from the documentation.

Other information

No response

Acknowledge

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

SamStephens commented 2 years ago

Note that the customizability of the build process @setu4993 provides via https://github.com/aws/aws-cdk/pull/15324 would probably be a prerequisite to this, as there's no way I can see to actually inject running tests into the build process right now.

setu4993 commented 2 years ago

Note that the customizability of the build process @setu4993 provides via #15324 would probably be a prerequisite to this, as there's no way I can see to actually inject running tests into the build process right now.

Probably, yeah. Though, I'm not sure if tests on Lambda functions should occur as a part of the tests for setting up the infra, i.e. at the CDK layer. There's 2 points at which tests might occur:

At the function code level: This could be done with moto to mock AWS services and standard Python unit tests and be executed before doing the CDK deployment. I have done this a few times so far. (Or using a custom Docker image but at that point, it isn't likely using the Python Lambda function construct.)
Infrastructure level: This tests whether the correct Lambda function was created. This is what most CDK integration tests try to accomplish, so that could serve as inspiration.
Once CDK executes and deploys the function: Using boto3 or another AWS SDK or the AWS console.

So, I'm not sure this should be a part of the build process for the infrastructure. It definitely should be a part of the CI / CD process that deploys the infrastructure, though, and any of the above could be executed as a part of those steps.

SamStephens commented 2 years ago

Note that the customizability of the build process @setu4993 provides via #15324 would probably be a prerequisite to this, as there's no way I can see to actually inject running tests into the build process right now.

Probably, yeah. Though, I'm not sure if tests on Lambda functions should occur as a part of the tests for setting up the infra, i.e. at the CDK layer. There's 2 points at which tests might occur:
1. At the function code level: This could be done with [moto](https://pypi.org/project/moto/) to mock AWS services and standard Python unit tests and be executed before doing the CDK deployment. I have done this a few times so far. (Or using [a custom Docker image](https://docs.aws.amazon.com/lambda/latest/dg/images-test.html) but at that point, it isn't likely using the Python Lambda function construct.)

2. Infrastructure level: This tests whether the correct Lambda function was created. This is what most [CDK integration tests](https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-lambda-python/test) try to accomplish, so that could serve as inspiration.

3. Once CDK executes and deploys the function: Using `boto3` or another AWS SDK or the AWS console.
So, I'm not sure this should be a part of the build process for the infrastructure. It definitely should be a part of the CI / CD process that deploys the infrastructure, though, and any of the above could be executed as a part of those steps.

So what I'm talking about is at the function code test, the unit test level, your point number 1. A complement to the unit tests that assert the CDK infrastructure you have defined is what you think it is.

For other languages, running unit tests is implicitly part of building code. For example, I have an application that includes Java lambda functions that are being built using Gradle, and unit tests are defined as part of that Gradle build and executed during it according to standard Gradle practices.

setu4993 commented 2 years ago

So what I'm talking about is at the function code test, the unit test level, your point number 1. A complement to the unit tests that assert the CDK infrastructure you have defined is what you think it is.

Yeah, that makes sense. I run them using pytest, but it is a precursor to building the Lambda function with CDK. I guess it's not a part of building because Python is an interpreted language :).

nija-at commented 2 years ago

As mentioned above, unit testing Python lambda handler code is like unit testing any python code. pytest would probably be my go to, but I don't regularly work on Python and so someone with more Python experience may point to better tools.

SamStephens commented 2 years ago

There's a lot more to unit testing Python lambda handler code than choosing pytest as the framework. Here are just a few of the decisions and challenges needed to get to the point where you are able to run a single command and unit test your Lambdas, as you need really as part of a build pipeline, or whatever CI/CD you're doing:

Generally a Python lambda is simply a collection of *.py files. However (unless there's something I'm missing) pytest requires the code it is testing to be installed as a module using pip install, with a setup.py file etc.
- Do you have a setup.py file for each Lambda function, treating each as a completely standalone package?
- Or do you have a single setup.py file for all your Lambdas, treating each lambda as a module within a single package for all your Lambdas?
How do you deal with installing dependencies from requirements.txt?
How do you deal with Lambdas that receive their dependencies from Lambda Layers?
How do you actually layout the folder structure for your Lambdas and associated unit tests?
How does your script iterate through each Lambda to test?
How do you unit test code that is in Lambda Layers, not Lambda Functions?
Do you use Docker to ensure unit tests are run in a clean room environment?

Once you look at this problem in more detail, it's not at all simple. I've put together a framework for unit testing my CDK python Lambdas, but it's not robust enough I'd share it as any kind of best practice.

This is why I am requesting a standard pattern. At the moment, anyone who wants to unit test their Python Lambdas with the CDK has to solve these non-trivial problems. I suspect there's people who are not unit testing their CDK Python Lambdas who would be if there was standard guidance for them to draw upon.

It'd be nice if one person solved this problem well, rather than every CDK Python Lambda development team solving this problem poorly, or not at all.

nija-at commented 2 years ago

Thanks for the detailed explanation @SamStephens.

All of these are valid considerations and solving this in a single effort would be very valuable to Python users.

However, none of these seem specific to the AWS CDK. This will apply to any Python project that is modeling Lambda handlers. So this might be the wrong forum for this.

If you feel passionately about solving this problem, there are a couple of options. The AWS SAM team might be interested in supporting such a use case - https://github.com/aws/serverless-application-model. Alternatively, if you put together a Github project or a set of instructions on how to achieve these, we'll happily link to them from our READMEs for other Python customers to benefit from.

deuscapturus commented 1 year ago

I also needed a way to test python container lambdas. cdk only builds container images during deploy, which is too late for testing. A way was needed to build the container images as defined in cdk and test them before deploying.

For this I created a simple module cdk-image-testing, that uses cdk-assets to build images as defined in the cloud assembly.

Example:

Dockerfile

FROM public.ecr.aws/lambda/python:3.9 as base

COPY . .

FROM base as test
ENTRYPOINT []
CMD ["pytest"]

FROM base as production

./test/container-image.test.ts

import { ImageAssetTesting } from "cdk-image-testing"
import { spawnSync } from 'child_process'

// set jest timeout to 5 minutes
jest.setTimeout(5 * 60 * 1000)

describe('container images', () => {
    const imageAssets = new ImageAssetTesting('cdk.out')
    it('runs test target from image builds', async () => {
        const imageTags = await imageAssets.buildAll("test")

        for (let tag of imageTags) {
            const result = spawnSync('docker', ['run', tag]);

            if (result.status !== 0) {
                console.error(result.stdout.toString())
                console.error(result.stderr.toString())
            }
            expect(result.status).toBe(0);
        }
    })
})

aws / aws-cdk