feat(core): Extend container support

Tranquility2 commented 4 months ago

Overview

So basically as you can see from the title this adds a functionally that I think is super useful, supporting working with Dockerfile for achieving even more compatibility with custom images.

from testcontainers.core.container import DockerContainer
from testcontainers.core.image import DockerImage

with DockerImage(path=".", tag="new_image") as image  # tag is optional
    with DockerContainer(image=image.name) as container:
        ...

The created image is cleaned and any any intermediate containers are removed.

Background

I'm a bit biased as I originally started from working on my own project: https://github.com/Tranquility2/dockerr that has some very clear similarities with testcontainers. I noticed testcontainers in Go and Java (for example) indeed support Dockerfile (and even some extra compatibility modes like Dockerfile DSL) so decided to take the time and add this to testcontainers-python as well.

Goal

The end goal will be supporting custom images that can be tested, in my use case:

FastAPI / Flask
Amazon Lambdas (using the built-in runtime interface emulator

Work Plan

This is what I'm planning to do

[x] Add this PR to support the core image
[x] Improve custom image support but easier usage, for example:
```
from testcontainers.core.generic import CustomContainer
```

with CustomContainer(tag="my-custom-image:latest", path="/path/to/Dockerfile") as server: # tag is optional server_url = custom_container.get_url()


- [x] Add some support for `get_api` to get the full user experience
- [x] Add Module for FastAPI
- [x] Add Module for Amazon Lambda

### Future work
- Add custom image support for more capabilities like the Dockerfile DSL 

---
Any feedback is welcomed and I'll be happy to update/fix as needed.

Update
Took me some time but I did found https://github.com/testcontainers/testcontainers-python/issues/83 which is quite old and https://github.com/testcontainers/testcontainers-python/pull/455 which takes a different approach from what I have here. 

Update 2 - Please note this proposal emphasize on 2 things
1. Cleanup as much as possible
2. Minimal changes to the current workflow. `DockerContainer` basically remains the same, this limits maintenance required and allows for easier integration.

Update 3 - Hopefully done ,this PR includes multiple feats 
- Image build support (using Dockerfile)
- Added SrvContainer
- Added FastAPI module
- Added AWS Lambda module

Tranquility2 commented 3 months ago

Image creation is now based on id as advised by @alexanderankin (tag is optional and not critical to the build flow)

Tranquility2 commented 3 months ago

Image cleanup is now optional (defaults to True to facilitate the use-cases known to me)

Tranquility2 commented 3 months ago

Please note that test_core.py now take double the time (~15sec on my machine)

alexanderankin commented 3 months ago

for fast api, i can imagine the plan is to create a Testcontainer for an intree service to test against from other intree services' test suites. what is the plan for aws lambda exactly?

Tranquility2 commented 3 months ago

I think its basically extending the paradigm for testcontainers-python. Instead of only providing the supporting services for testing something, one can now test a service ("dockerized" one). I updated the doctest for SrvContainer (and code ofc) to better reflect this, so I hope it will be more clear, any python web service could benefit from testing it like this + we can provide supported services as neeed, for example:

with RedisContainer() as redis_container:
    redis_params = redis_container.get_client_params()
    with FastAPIContainer() as server: # this is very close to SrvContainer
        # do some calls to the server and utilize the fact that its connected to redis

*at this point I'm not as sure we will need a module for FastAPI but still needs to think about it

Now regarding aws lambda its very similar to the localstack module already available, and very ironically will probably also be used with it quite a lot. if we look at an example:

with LocalStackContainer() as localstack:
    dynamo_client = localstack.get_client("dynamodb")
    # prepare something in dynamo
    with AwsLambdaContainer() as lambda:
        # run some requests to http://localhost:9000/2015-03-31/functions/function/invocations
        # this way we can be validating flow with dynamo 
        # and maybe even checking the changes on dynamo itself

this is static URL in all AWS lambdas AwsLambdaContainer will be a new module similar=have all the same env vars, as its all AWS eco system

Tranquility2 commented 3 months ago

Decided to implement a module for FastAPI, I think it will be good even on a didactic level for anyone who needs to use it, as it include a working example and all the sources.

Tranquility2 commented 3 months ago

Added Module for Amazon Lambda

Reorganized the PR commits:

feat(core): Added AWS Lambda module
feat(core): Added FastAPI module
feat(core): Added SrvContainer
feat(core): Image build support (using Dockerfile)

They now reflect better the improvements and the work that was done.

Tranquility2 commented 3 months ago

I think I should split this PR to the 4 commit (PR for each)

feat(core): Image build support (using Dockerfile)
feat(core): Added SrvContainer
feat(core): Added FastAPI module
feat(core): Added AWS Lambda module

So it will be easier to review and keep track Any advice on how to proceed will be welcomed

kiview commented 3 months ago

@Tranquility2 Thank you for your work here and joining efforts with the project, much appreciated 🙏. I agree with your last proposal, splitting these changes into 4 dedicated PR would be much better.

Fun fact: That is also how I originally joined the Testcontainers project as a maintainer, since I authored a very similar project in Java back in the day 😉

Tranquility2 commented 3 months ago

1st part is up: https://github.com/testcontainers/testcontainers-python/pull/585

Tranquility2 commented 3 months ago

2nd part is up: #595

Tranquility2 commented 1 month ago

4th part is up: https://github.com/testcontainers/testcontainers-python/pull/655

Tranquility2 commented 1 month ago

Done:

testcontainers / testcontainers-python