fix(cosmosdb): Add support for the CosmosDB Emulator

mbenabda commented 3 months ago

Adds support for the CosmosDB Emulator container

alexanderankin commented 3 months ago

since we do not count community features towards testcontainers features, i have tagged this PR with feat-community but renamed the PR to "fix" - i have fixed the docs build for you but i am not going to get the module tests to pass, they seemed to not pass on the last run

alexanderankin commented 3 months ago

you can run the tests locally with poetry install -E cosmosdb --with dev && poetry run pytest -s modules/cosmosdb/tests in case it helps you. i was able to validate that the waiting you are doing is in fact working - the log, the wait for url, etc, I suspect the emulator just needs more than two minutes to startup and is timing out? not sure.

mbenabda commented 3 months ago

thank you @alexanderankin for looking into this.

Indeed, the container unfortunately takes a while to start up, and I believe you're correct: this is timeout-related failure.

locally, the test passes, and runs consistently in ~1.46s in avg (avg calculated over 100 successive test runs. all passed)

$ time pytest -s modules/cosmosdb/tests
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.4.0
rootdir: /tmp/tmp.olcVZbq8gB/testcontainers-python
configfile: pyproject.toml
plugins: asyncio-0.23.5, anyio-4.3.0, cov-4.1.0
asyncio: mode=strict
collected 1 item                                                                                                                                                                                           

modules/cosmosdb/tests/test_cosmosdb.py::test_docker_run Pulling image testcontainers/ryuk:0.7.0

---------------------------------------------------------------------------------------------- live log call -----------------------------------------------------------------------------------------------
INFO     testcontainers.core.container:container.py:88 Pulling image testcontainers/ryuk:0.7.0
Container started: 958785fad0f5
INFO     testcontainers.core.container:container.py:101 Container started: 958785fad0f5
Waiting for container <Container: 958785fad0f5> with image testcontainers/ryuk:0.7.0 to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 958785fad0f5> with image testcontainers/ryuk:0.7.0 to be ready ...
Pulling image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
INFO     testcontainers.core.container:container.py:88 Pulling image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
Container started: 2435177c5b4e
INFO     testcontainers.core.container:container.py:101 Container started: 2435177c5b4e
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...

(...)

================================================================================ 1 passed, 2 warnings in 105.71s (0:01:45) =================================================================================

real    1m45,927s
user    0m0,875s
sys 0m0,079s

mbenabda commented 3 months ago

@alexanderankin could we move this PR back to a draft ? it'll take more time than i expected to have something proper:

The emulator container doesn't support port mapping (known limitation)
I'm binding port 8081 to the host to check for readiness of the emulator's endpoints => there can't be 2 running containers at any given time
Tests for != python versions seem to be running on the same host => tests are bound to fail

I plan on removing the endpoint readiness check from the module, and document how one would proceed to check for that.

Sorry for the inconvenience this has been ...

alexanderankin commented 3 months ago

no worries, if you can get the tests to pass we can merge as is and make changes later as well.

i noticed the port limitation myself so i wouldn't worry about changing that

if you can change the timeout configuration during tests so that the tests do actually pass - this is probably the most critical thing - as it is demonstrated in #578, #532

mbenabda commented 3 months ago

@alexanderankin Hi, i'd like to try running the tests workflows with this. is it possible ? thank you

alexanderankin commented 3 months ago

looks like "failing after 10m" still taking a while

mbenabda commented 3 months ago

@alexanderankin looks like we're good now: https://github.com/mbenabda/testcontainers-python/actions/runs/9254875065

I missed the part where embeded code examples were actually executed as part of the CI

I had to skip the examples from this module from being executed because to have them run successfully in CI, the examples would have to be changed in a way that doesn't reflect the actual module usage (ie: add bind_ports=False to the container constructors parameters)

unrelated question: shall I squash my commits ?

Thanks

kiview commented 3 months ago

What does this mean?

The emulator container doesn't support port mapping (known limitation)

Wondering, since I am not aware of any limitation in this regard from our Java implementation.

mbenabda commented 3 months ago

@kiview

What does this mean?

The emulator container doesn't support port mapping (known limitation)

Wondering, since I am not aware of any limitation in this regard from our Java [implementation]>(https://github.com/testcontainers/testcontainers-java/blob/main/modules/azure/src/main/java/org/testcontainers/containers/CosmosDBEmulatorContainer.java).

Now that you mention it, it sounds like BS.

To be honest I read that on a couple of occasions and did not bother to check :-1: I'll look into it and push changes to remove the static port bindings if that pans out

Thank you for pointing that out

alexanderankin commented 3 months ago

squashing the commits is not necessary. i am really not sure how it works in java because when i was stepping through the python driver, it had some abstractions around determining this port and it was not clear how to inform the azure cosmos driver that it should be using a different port (from the standpoint of my debugging - perhaps if i started from the docs instead it would have been clear)

eddumelendez commented 3 months ago

cosmosdb client can use two modes: direct and gateway. gateway works in java with random ports and direct is not supported. See https://github.com/testcontainers/testcontainers-java/issues/5518

saikotek commented 2 months ago

Hello, when it will be added? I too implemented comsosdb emulator support but @mbenabda did it better.

squashing the commits is not necessary. i am really not sure how it works in java because when i was stepping through the python driver, it had some abstractions around determining this port and it was not clear how to inform the azure cosmos driver that it should be using a different port (from the standpoint of my debugging - perhaps if i started from the docs instead it would have been clear)

let's be realistic, this emulator kinda sucks and is heavily restricted. I had the problem with binding ports other than 8081 not because of emulator, but because of python's cosmosdb client ignoring my port configuration. Also official documentation publishes ports 10250-10255 and they do not even say what they are for nor if they are needed.. https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Cpython&pivots=api-nosql#start-the-emulator btw I'd suggest squashing the commits to keep the main branch clean.

alexanderankin commented 2 months ago

should i just clean up and attempt to merge what is in here?

mbenabda commented 2 months ago

@saikotek @alexanderankin Sorry, I didn't find the time to get back to this these past couple of weeks.

@alexanderankin feel free clean up and merge, it should work with the caveat of binding static ports.

I'll get back to this as soon as things calm down on my side (hopefully starting from next week), and if you've merged by then, open another PR to improve on this basis.

alexanderankin commented 2 months ago

present pr - https://github.com/testcontainers/testcontainers-python/compare/main...a85a9b68e95ad16a5211c82cc1d1d5c040447a49

rebased - https://github.com/testcontainers/testcontainers-python/compare/main...pr/cosmosdb_emulator.tmp rebased permalink - https://github.com/testcontainers/testcontainers-python/compare/main...b65063412fce15b5f031a55edf2aaf68db909db3

rebased main - https://github.com/testcontainers/testcontainers-python/compare/main...pr/cosmosdb_emulator.tmp2 rebased main permalink - https://github.com/testcontainers/testcontainers-python/compare/main...fd045544032464879fa3f90f25fa19c092473729

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Please upload report for BASE (main@e575b28). Learn more about missing BASE report.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #579 +/- ## ======================================= Coverage ? 76.26% ======================================= Files ? 11 Lines ? 573 Branches ? 83 ======================================= Hits ? 437 Misses ? 110 Partials ? 26 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

testcontainers / testcontainers-python

fix(cosmosdb): Add support for the CosmosDB Emulator #579

Codecov Report