Cosmos DB Container is failing on Github

[cemo]

Our builds have started to fail on github. We are using ubuntu-latest for build environment and our test container version is v1.16.2. The builds are passing on our local boxes fine.

Can anybody help me to resolve it?

14:04:30.540 INFO  🐳 [mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest] - Container mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest is starting: decef015fcb0d81d09a60e3adc2680c6437728fda214681bdf126b3a452f85a0
14:05:30.597 ERROR 🐳 [mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest] - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '(?s).*Started\r\n$'
    at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:49)
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)

[cemo]

ubuntu-latest is failing but ubuntu 18-04 is not failing. Which library do you think that causing problem? I had updated docker too but it did not fix issue.

[kiview]

@cemo Thanks for reporting, can you please share all the container logs as well? They should be logged after the container failed to start.

[connerza]

I'm seeing this as well - it is hanging when starting the 3rd partition and timing out.

16:14:35.235 [docker-java-stream--1759158432] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire - http-outgoing-2 << "[\r][\n]"
16:14:35.236 [main] DEBUG org.testcontainers.containers.output.WaitingConsumer - STDOUT: Started 2/11 partitions
16:24:20.109 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000013: cancel
16:24:20.109 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.DefaultManagedHttpClientConnection - http-outgoing-2: close connection IMMEDIATE
16:24:20.109 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000013: endpoint closed
16:24:20.109 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000013: discarding endpoint
16:24:20.109 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000013: releasing endpoint
16:24:20.109 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000013: connection is not kept alive
16:24:20.109 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000013: connection released [route: {}->unix://localhost:2375][total available: 0; route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
16:24:20.110 [main] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire - http-outgoing-2 << "end of stream"
16:24:20.114 [main] DEBUG 🐳 [mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest] - Wait strategy threw an exception
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '(?s).*Started\r\n$'
    at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:49)
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
        ...

[kiview]

@connerza Can you please also share the container logs? They should be printed after the container launch failed.

[connerza]

This is all I'm seeing @kiview It just hangs after the 2/11 partitions log message. I can confirm going back to Microsoft's hosted Ubuntu 18 Agent does resolve the issue though.

16:24:20.257 [main] ERROR 🐳 [mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest] - Log output from the failed container:
This is an evaluation version.  There are [99] days left in the evaluation period.
Starting
Started 1/11 partitions
Started 2/11 partitions

[kiview]

Ok, so this indicates a general incompatibility of the image with ubuntu-latest. It might be a good idea to verify this with direct Docker usage and without any Testcontainers involved and raise an issue in upstream (either with maintainers of ubuntu-latest image or with maintainers of the azure-cosmos-emulator image).

[Manuel-Moya]

Any progress on this? The Ubuntu 18.04 is being deprecated https://github.blog/changelog/2022-08-09-github-actions-the-ubuntu-18-04-actions-runner-image-is-being-deprecated-and-will-be-removed-by-12-1-22/

[eddumelendez]

@Manuel-Moya this is not an issue in testcontainers. we have raised an issue https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/56 but have not heard back

[eddumelendez]

Closing because of cosmosdb image is the one failing in ubuntu 20.04 and 22.04.