WaitForHttp not working for CosmosDb emulator container and no timeout

I am using the library to spin up a CosmosDb emulator. The problem with this CosmosDb emulator is that it takes a while until it's ready to be used, so I need a good mechanism to wait.

I have tried the following because I know that it shows that message at the end. But, although it waits for it, it's not enough, as there is still a gap between the log message is added until the emulator is fully ready.

var cosmosDbService =
            new Builder()
                .UseContainer()
                .WithName(Constants.ContainerNames.CosmosDb)
                .UseImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
                .ExposePort(8081, 8081)
                .ExposePortRange(10250, 10255)
                .WithEnvironment(
                    "AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1")
                .DeleteIfExists()
                .RemoveVolumesOnDispose()
                .UseNetwork(network)
                .WaitForMessageInLog("Started 11/11 partitions")
                .Build();

I think the best way is to wait for an Https file to be available such as the https://127.0.0.1:8081/_explorer/emulator.pem but I suspect the https may be causing problems, because it keeps waiting indefinitely without any errors when I run the service.

var cosmosDbService =
    new Builder()
        .UseContainer()
        .WithName(Constants.ContainerNames.CosmosDb)
        .UseImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
        .ExposePort(8081, 8081)
        .ExposePortRange(10250, 10255)
        .WithEnvironment(
            "AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1")
        .DeleteIfExists()
        .RemoveVolumesOnDispose()
        .UseNetwork(network)
        .WaitForHttp(
            "https://localhost:8081/_explorer/emulator.pem", contentType: "text/plain")
        .Build();

However, I've added the cosmosDb certificate to the cert manager with elevated permissions

curl -k https://localhost:8081/_explorer/emulator.pem > emulatorcert.crt
certutil.exe -addstore root emulatorcert.crt

to ensure it is trusted. That didn't make any difference. I've also tried

.WaitForHttp("https://localhost:8081")

which returns a 401 when the container is ready, but no luck either. It keeps waiting.

Is there any way I can see what's going on or any better approach for this scenario? The strange thing also is that if I set a small timeout, it timesout, but if I set something like 60000 it does not time out.

PS: The .WaitForHealthy() does not wait enough.

I had a look at how testcontainers for CosmosDb is waiting, and it follows the strategy of waiting for https://localhost/_explorer/emulator.pem until there is a successful response

https://github.com/testcontainers/testcontainers-dotnet/blob/a0f1f7694b4602aa2ba7da6f167ddc4a56670ecc/src/Testcontainers.CosmosDb/CosmosDbBuilder.cs#L70

Not sure about the magic of having the url without any port.

Hi @diegosasw, could you try to do a "manual" wait instead to see if there's a bug or otherwise in FluentDocker.

You may add any "lambda" by using the .Wait("", (service, count) => "lambda") where the lambda function return 0, the wait is over and a positive integer is the number of milliseconds to wait before trying the lambda again.

A pseudo example:

    // Add the wait instead of WaitForHttp in the container builder
    fd.Wait("Wait for CosmosDb", (service, count) => MyWaitFunc(count, service))

    // The custom wait function
    private static int MyWaitFunc(int count, IContainerService service)
    {
      if (count > 10)
        throw new FluentDockerException("Failed to wait for cosmosDb");

      var ep = service.ToHostExposedEndpoint("80/tcp");

      // Either manual
      var response = await $"http://{ep}/my_file.pem".Wget();

     // or use the FluentDocker WaitForHttp to see why it is failing (and fix the bug)     
      service.WaitForHttp(...)
   }

The http wait function is pretty simple:

    public static void WaitForHttp(this IContainerService service, string url, long timeout = 60_000,
      Func<RequestResponse, int, long> continuation = null, HttpMethod method = null,
      string contentType = "application/json", string body = null)
    {
      var wait = null == continuation ? timeout : 0;
      var count = 0;
      do
      {
        var time = Millis;

        var request = url.DoRequest(method, contentType, body).Result;
        if (null != continuation)
        {
          wait = continuation.Invoke(request, count++);
        }
        else
        {
          time = Millis - time;
          wait = request.Code != HttpStatusCode.OK ? wait - time : -1;
        }

        if (wait > 0)
          Thread.Sleep((int)wait);

      } while (wait > 0);
    }

Cheers, Mario

Thanks for your help, it seemed, indeed, a problem with the SSL, but thanks to your suggestion I managed to add a custom Wait logic that works well. Here are the extensions methods

public static class ContainerBuilderExtensions
{
    public static ContainerBuilder ExposePortRange(this ContainerBuilder containerBuilder, int start, int end)
    {
        for (var port = start; port <= end; port++)
        {
            containerBuilder.ExposePort(port);
        }

        return containerBuilder;
    }

    public static ContainerBuilder WaitForHttps(
        this ContainerBuilder builder, 
        string url, 
        bool ignoreSslErrors = false,
        int retries = 40, 
        int delayMilliseconds = 5000)
    {
        return builder.Wait("Wait for Https", (_, count) =>
        {
            if (count > retries)
            {
                var secondsWaited = count * (delayMilliseconds / 1000);
                throw new FluentDockerException($"Failed to wait for {url} after {secondsWaited} seconds");
            }

            var httpClientHandler =
                ignoreSslErrors
                    ? new HttpClientHandler { ServerCertificateCustomValidationCallback = (_, _, _, _) => true }
                    : new HttpClientHandler();

            using var client = new HttpClient(httpClientHandler);
            try
            {
                var response = client.GetAsync(url).Result;
                if (response.IsSuccessStatusCode)
                {
                    return 0;
                }
            }
            catch (Exception)
            {
                // ignored
            }

            Thread.Sleep(delayMilliseconds);
            return 1;
        });
    }
}

And it can be used with CosmosDB (or any other service which requires waiting for an https url) specifying that SSL errors should be ignored

var cosmosDbService =
    new Builder()
        .UseContainer()
        .WithName(Constants.ContainerNames.CosmosDb)
        .UseImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
        .ExposePort(8081, 8081)
        .ExposePortRange(10250, 10255)
        .WithEnvironment(
            "AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1")
        .DeleteIfExists()
        .RemoveVolumesOnDispose()
        .WaitForHttps("https://localhost:8081/_explorer/emulator.pem", ignoreSslErrors: true)
        .UseNetwork(network)
        .Build();

cosmosDbService.Start();

Thanks for publishing your solution, I'll be sure to add the verifySSL {true, false} flag to the WaitForHttp in future.

Cheers, Mario

mariotoffia / FluentDocker

WaitForHttp not working for CosmosDb emulator container and no timeout #313