Seeing 503 (Service Unavailable) from the CosmosDB Emulator

When my container(Gateway-Service) starts and wants to create the database, the service stops at EnsureCreated and gives the following error after some time. The strange thing is that it creates some containers, but not all, as you can see in the screenshot. The Services runs in a Docker-Compose Enviorment. For example, ‘Identity_UserRoles’ is missing.

ASP.NET - Program.cs

using Gateway_Service;
using Gateway_Service.Database;
using MassTransit;
using Microsoft.EntityFrameworkCore;
using Microsoft.AspNetCore.Identity;
using Asp.Versioning.ApiExplorer;
using Asp.Versioning.Conventions;
using Asp.Versioning;
using Gateway_Service.Swagger;
using Microsoft.Extensions.Options;
using Swashbuckle.AspNetCore.SwaggerGen;
using AspNetCore.Identity.CosmosDb.Extensions;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.IdentityModel.Tokens;
using System.Text;
using Gateway_Service.V1.Services.Token;
using Gateway_Service.V1.Entities;
using Microsoft.OpenApi.Models;
using Core_Library.V1.Services.FileTransport;
using Core_Library.V1.Services.FileTransport.Blob;
using Microsoft.Extensions.Azure;
using Azure.Storage.Blobs;

var builder = WebApplication.CreateBuilder(args);

builder.Configuration.AddEnvironmentVariables();
builder.Services.AddTransient<IConfigureOptions<SwaggerGenOptions>, ConfigureSwaggerOptions>();

string connectionStringKey = builder.Environment.IsDevelopment() ? "Development" : "Productive";
string cosmosConnectionString = builder.Configuration.GetConnectionString($"DbContextConnection:{connectionStringKey}") ?? "";
string cosmosIdentityDbName = builder.Configuration.GetValue<string>("CosmosIdentityDbName") ?? "";
string setupCosmosDb = builder.Configuration.GetValue<string>("SetupCosmosDb") ?? "";
if (bool.TryParse(setupCosmosDb, out var setup) && setup)
{
    var contextOptionsBuilder = new DbContextOptionsBuilder<ApplicationDbContext>();
    contextOptionsBuilder.UseCosmos(cosmosConnectionString, cosmosIdentityDbName);
    var dbContext = new ApplicationDbContext(contextOptionsBuilder.Options);
    dbContext.Database.EnsureCreated();
}

builder.Services.AddDbContext<ApplicationDbContext>(options => options.UseCosmos(connectionString: cosmosConnectionString, databaseName: cosmosIdentityDbName));
builder.Services.AddCosmosIdentity<ApplicationDbContext, AuthUser, IdentityRole, string>(
        options => options.SignIn.RequireConfirmedAccount = true).AddDefaultTokenProviders();

ASP.NET - ApplicationDbContext.cs

using Microsoft.AspNetCore.Identity.EntityFrameworkCore;
using Microsoft.AspNetCore.Identity;
using Microsoft.EntityFrameworkCore;
using AspNetCore.Identity.CosmosDb;
using Gateway_Service.V1.Entities;

namespace Gateway_Service.Database
{
    public class ApplicationDbContext : CosmosIdentityDbContext<AuthUser, IdentityRole, string>
    {
        public ApplicationDbContext(DbContextOptions options)
            : base(options)
        {
        }
    }
}

Docker-Compose

version: "3.8"
name: 'randalo'

services:
  gatewayservice:
    user: root
    depends_on:
      cosmosdb:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
      azurestorageemulator:
        condition: service_started
    build:
      context: ./Gateway-Service/Gateway-Service
      dockerfile: Dockerfile
      args:
        PAT: xxxxxxx
    ports:
      - "8585:8585"
    environment:
      - COSMOS_ENDPOINT=https://cosmosdb:8081/
      - ConnectionStrings:DbContextConnection:Development=AccountEndpoint=https://cosmosdb:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==;DisableServerCertificateValidation=true
      - ConnectionStrings:AzureBlobStorage:Development=DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurestorageemulator:10000/devstoreaccount1;
      - ConnectionStrings:MessageQueue:Development=rabbitmq
      - CosmosIdentityDbName=Randalo
      - SetupCosmosDb=true
      - ASPNETCORE_HTTP_PORTS=8585
      - ASPNETCORE_ENVIRONMENT=Development

  cosmosdb:
    restart: always
    tty: true
    hostname: "cosmosdb"
    image: "mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest"
    ports:
      - '8081:8081' # Data Explorer
      - '8900:8900'
      - '8901:8901'
      - '8902:8902'
      - '10250:10250'
      - '10251:10251'
      - '10252:10252'
      - '10253:10253'
      - '10254:10254'
      - '10255:10255'
      - '10256:10256'
      - '10350:10350'
    expose:
      - "8081"
      - "10251-10255"
    environment:
      - AZURE_COSMOS_EMULATOR_PARTITION_COUNT=5
      - AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
    healthcheck:
      test: [ "CMD", "curl", "-fk", "https://localhost:8081/_explorer/emulator.pem" ]
      interval: 5s
      timeout: 5s
      retries: 60
    entrypoint: [ "/bin/bash", "-c" ]
    command:
      - |
        apt-get update -y && apt-get install -y curl
        /usr/local/bin/cosmos/start.sh

  rabbitmq:
    image: "masstransit/rabbitmq:latest"
    ports:
      - "5672:5672"
      - "15672:15672"
    healthcheck:
      test: rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms
      interval: 30s
      timeout: 30s

  azurestorageemulator:
    image: "mcr.microsoft.com/azure-storage/azurite:latest"
    ports:
      - "10000:10000"
      - "10001:10001"
      - "10002:10002"

Stack traces

2024-08-21 21:10:59 Unhandled exception. Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: The maximum number of retries (6) was exceeded while executing database operations with 'CosmosExecutionStrategy'. See the inner exception for the most recent failure.
2024-08-21 21:10:59  ---> Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: ServiceUnavailable (503); Substatus: 1007; ActivityId: a8667ba2-864f-4d81-8d1f-d2ab00fd03d7; Reason: ({"code":"ServiceUnavailable","message":"Sorry, we are currently experiencing high demand in this region South Central US, and cannot fulfill your request at this time Wed, 21 Aug 2024 19:10:59 GMT.\r\nActivityId: a8667ba2-864f-4d81-8d1f-d2ab00fd03d7, Microsoft.Azure.Documents.Common/2.14.0"}
2024-08-21 21:10:59 RequestUri: https://172.27.0.4:8081/dbs/Randalo/colls;
2024-08-21 21:10:59 RequestMethod: POST;
2024-08-21 21:10:59 Header: Authorization Length: 80;
2024-08-21 21:10:59 Header: x-ms-date Length: 29;
2024-08-21 21:10:59 Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1;
2024-08-21 21:10:59 Header: x-ms-activity-id Length: 36;
2024-08-21 21:10:59 Header: Cache-Control Length: 8;
2024-08-21 21:10:59 Header: User-Agent Length: 122;
2024-08-21 21:10:59 Header: x-ms-version Length: 10;
2024-08-21 21:10:59 Header: Accept Length: 16;
2024-08-21 21:10:59 , Request URI: /dbs/Randalo/colls, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/12 cosmos-netstandard-sdk/3.34.4);
2024-08-21 21:10:59    at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)
2024-08-21 21:10:59    at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)
2024-08-21 21:10:59    at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)
2024-08-21 21:10:59    at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)
2024-08-21 21:10:59    at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)
2024-08-21 21:10:59    at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)
2024-08-21 21:10:59 --- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(503, 1007)":1}},"name":"CreateContainerStreamAsync","start datetime":"2024-08-21T19:10:59.226Z","duration in milliseconds":92.0002,"data":{"Client Configuration":{"Client Created Time Utc":"2024-08-21T19:09:58.5159620Z","MachineId":"hashedMachineName:c4a03891-bd20-b16e-a83f-43321c7f98e8","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.41.0|1|X64|Debian GNU Linux 12 bookworm |.NET 8.0.8|N| Microsoft.EntityFrameworkCore.Cosmos/8.0.7","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":12}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":91.8595,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":91.8081,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-08-21T19:09:59.4667285Z","cpu":0.000,"memory":13656548.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32765,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-08-21T19:10:09.4719413Z","cpu":4.356,"memory":13554568.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1367,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":1},{"dateUtc":"2024-08-21T19:10:19.4725267Z","cpu":1.149,"memory":13537776.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0791,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":1},{"dateUtc":"2024-08-21T19:10:29.4730076Z","cpu":3.625,"memory":13535196.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0979,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":1},{"dateUtc":"2024-08-21T19:10:39.4733356Z","cpu":0.834,"memory":13524976.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0848,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":1},{"dateUtc":"2024-08-21T19:10:49.4735920Z","cpu":1.164,"memory":13519596.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0715,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":1}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":91.8014,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":91.7967,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":91.7692,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":91.7662,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":91.1873,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-08-21T19:10:59.2267579Z","DurationInMs":90.6802,"RequestUri":"https://172.27.0.4:8081/dbs/Randalo/colls","ResourceType":"Collection","HttpMethod":"POST","ActivityId":"a8667ba2-864f-4d81-8d1f-d2ab00fd03d7","StatusCode":"ServiceUnavailable","ReasonPhrase":"Service Unavailable"}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"a8667ba2-864f-4d81-8d1f-d2ab00fd03d7","ResponseTimeUtc":"2024-08-21T19:10:59.3181543Z","StatusCode":503,"SubStatusCode":1007,"RequestCharge":0,"RequestUri":"dbs/Randalo","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: {\"code\":\"ServiceUnavailable\",\"message\":\"Sorry, we are currently experiencing high demand in this region South Central US, and cannot fulfill your request at this time Wed, 21 Aug 2024 19:10:59 GMT.\\r\\nActivityId: a8667ba2-864f-4d81-8d1f-d2ab00fd03d7, Microsoft.Azure.Documents.Common/2.14.0\"}\nRequestUri: https://172.27.0.4:8081/dbs/Randalo/colls;\nRequestMethod: POST;\nHeader: Authorization Length: 80;\nHeader: x-ms-date Length: 29;\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\nHeader: x-ms-activity-id Length: 36;\nHeader: Cache-Control Length: 8;\nHeader: User-Agent Length: 122;\nHeader: x-ms-version Length: 10;\nHeader: Accept Length: 16;\n, Request URI: /dbs/Randalo/colls, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/12 cosmos-netstandard-sdk/3.34.4\n   at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\n   at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\n   at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n   at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]}]}]}]}]}]}
2024-08-21 21:10:59    --- End of inner exception stack trace ---
2024-08-21 21:10:59    at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state)
2024-08-21 21:10:59    at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
2024-08-21 21:10:59    at Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.CreateContainerIfNotExists(ContainerProperties properties)
2024-08-21 21:10:59    at Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosDatabaseCreator.EnsureCreated()
2024-08-21 21:10:59    at Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.EnsureCreated()
2024-08-21 21:10:59    at Program.<Main>$(String[] args) in /src/Program.cs:line 39

CS-Project

net8.0 enable enable Gateway_Service xxxx Linux

Include provider and version information

EF Core version: Database provider: (e.g. Microsoft.EntityFrameworkCore.CosmosDb) Target framework: (e.g. .NET 8.0) Operating system: Linux

dotnet / efcore