search

Azure / azure-cosmos-db-emulator-docker

This repo serves as hub for managing issues, gathering feedback, and having discussions regarding the Cosmos DB Emulator Docker.
https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Ccsharp&pivots=api-nosql
MIT License
164 stars 47 forks source link

Container is not ready after startup #87

Open HeneryHawk opened 9 months ago

HeneryHawk commented 9 months ago

Hi I have the problem that often the Linux emulator container is not finished booting, although the console says Started and all partitions have been created according to the console. It doesn't happen every time the container is started, but at least 50% of the time.

I could observe the following: When the container is started and all partitions are created immediately, the container is immediately ready and, for example, the Explorer can be accessed. However, if the creation of the last partition is not completed immediately, the container is not immediately ready, even though Started is printed to the console. It then takes at least 30 seconds or more for the Explorer to be accessible, for example.

The behavior could be observed on several dev notebooks, as well as in our GitLab CI.

We noticed the behavior in connection with Java Testcontainers and the official Azure module contained therein. Our integration tests start a Cosmos emulator container via it. In the official Azure module, the Cosmos container is implemented in such a way that it waits for the Started log to appear in the console before the tests are executed or the emulator certificate is downloaded and added to the Java Keystore. Unfortunately, this leads to errors if Started is printed on the console but the container cannot yet be called and therefore the certificate cannot be downloaded and no connection from the Cosmos Java SDK to the emulator can be established. Since the behavior has been noticed both when using the Java Testconainers Framework and when the container is started via Docker CLI, I assume that this is not related to Testcontainers. And since our integration tests are executed in the GitLab CI, this leads to countless faulty pipelines.

Is this a known error? Is there a way to get more logs? What could be the reason for this?

Screenshots fast startup and container is ready immediately fast startup and container is ready immediately

slow startup and container is not immediately ready slow startup and container is not immediately ready

Desktop (please complete the following information):

Docker Images Used:

Arguments && Environment variables to start Docker: docker run -p 8081:8081 -p 10250-10255:10250-10255 -it --rm -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=3 mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Docker Environment

v1k1 commented 9 months ago

Hi, currently the emulator is marked UP as soon as all the partitions are created. The Service (Gateway) which serves the REST APIs and explorer, sometime takes longer to boot, and this could cause the problem you face.

We will try to improve this behavior in next releases. For now please use following script for checking Emulator startup - check_emulator_startup.sh

HeneryHawk commented 8 months ago

Hi @v1k1, thanks for your reply. We didn't notice this behavior before, only since the end of January we have noticed this. Furthermore, the container starts very slowly, which makes the implementation of tests very time-consuming, as each execution of the tests takes several minutes. Most of the time is taken up by waiting for the container to start successfully. This is very annoying. Please address this issue so that the container is available again more quickly.

andersosthus commented 8 months ago

We're seeing the same as @HeneryHawk. We are using the CosmosDB Emulator in our integration tests, sometime early this year we noticed that the runtime of our integration tests had been significantly increased. Further investigation shows that the added runtime for our tests is that the CosmosDB Emulator now takes a lot longer to get ready (i.e. the Gateway is ready to accept requests).

I've always thought that the CosmosDB Emulator was a bit slow to start (up to 30 seconds), but now I'm seeing startup time of about 1m45s on my dev machine for CosmosDB Emulator, so something has changed somewhere to cause this. We are running the CosmosDB Emulator in docker using mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator with 4 partitions.

seanke commented 7 months ago

We're seeing the same as @HeneryHawk. We are using the CosmosDB Emulator in our integration tests, sometime early this year we noticed that the runtime of our integration tests had been significantly increased. Further investigation shows that the added runtime for our tests is that the CosmosDB Emulator now takes a lot longer to get ready (i.e. the Gateway is ready to accept requests).

I've always thought that the CosmosDB Emulator was a bit slow to start (up to 30 seconds), but now I'm seeing startup time of about 1m45s on my dev machine for CosmosDB Emulator, so something has changed somewhere to cause this. We are running the CosmosDB Emulator in docker using mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator with 4 partitions.

I am seeing the same thing. Start up has increased significantly recently.

pvpmartins commented 7 months ago

The issue that I am facing is that randomly slows down. This is really time consuming when testing. Please fix this behavior!

Danielku15 commented 4 months ago

We also face rather long stratup times. I wonder if there is a way to increase the log level that we at least can see that it is still doing something.

Troubleshooting is hardly possible. I'm trying to startup the emulator for some end-to-end testing in a K8S cluster and cosmosdb is the only big pain point (TLS/SSL only needing special ingress treatment, not even started even after 15mins, no logs about the real status, constant CPU load...).

sajeetharan commented 4 months ago

@v1k1 @niteshvijay1995 Can you look at this ?

Danielku15 commented 4 months ago

Not sure if my problem aligns with the problems of others, but to share some insights:

I am deploying the a pod via this file (I also tried keeping port 8081 - the custom port is just an attempt to see if it changes something).

deployment.yaml ```yml apiVersion: apps/v1 kind: Deployment metadata: name: cosmosdb spec: replicas: 1 selector: matchLabels: app: cosmosdb template: metadata: labels: app: cosmosdb spec: nodeSelector: "kubernetes.io/os": linux containers: - name: cosmosdb image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest ports: - containerPort: 8081 env: - name: AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE value: 127.0.0.1 - name: AZURE_COSMOS_EMULATOR_ARGS value: /port=10000 ```
Output of `env` ```bash root@cosmosdb-7fc76bfb64-fqwgw:/usr/local/bin/cosmos# env KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 HOSTNAME=cosmosdb-7fc76bfb64-fqwgw PWD=/usr/local/bin/cosmos AZURE_COSMOS_EMULATOR_ARGS=/port=10000 HOME=/root KUBERNETES_PORT_443_TCP=tcp://10.xx.0.1:443 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: TERM=xterm SHLVL=1 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=10.xx.0.1 AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1 KUBERNETES_SERVICE_HOST=10.xx.0.1 KUBERNETES_PORT=tcp://10.xx.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/env ```
Output of `kubectl logs` ``` This is an evaluation version. There are [101] days left in the evaluation period. Starting Started 1/11 partitions Started 2/11 partitions Started 3/11 partitions Started 4/11 partitions Started 5/11 partitions Started 6/11 partitions Started 7/11 partitions Started 8/11 partitions Started 9/11 partitions Started 10/11 partitions Started 11/11 partitions Started ```
contents of `/tmp/cosmosdb/gateway.log` ``` root@cosmosdb-7fc76bfb64-dkcw7:/usr/local/bin/cosmos# cat /tmp/cosmos/appdata/gateway.log DocDBTrace Information: 0 : Starting Gateway WinFab host DocDBTrace Information: 0 : Starting GatewayService, type Microsoft.Azure.Cosmos.RoutingGateway.EntryPoint.GatewayService DocDBTrace Information: 0 : GatewayService: BEGIN Initialize DocDBTrace Information: 0 : GatewayService Initialize DocDBTrace Information: 0 : BEGIN: GatewayService InitializeServices() DocDBTrace Information: 0 : Loading configuration package for service manifest GatewayService, code package name GatewayService.Code, configuration package name GatewayService.Config and package version 1.0 DocDBTrace Information: 0 : FabricClientFacade: Acquired 'null' credentials from EmulatorFabricClientCredentialsProvider. DocDBTrace Information: 0 : FabricClientFacade: Initialized client for endpoint(s) 127.0.0.1:0 for component 'GatewayService'; FriendlyName: 'GatewayService-Tfgt4'; Credentials: DocDBTrace Information: 0 : GatewayService.InitializeServices aadAuthLoginEndpoint = , aadAuthGraphEndpoint = , aadAuthFirstPartyApplicationId = DocDBTrace Warning: 0 : AadTokenProviderFactory: Certficate location AzureClientCertificate was not present in the configuration provider... DocDBTrace Warning: 0 : AadTokenProviderFactory: Certficate location AzureCosmosDBToAzureNRPAuthCertificate was not present in the configuration provider... DocDBTrace Information: 0 : Account : DocumentDBTest, Namespace : Onebox, Tenant: EmulatorFederation, Role: EmulatorNode, RoleInstance: EmulatorNode_IN_1 DocDBTrace Information: 0 : Initializing MDM with MonitoringAccount=DocumentDBTest Namespace=Onebox RegionalMonitoringAccountPrefix=CdbOnebox isRegionalMDMSupported=True. DocDBTrace Information: 0 : MDM RuntimeEventSource Initialization is Successful DocDBTrace Information: 0 : MDM ConfigStoreEventSource Initialization is Successful DocDBTrace Information: 0 : END: GatewayService InitializeServices() DocDBTrace Information: 0 : Emitting GatewayInit metric with dimensions as ProcessName: Microsoft.Azure.Cosmos.GatewayService, AssemblyName: DocumentDB.GatewayService, AssemblyVersion: 2.14.0 DocDBTrace Information: 0 : GatewayService: END Initialize DocDBTrace Information: 0 : GatewayService: BEGIN OpenAsync DocDBTrace Information: 0 : ConfigurationStoreConfigurationProvider: initialization started. CallingComponent: GatewayService, federationId: emulatorfederation, disableConfigurationRefreshTimer: False DocDBTrace Information: 0 : CompositeConfigurationReader.ReadConfigurationStoreConfigurationAsync, callingComponent = ConfigurationStoreConfigurationProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: ConfigurationStoreConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : CompositeConfigurationReader.ListConfigurationStoreAccountNamesAsync, callingComponent = ConfigurationStoreConfigurationProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: ConfigurationStoreAccountName, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : ConfigurationStoreConfigurationProvider: Log ConfigurationStoreConfiguration. hasConfigurationChanged False, UseNamingServiceForWritingFederationConfiguration: True, UseCosmosStoreForWritingFederationConfiguration: False, PerformReadValidationForFederationConfiguration: False, UseCosmosStoreForReadingFederationConfiguration: False, EnableFallbackForReadingFederationConfiguration: False, UseNamingServiceForWritingDatabaseAccountConfiguration: True, UseCosmosStoreForWritingDatabaseAccountConfiguration: False, PerformReadValidationForDatabaseAccountConfiguration: False, UseCosmosStoreForReadingDatabaseAccountConfiguration: False, EnableFallbackForReadingDatabaseAccountConfiguration: False, ConfigurationStoreConfigurationRefreshIntervalInSeconds: 300, federationId: emulatorfederation, UseCosmosStoreForWritingFederationVisibilityConfiguration: False, PerformReadValidationForFederationVisibilityConfiguration: False, UseCosmosStoreForReadingFederationVisibilityConfiguration: False, EnableFallbackForReadingFederationVisibilityConfiguration: False, SkipExceptionsFromSecondaryStoreForFederationConfiguration: True, SkipExceptionsFromSecondaryStoreForFederationVisibilityConfiguration: True, SkipExceptionsFromSecondaryStoreForDatabaseAccountConfiguration: True, ConfigurationStoreAccountNames: DocDBTrace Information: 0 : ConfigurationStoreConfigurationProvider initialization ended GatewayService, federationId: emulatorfederation, disableConfigurationRefreshTimer: False DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: ConfigurationStoreConfigurationProvider initialized successfully DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Invoked from InitializeAsync DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFederationConfigAsync, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : ReadFederationLevelConfigFromConfigStoreAsync: Generate compositeETag ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= for local federation decoupleAvailableBindableService True DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: ReadFederationLevelConfigFromConfigStoreAsync, IsSuccess: True DocDBTrace Information: 0 : FederationConfiguration initialized property SecondarySystemKeyReadWriteEncrypted DocDBTrace Information: 0 : FederationConfiguration initialized property federationEndpoint = http://127.0.0.1/ DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A DocDBTrace Information: 0 : FederationConfiguration initialized primary compute key with hash = 7B76D81B9CB723B1D0FEF518331D8DC853E8899AC0D13AF87FD1BE357CC835C8 DocDBTrace Information: 0 : FederationConfiguration initialized property UsePrimarySystemKeyReadWrite = true DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A DocDBTrace Information: 0 : FederationConfiguration initialized secondary compute key SecondaryComputeGatewayKeyEncrypted with hash = 38C6CD57974A7E24C49321C44F7D94463F21DAB1FC1C99B2164184486AC09ABA DocDBTrace Information: 0 : FederationConfiguration initialized property resourceCacheRefreshIntervalInSeconds = 60 DocDBTrace Information: 0 : FederationConfiguration initialized property availableBindableServices DocDBTrace Information: 0 : FederationConfiguration initialized property UsePrimarySystemKeyReadOnly = true DocDBTrace Information: 0 : FederationConfiguration initialized property serviceCosmosDBGatewayServiceRunningMode = Idle DocDBTrace Information: 0 : FederationConfiguration initialized property replicatorUpgradePhase = 5 DocDBTrace Information: 0 : FederationConfiguration initialized property IsGremlinEndpointEnabled = false DocDBTrace Information: 0 : FederationConfiguration initialized property IsMongoTcpEndpointEnabled = false DocDBTrace Information: 0 : FederationConfiguration initialized property federationGenerationKind = gen3 DocDBTrace Information: 0 : FederationConfiguration initialized property nodeCountOfRoleHostingGateway = 1 DocDBTrace Information: 0 : FederationConfiguration initialized property isStrictServiceStateCheckEnabled = true DocDBTrace Information: 0 : FederationConfiguration initialized property federationName = emulatorfederation DocDBTrace Information: 0 : FederationConfiguration initialized property configurationRefreshIntervalInSec = 30 DocDBTrace Information: 0 : FederationConfiguration initialized property UsePrimarySystemKeyAll = true DocDBTrace Information: 0 : FederationConfiguration initialized property gatewayNamingServiceConfigRefreshIntervalInSeconds = 30 DocDBTrace Information: 0 : FederationConfiguration initialized property IsSqlComputeEndpointEnabled = false DocDBTrace Information: 0 : FederationConfiguration initialized property IsTableEndpointEnabled = false DocDBTrace Information: 0 : FederationConfiguration initialized property region = South Central US DocDBTrace Information: 0 : FederationConfiguration initialized property enableRG = false DocDBTrace Information: 0 : FederationConfiguration initialized property reservedDnsName = https://localhost:443/ DocDBTrace Information: 0 : FederationConfiguration initialized property PrimarySystemKeyReadOnlyEncrypted DocDBTrace Information: 0 : FederationConfiguration initialized property isServiceReuseEnabled = true DocDBTrace Information: 0 : FederationConfiguration initialized property PrimarySystemKeyAllEncrypted DocDBTrace Information: 0 : FederationConfiguration initialized property isAvailabilityZoneFederation = false DocDBTrace Information: 0 : FederationConfiguration initialized property siblingFederations = [{"SiblingFederationId":"emulatorfederation","FabricClientEndpoints":["127.0.0.1:9000"],"FabricClientAdminAccessCertificateCommonName":"","EnabledForCrossFederationDatabaseAccountScaleout":true}] DocDBTrace Information: 0 : FederationConfiguration initialized property IsCassandraTcpEndpointEnabled = false DocDBTrace Information: 0 : FederationConfiguration initialized property isCrossFederationDatabaseAccountScaleoutEnabled = true DocDBTrace Information: 0 : FederationConfiguration initialized property IsDataTransferControlPlaneEndpointEnabled = false DocDBTrace Information: 0 : FederationConfiguration initialized property SecondarySystemKeyAllEncrypted DocDBTrace Information: 0 : FederationConfiguration initialized property SecondarySystemKeyReadOnlyEncrypted DocDBTrace Information: 0 : FederationConfiguration initialized property PrimarySystemKeyReadWriteEncrypted DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: ETag: old , current ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A DocDBTrace Warning: 0 : InitializeKeysInFederationConfiguration: Failed querying federationVisibilityConfigurationWithAllKeys for sibling federation emulatorfederation DocDBTrace Information: 0 : InitializeKeysInFederationConfiguration: Completed initializing systemkeys for local federation DocDBTrace Information: 0 : GetSiblingFederations : Skipping sibling federation config fetch in gateway boot ReadAllFederationLevelConfigsAsync DocDBTrace Information: 0 : CompositeConfigurationReader.ListPrivateEndpointMapsAsync, region = southcentralus, callingComponent = GatewayService-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: PrivateEndpointMapsConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : GetSiblingFederations : Skipping sibling federation config fetch in gateway boot RefreshAllAvailableBindableServicesAsync DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllAvailableBindableServicesAsync, callingComponent InitializeAsync federations emulatorfederation DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True DocDBTrace Information: 0 : GetSiblingFederations : Skipping sibling federation config fetch in gateway boot RefreshAllServiceManagerConfigurationsAsync DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllServiceManagerConfigurationsAsync, callingComponent InitializeAsync federations emulatorfederation DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True DocDBTrace Information: 0 : RefreshFederationCapacityInformation. Using default value of all federations emulatorfederation,emulatorfederation DocDBTrace Information: 0 : RequestSamplingConfigurations changed. enableRequestSampling: true, minimumRequestSamplingLatencyThresholdInMilliseconds: 2500, requestSamplingCategories: Document,DocumentFeed, requestSamplingPercent: 30 DocDBTrace Information: 0 : CompositeConfigurationReader ListOffersConfigurationAsync callingComponent = GatewayService-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Starting periodic timer for callingComponent PeriodicallyRefreshFederationConfigurationAsync with refresh time(sec) 38 DocDBTrace Information: 0 : Starting periodic timer for callingComponent StartPrivateEndpointMapConfigurationRefreshTimerAsync with refresh time(sec) 30 DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: ConfigurationProvider initialized local federation configuration successfully DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: ConfigurationProvider initialized successfully DocDBTrace Information: 0 : GatewayWarmpathAggregator: requestaggregation is disabled DocDBTrace Information: 0 : Initializing MutualTlsClient with the following configuration: cloudName=Microsoft enableMtlsAuthentication=True allowedSubjectNames=cosmosdbemulatormtls.localhost,client.mtls.cdb.core.azure-test.net deniedThumbprints= allowedThumbprints=4E30B33D02ABE0CFED415A722F45FBF361BE243F optionalBase64EncodedAllowedIssuersCount=0 DocDBTrace Information: 0 : Refresh private endpoint map start DocDBTrace Information: 0 : CompositeConfigurationReader.ListPrivateEndpointMapsAsync, region = southcentralus, callingComponent = GatewayService-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: PrivateEndpointMapsConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Warning: 0 : Private Endpoint Map Refresh fails. Will retry in the next cycle. DocDBTrace Information: 0 : Starting periodic timer for callingComponent StartPrivateEndpointMapConfigurationRefreshTimerAsync with refresh time(sec) 30 DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: MutualTlsClient initialized: True DocDBTrace Information: 0 : NetworkIPRangePolicy Intializing IP ranges for 0, with C:\Packages\GatewayService\GatewayService.Code\TestNetworksFromVM.xml last written at: 05/21/2024 13:29:12. DocDBTrace Information: 0 : Networks.xml file path is C:\Packages\GatewayService\GatewayService.Code\TestNetworksFromVM.xml DocDBTrace Information: 0 : NetworkIPRangePolicy.SetRegionalIPRanges() Count = 676 DocDBTrace Information: 0 : NetworkIPRangePolicy.SetAllAzureIpRanges() Count = 10417 DocDBTrace Information: 0 : regionName = South Central US, IsEmulated = True, regionId is set to 35 DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: NetworkIPRangePolicy initialized successfully DocDBTrace Information: 0 : GatewayService OpenAsync configurations - RequestTimeoutInSeconds: 10, MaxConcurrentConnectionOpenRequestsPerProcessor: 25, ProcessorCount: 4, BackendConnectionOpenTimeoutInSeconds: 1, IdleConnectionTimeoutInSeconds: 1800, TransportTimerPoolGranularityInSeconds: 1, MaxRntbdChannels: 65535, RntbdPartitionCount: 1, MaxRequestsPerRntbdChannel: 12, PortResuseMode: PrivatePortPool, RntbdPortPoolReuseThreshold: 256, RntbdPortPoolBindAttempts: 5, RntbdChannelReceiveHangDetectionTimeSeconds: 65, RntbdChannelSendHangDetectionTimeSeconds: 10, GatewayClientRequestTimeoutInSeconds: 60, UseGatewayToReadDatabaseAccountConfiguration: False, HttpEndPointPort: 10000, HttpEndPointProbePort: 81, FabricNameResolutionTimeoutInSeconds: -1, NameresolutionTimeoutInSeconds: 10, IsGatewayLocalhostOnlyString: false, IPv6ListenersEnabledForEmulator: False. IsMongoProxyEnabled: FalseHttpEndPointProbePathSettingName: /probe, RegionName: South Central US, ProxyEndPointProbePathSettingName: /proxyprobe, MaxServicePoolProbeCount: 25InitialRetryIntervalMilliseconds: 10, MaximumRetryIntervalMilliseconds: 1000, RandomSaltMaxValueMilliseconds: 5, TotalWaitTimeMilliseconds: 30000, EnableStorageServiceConfigurationLoadOnBoot False IsKeyBasedEncryptionEnabledForStorage: False, EnableMutualTlsServerSide: True, EnableMutualTlsServerSideAuthorization: True, EnableMutualTlsClientSideUsage: False DocDBTrace Information: 0 : Setting up Distributed tracing DocDBTrace Information: 0 : Enabling/Disabling Distributed Tracing for SDK: True DocDBTrace Information: 0 : CertificateUtil: Returning certificate CN=localhost(4E30B33D02ABE0CFED415A722F45FBF361BE243F) for FindByThumbprint '4E30B33D02ABE0CFED415A722F45FBF361BE243F' from LocalMachine\My DocDBTrace Information: 0 : Creating RNTBD TransportClient with options Rntbd.TransportClient.Options OpenTimeout: 00:00:01 RequestTimeout: 00:00:10 TimerPoolResolution: 00:00:01 MaxChannels: 65535 PartitionCount: 1 MaxRequestsPerChannel: 12 ReceiveHangDetectionTime: 00:01:05 SendHangDetectionTime: 00:00:10 IdleTimeout: 00:30:00 UserAgent: Microsoft.Azure.Documents.Common/2.14.0RoutingGateway Suffix: RoutingGateway CertificateHostNameOverride: localhost LocalRegionTimeout: 00:00:01 EnableChannelMultiplexing: False MaxConcurrentOpeningConnectionCount: 65535 Use_RecyclableMemoryStream: False Use_CustomDnsResolution: False IsDistributedTracingEnabled: True DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 1 DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 30 DocDBTrace Information: 0 : Creating RNTBD TransportClient with options Rntbd.TransportClient.Options OpenTimeout: 00:00:01 RequestTimeout: 00:00:10 TimerPoolResolution: 00:00:01 MaxChannels: 65535 PartitionCount: 1 MaxRequestsPerChannel: 1 ReceiveHangDetectionTime: 00:01:05 SendHangDetectionTime: 00:00:10 IdleTimeout: 00:30:00 UserAgent: Microsoft.Azure.Documents.Common/2.14.0RoutingGateway Suffix: RoutingGateway CertificateHostNameOverride: localhost LocalRegionTimeout: 00:00:01 EnableChannelMultiplexing: False MaxConcurrentOpeningConnectionCount: 65535 Use_RecyclableMemoryStream: False Use_CustomDnsResolution: False IsDistributedTracingEnabled: True DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 1 DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 30 DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: CreateNameAsync, ConfigurationLevel: , IsWriteRequest: True, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: Transaction log manager initialized successfully DocDBTrace Information: 0 : Started listening on address: https://0.0.0.0:10000/ DocDBTrace Information: 0 : Started listening on probe address: http://0.0.0.0:81/ DocDBTrace Information: 0 : BEGIN: StartAsync() DocDBTrace Information: 0 : Listening on: https://0.0.0.0:10000/ DocDBTrace Information: 0 : MaxAttachmentContentSize config setting is 0 DocDBTrace Information: 0 : SingleBlobUploadThresholdInBytes config setting is 8388608 bytes DocDBTrace Information: 0 : AzureBlobUploadTimeoutSeconds config setting is 00:01:30 seconds DocDBTrace Information: 0 : MaxAttachmentContentSize config setting is 0 DocDBTrace Information: 0 : SingleBlobUploadThresholdInBytes config setting is 8388608 bytes DocDBTrace Information: 0 : AzureBlobUploadTimeoutSeconds config setting is 00:01:30 seconds DocDBTrace Information: 0 : SingleBlobUploadThresholdInBytes config setting is 8388608 bytes DocDBTrace Information: 0 : AzureBlobUploadTimeoutSeconds config setting is 00:01:30 seconds DocDBTrace Information: 0 : AzureBlobDownloadTimeoutSeconds config setting is 00:01:30 seconds DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating DrainEntityBody from default to 120 DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating EntityBody from default to 120 DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating MinSendBytesPerSecond from default to 8192 DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating RequestQueue from default to 120 DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating HeaderWait from default to 30 DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating IdleConnection from default to 120 DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: New timeouts: HttpTimeoutLimitInfo { DrainEntityBodyInSeconds = 120, EntityBodyInSeconds = 120, HeaderWaitInSeconds = 30, IdleConnectionInSeconds = 120, MinSendBytesPerSecond = 8192, RequestQueueInSeconds = 120 } DocDBTrace Information: 0 : Enabling DistributedTracing for HttpRequestDispatcher, Master Switch ON DocDBTrace Information: 0 : HttpRequestDispatcher: HttpListener started on 1 prefixes. (HttpTimeoutLimitInfo { DrainEntityBodyInSeconds = 120, EntityBodyInSeconds = 120, HeaderWaitInSeconds = 30, IdleConnectionInSeconds = 120, MinSendBytesPerSecond = 8192, RequestQueueInSeconds = 120 }) DocDBTrace Information: 0 : END: StartAsync() DocDBTrace Information: 0 : GatewayRequestProcessor useGatewayToReadDatabaseAccountConfiguration = False DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: CreateNameAsync, ConfigurationLevel: , IsWriteRequest: True, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: CreateNameAsync, ConfigurationLevel: , IsWriteRequest: True, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: GatewayRequestProcessor initialized successfully DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: GatewayDatabaseAccountConfigurationProviderV2 DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: GatewayDatabaseAccountBackendResourceProvider DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: GatewayDatabaseAccountConfigurationProvider DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: DatabaseAccountRequestProcessorFactory DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: GatewayCacheMetricsPublisher initialized successfully DocDBTrace Information: 0 : Emitting RoutingGatewayOpen metric with dimensions as IsSuccess: True DocDBTrace Information: 0 : Emitting RoutingGatewayOpenLatency metric with no additional dimensions. DocDBTrace Information: 0 : GatewayService: END OpenAsync DocDBTrace Information: 0 : Emitting FederationConfigurationRefresh metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Invoked from PeriodicallyRefreshFederationConfigurationAsync DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFederationConfigAsync, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : ReadFederationLevelConfigFromConfigStoreAsync: Generate compositeETag ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= for local federation decoupleAvailableBindableService True DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: ReadFederationLevelConfigFromConfigStoreAsync, IsSuccess: True DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: ETag: old ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4=, current ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Skipping sibling federation config refresh DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllAvailableBindableServicesAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllServiceManagerConfigurationsAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: ServerService and bindable capacity: 5 DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: MasterService and bindable capacity: 5 DocDBTrace Information: 0 : CompositeConfigurationReader ListOffersConfigurationAsync callingComponent = GatewayService-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Refreshing Mutual TLS configurations. DocDBTrace Information: 0 : Starting periodic timer for callingComponent PeriodicallyRefreshFederationConfigurationAsync with refresh time(sec) 38 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: cpu-usage, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: working-set, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gc-heap-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-0-gc-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-1-gc-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-2-gc-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: threadpool-thread-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: monitor-lock-contention-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: threadpool-queue-length, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: threadpool-completed-items-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: alloc-rate, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: active-timer-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gc-fragmentation, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gc-committed, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: exception-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: time-in-gc, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-0-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-1-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-2-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: loh-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: poh-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: assembly-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: il-bytes-jitted, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: methods-jitted-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: time-in-jit, Type: Sum, ServiceName: GatewayService, ProcessId: 452 DocDBTrace Information: 0 : Refresh private endpoint map start DocDBTrace Information: 0 : CompositeConfigurationReader.ListPrivateEndpointMapsAsync, region = southcentralus, callingComponent = GatewayService-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: PrivateEndpointMapsConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Warning: 0 : Private Endpoint Map Refresh fails. Will retry in the next cycle. DocDBTrace Information: 0 : Starting periodic timer for callingComponent StartPrivateEndpointMapConfigurationRefreshTimerAsync with refresh time(sec) 30 DocDBTrace Information: 0 : Emitting FederationConfigurationRefresh metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Invoked from PeriodicallyRefreshFederationConfigurationAsync DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFederationConfigAsync, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : ReadFederationLevelConfigFromConfigStoreAsync: Generate compositeETag ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= for local federation decoupleAvailableBindableService True DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: ReadFederationLevelConfigFromConfigStoreAsync, IsSuccess: True DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: ETag: old ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4=, current ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Skipping sibling federation config refresh DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllAvailableBindableServicesAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllServiceManagerConfigurationsAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: ServerService and bindable capacity: 5 DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: MasterService and bindable capacity: 5 DocDBTrace Information: 0 : CompositeConfigurationReader ListOffersConfigurationAsync callingComponent = GatewayService-GatewayConfigurationReaderProvider DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: DocDBTrace Information: 0 : Refreshing Mutual TLS configurations. DocDBTrace Information: 0 : Starting periodic timer for callingComponent PeriodicallyRefreshFederationConfigurationAsync with refresh time(sec) 38 ```
Output of lsof -i ``` root@cosmosdb-775768b69b-h78h5:/usr/local/bin/cosmos# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME cosmosdb- 19 root 73u IPv4 951851706 0t0 TCP *:10251 (LISTEN) cosmosdb- 19 root 75u IPv4 951853360 0t0 TCP *:10253 (LISTEN) ``` `lsof -i :8081` (or port 10000) doesn't list anything. `curl --insecure https://localhost:8081` (or port 10000) fails to call. But this doesn't seem to be a general issue. On our Rancher K8S Cluster it doesn't work, on docker-desktop CosmosDB starts listening. Using `top` I can see the cosmosdb emulator running and consuming 10-20% CPU.
aratz-lasa commented 4 months ago

@v1k1 @niteshvijay1995 Can you look at this ?

Is there an ETA?

HEllRZA commented 4 months ago

It's really slow to be ready ... after the container shows started it still takes a while. I wrote this little bash oneliner to ensure Cosmos DB has started completely.

echo -n "Waiting for Cosmos DB emulator ..."; while true; do result=$(curl -s -k https://localhost:8081 | jq -r '.code'); if [ "$result" = "Unauthorized" ]; then break; fi; echo -n "."; sleep 2; done; echo " ready."

And it produces this output when run with time command:

Waiting for Cosmos DB emulator ........................................................ ready.

real    1m47.743s
user    0m0.941s
sys     0m0.098s

And that's on a i7 12700k, the database is basically empty. (2 datasets).

@aratz-lasa ETA would be 2 minutes then

Cotspheer commented 4 months ago

I experience the same really slow startup. It is a real pain as sometimes I have to wait 5 to 10 minutes until I can debug one line of code. Reducing the partition count to 1 did improve the time to start a bit but not really as much as I hoped.

Current setup:

I'm using Aspire 8.0.2 and I have a small application that uses CosmosDB for persistence. I want to run IntegrationTests using the DistributedApplicationTestingBuilder. I currently have to poll the "/alive" endpoint and added AzureCosmosDB-HealthCheck that has the "live"-Tag, so the application is only considered alive / running when the cosmos emulator instance is accessible.

On the other hand the Azurite-Instance that I have to run as well is up almost immediately.

using Projects;

var builder = DistributedApplication.CreateBuilder(args);

var azureStorage = builder.AddAzureStorage("storage")
                    .RunAsEmulator(r =>
                        r.WithArgs("azurite", "-l", "/data", "--blobHost", "0.0.0.0", "--queueHost", "0.0.0.0", "--tableHost", "0.0.0.0", "--skipApiVersionCheck")
                    );

// ## Synchronization

// Aspire currently only supports Azure Storage for Orleans.
var synchronizationClusterStore = azureStorage.AddTables("synchronization-cluster-table");
var synchronizationStateStore = azureStorage.AddBlobs("synchronization-grains-state");

var synchronizationOrleans = builder.AddOrleans("synchronization-cluster")
                     .WithClustering(synchronizationClusterStore)
                     .WithGrainStorage(synchronizationStateStore)
                     ;

var synchronizationInboxDatabase = builder.AddAzureCosmosDB("SynchronizationInboxDatabase")
    .RunAsEmulator(r => r
        .WithEnvironment("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", "127.0.0.1")
        .WithEnvironment("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "1")
    );

builder.AddProject<Synchronization_Silo>("synchronization")
    .WithReference(synchronizationOrleans)
    .WithReference(synchronizationInboxDatabase)
    .WithExternalHttpEndpoints()
    .WithReplicas(1)
    .WithEnvironment("ASPNETCORE_ENVIRONMENT", builder.Environment.EnvironmentName)
    .WithEnvironment("DOTNET_ENVIRONMENT", builder.Environment.EnvironmentName)
    ;

using var app = builder.Build();
await app.RunAsync();
HeneryHawk commented 4 months ago

Regarding the very slow startup, we have implemented a workaround that pings /_explorer/emulator.pem until this URL is accessible. This has now worked well for a while, but further errors have been occurring for a few weeks now. For inexplicable reasons, our Gradle tests lose the connection to the emulator during execution in the pipeline. So the tests fail and so does the pipeline.

ERROR  c.a.c.i.GlobalEndpointManager - Fail to reach global gateway [https://172.17.0.2:8081/]
ERROR  c.a.c.i.GlobalEndpointManager - startRefreshLocationTimerAsync() - Unable to refresh database account from any location.

We have not yet found a workaround for this and the Gradle tests regularly break in the pipeline. It sometimes takes 5 - 10 executions until the test job is successful.

Is there an update for this issue? Or is there an update on the way? Cosmos DB is the flagship database on Azure and is advertised by Microsoft. It is a great pity that the associated emulator really does not work well. I would expect an emulator that is regularly maintained, starts quickly, works without major problems and fulfills expectations. Unfortunately, this is not currently the case. The instability of the Linux emulator is causing us more and more problems, our development is slowing down and we are losing more and more money as a result. Please Microsoft, do something about it and satisfy the expectations of the developers. That should also be your own standard.

@sajeetharan @v1k1

eptw4by commented 3 months ago

We also experience same problem now: on Ubuntu 20 host cosmosDB emulator prints Started in console, but would never be able to serve the certificate for Java to connect to it. However, in my case (I am executing tests in emulator on pretty beefy VMs using Ubuntu 20 as OS) problem was in disabled IPv6. AFter enabling it - emulator started pretty fast

seantleonard commented 1 month ago

I'm experiencing the same issue where the CosmosDB Emulator takes about 2m45s on Azure Dev Ops Windows images, even though it is pre-installed. This is a lot of time that slows down testing.

isaranghi commented 1 month ago

I am running CosmosDb emulator as docker container for executing integration tests... but the emulator takes so long to start (even after container is up)... all tests are failing..... is there any workaround?

chrisflem commented 1 month ago

@isaranghi The problem is that cosmosdb is not ready even after the container is started. To get around this I poll a known address on it, until I get a http 200 reply. I'm using testcontainers to start the cosmosdb docker image. It's kinda slow, but it gets the job done.

Here is the code i use to start the container:

public class DatabaseHandler : IAsyncLifetime
{
    private readonly CosmosDbContainer _dbContainer = new CosmosDbBuilder()
        .WithImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
        .Build();

    public CosmosClient? CosmosClient;

    public async Task InitializeAsync()
    {
        await _dbContainer.StartAsync();
        var connStr = _dbContainer.GetConnectionString();
        await CosmosHelper.PollForDockerImageReadyness(connStr);

        CosmosClient = new CosmosClient($"{connStr};DisableServerCertificateValidation=True;", new CosmosClientOptions
        {
            ConnectionMode = ConnectionMode.Gateway,
            HttpClientFactory = () => _dbContainer.HttpClient
        });
    }

    public async Task DisposeAsync()
    {
        await _dbContainer.StopAsync();
    }
}

Here is the code used to poll for cosmos readyness:

public class CosmosHelper
{
......

  /// <summary>
  ///  Fix for error in docker image for cosmosdb(Hopefully fixed soon). It says it's ready before the container is ready.
  /// Will poll until we get a http 200 response for a known correct url
  /// </summary>
  /// <param name="connectionstring"></param>
  /// <returns></returns>
  public static async Task PollForDockerImageReadyness(string connectionstring)
  {
      const int delayInMs = 2000;
      const int maxAttempts = 30;
      var currAttempt = 0;
      var splitArr = connectionstring.Split(';', '=');
      var parsedHostAndPort = splitArr[1];
      var pathToTest = $"{parsedHostAndPort}/_explorer/emulator.pem";
      using var client = new HttpClient(new HttpClientHandler { ServerCertificateCustomValidationCallback = (_, _, _, _) => true });
      do
      {
          currAttempt++;
          if (currAttempt > maxAttempts)
              throw new InvalidOperationException("Cannot connect to cosmosdb");
          await Task.Delay(delayInMs);
          try
          {
              await client.GetByteArrayAsync(pathToTest);
              return;
          }
          catch (Exception)
          {
              // Ignored
          }
      } while (true);
  }
}
kntajus commented 1 month ago

@chrisflem You don't need to do that manual check with TestContainers, they've already been doing that check for you since version 3.8.

(You also don't need to specify the Docker image manually at the top, since that's what they use anyway.)