CosmosDB Emulator Hangs When Connecting via Connection String

sami-sharma commented 2 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug

Description: I have been using Aspire to locally test a service with Azure CosmosDB. Following the tutorial here and using RunAsEmulator().

Issue: My service uses IHostBuilder which is incompatible with the AddAzureCosmosClient method from Aspire.Microsoft.Azure.Client. Therefore, I use a callback to pass the connection string to the CosmosDB Emulator as an environment variable to my service. This is the same connection string that Aspire generates and uses in 'AddAzureCosmosClient'.

I have been able to successfully connect to the Storage Emulators and EventHub Emulator the same way.

Symptoms:

I can connect to the instance and perform some functions, such as reading account properties.
When attempting to create a database (CreateDatabaseAsync(), CreateDatabaseIfNotExistsAsync(), etc.), the application hangs (no exception thrown, no timeout).

Attempts to Resolve:

Tried using these methods synchronously with GetAwaiter().GetResult().
Tried using these methods asynchronously with await.
Attempted to use the Azure CosmosDB Emulator directly (without Aspire) which works with the same code.

This is the relevant part of my AppHost Program.cs file:

This is the relevant part of my Service Program.cs file:

Expected Behavior

The application should create the database without hanging.

Steps To Reproduce

I have used the 'CosmosEndToEnd' example from the Aspire repository and modified the 'Program.cs' file of the API project. The only change I made was to create the Cosmos Client manually with the connection string within an endpoint mapping, instead of adding it in the builder configuration with builder.AddAzureCosmosClient("cosmos"). However, the application hangs when attempting to create a database this way.

Here are the relevant files of code:

CosmosEndToEnd.ApiService

CosmosEndToEnd.AppHost

This is where I have forked the Aspire code: https://github.com/sami-sharma/CosmosDbEmulatorTest. Please view the CosmosEndToEnd folder within playground.

Exceptions (if any)

No response

.NET Version info

Aspire: 8.1.1-preview.1.24373

.NET SDK : 8.0.400

Anything else?

I have used the EventHub Emulator and Storage emulators connecting via connection strings manually and have not faced issues connecting.

sami-sharma commented 2 months ago

UPDATE:

I was able to identify that the hanging of the emulator is caused by the lack of CosmosClient configurations (LimitEndpoint = True and GatewayMode = Connection). Unfortunately, in my service the CosmosClient is being used by a transitive dependency and I cannot change how the client is initialized. I urge the Aspire team to configure the emulator/the corresponding connection string to not require these additional CosmosClient properties when connecting to the emulator via the connection string.

davidfowl commented 1 month ago

@mitchdenny @eerhardt We make assumptions in the aspire client integration based on that we set options like gateway mode and limit to endpoint right? I'm not sure we can do anything here to change how arbitrary clients get wired up.

This might be feedback for @Pilchie

Pilchie commented 1 month ago

@sami-sharma - can you report this over at the emulator repo to see why it might be hanging: https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/ ?

ggirard07 commented 1 month ago

There is this issue about emulator startup that has been a pain for quite some time now, even without Aspire stuff. Not sure if it could be linked to OP issue... https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/87

rorozcov commented 3 weeks ago

I will say that I have been working with the CosmosDB emulator just as .NET Aspire's documentation described and it is quite unreliable. I suspect it is because of the issue @ggirard07 mentioned.

@davidfowl Since the CosmosDB emulator takes so long to start, I am wondering if there is a conversation with the team in charge of the emulator, since Aspire will report the emulator as running where it clearly isn't available for use.

This makes it hard to use in the AppHost or via integration testing since we have to wait for minutes before we can trust that it is usable.

I haven't looked at the code but I do wonder if for Azurite, Aspire also checks for a heartbeat before reporting as running.

My experience using Azurite with Aspire has been nothing but fantastic!

davidfowl commented 3 weeks ago

We have 2 features that help with this in Aspire 9:

The ability to wait for resources to start and be healthy. So you can avoid starting your application (after the 2 or so minutes cosmos takes to be healthy) before cosmos is ready to work.
The ability to mark a container as persistent. This helps amortize the cost of slow starting containers (as it keeps them alive).

ggirard07 commented 3 weeks ago

@davidfowl is Aspire 9 going to be dotnet 8 compatible? This is not obvious as it seems Aspire follows the dotnet version. The 2 points you are mentioning are kind of deal breaker for us right now as we heavily leverage those features in current Docker Desktop integration.

davidfowl commented 3 weeks ago

@davidfowl is Aspire 9 going to be dotnet 8 compatible?

Yes.

The 2 points you are mentioning are kind of deal breaker for us right now as we heavily leverage those features in current Docker Desktop integration.

Then you're going to love using 9.x 😄.

joperezr commented 1 week ago

@Pilchie can you please take a look on this one?

Pilchie commented 1 week ago

As I said above - please report at https://github.com/Azure/azure-cosmos-db-emulator-docker/issues

dotnet / aspire