j82w
commented
4 years ago Is it just query or do point operations also get stuck?
Closed lukasz-pyrzyk closed 4 years ago
j82w
commented
4 years ago Is it just query or do point operations also get stuck?
lukasz-pyrzyk
commented
4 years ago Yes. It gets stuck on point operation with ReadItemAsync and Stored Procedure execution.
Both cases got stuck on the LoadBalancingPartition.RequestAsync. Parallel stacks below
ReadItemAsync:
Stored Procedure:
I think the issue might be somehow related to the thread context.
My app has background thread which scans the database for idler users. It uses the same base provider, repository etc. Queries done by this background thread complete successfully, even when other web threads are hang.
When thread from System.Web touches DB for second time, it gots stuck.
j82w
commented
4 years ago Are you using the latest 3.4.1 .NET SDK? There was a bug in version before 3.2.0 that was fixed in PR #761 . Troubleshooting this will be a lot easier once you provide the repo.
lukasz-pyrzyk
commented
4 years ago Yes, I use the latest SDK. @j82w, the smallest repro is in https://github.com/lukasz-pyrzyk/CosmosDbSyncOverAsync . You need to setup environment variables to connect to the database, I've added notes to the readme.
ealsur
commented
4 years ago Are you using IHttpHandler in the real solution or was that only for the repro?
lukasz-pyrzyk
commented
4 years ago It's taken from the real app
ealsur
commented
4 years ago Have you consider IHttpAsyncHandler? Since this involves an async operation?
You mention that you were using V2 SDK, could you share how were the queries being executed?
For example, What was the V2 version of :
var dbClient = _containerFactory;
var partitionKey = new PartitionKey(clientId.ToString());
var query = dbClient
.GetItemLinqQueryable<ClientEntity>(requestOptions: new QueryRequestOptions { PartitionKey = partitionKey })
.Where(x => x.ClientId == clientId)
.ToFeedIterator();
var client = await query.ExecuteFirstOrDefault(nameof(GetClient)).ConfigureAwait(false);
return client;I also managed to get the same deadlock using the V2 SDK client on the Repro project, calling DocumentClient.ReadDatabaseAsync from within the HeartbeatRouteHandler locks in the same way.
lukasz-pyrzyk
commented
4 years ago Have you consider IHttpAsyncHandler? Since this involves an async operation?
I did, unfortunately I cannot move. I'm reusing 3rd party handlers and injecting my CosmosDBRepository to them with a strictly coupled synchronous interface. That's why I did ugly sync-over-async.
You mention that you were using V2 SDK, could you share how were the queries being executed?
Sure. I've added new branch called v2 to the repository, https://github.com/lukasz-pyrzyk/CosmosDbSyncOverAsync/tree/v2
I also managed to get the same deadlock using the V2 SDK client on the Repro project, calling DocumentClient.ReadDatabaseAsync from within the HeartbeatRouteHandler locks in the same way.
Can you show me the code? In the branch, I've committed a code which works correctly for multiple requests, at least on my machine :)
ealsur
commented
4 years ago To rule out anything about Queries, I added a method on the Repository:
public async Task<bool> CheckIfDatabaseExistsV2()
{
try
{
await _oldClient.ReadDatabaseAsync(UriFactory.CreateDatabaseUri("Test"));
return true;
}
catch (DocumentClientException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
{
return false;
}
}The _oldClient is created on the constructor:
_oldClient = new DocumentClient(new Uri(_settings.Endpoint), _settings.Key, new ConnectionPolicy()
{
ConnectionMode = _settings.UseDirectConnectionMode ? Microsoft.Azure.Documents.Client.ConnectionMode.Direct : Microsoft.Azure.Documents.Client.ConnectionMode.Gateway
});Then from inside the RouteHandler, I call:
repository.CheckIfDatabaseExistsV2().ConfigureAwait(false).GetAwaiter().GetResult();And it locks.
I switched to Gateway mode to rule out any Direct mode issues.
ealsur
commented
4 years ago It looks like the issue is, when running async over sync, the Synchronization Context.
If I put the V2 ReadDatabaseAsync call inside the Repository class and call it with GetAwaiter from the Handler, it locks (the Repository class has an await).
But, if I call the V2 ReadDatabaseAsync call from the Handler, with GetAwaiter, it works. (leaving the Repository class out of the picture).
One easy workaround for this is to wrap any call you make around Task.Run, for example:
ClientEntity clientEntity;
Task.Run(() =>
{
clientEntity = repository.GetClient(clientId).ConfigureAwait(false).GetAwaiter().GetResult();
}
).Wait();Thanks @ealsur for digging in! The issue is definitely related to the Synchronization Context. I tried to reproduce the issue with xunit tests and it doesn't occur, only with ASP.NET which has a synchronization context.
According to your sample, I would like to point out one thing. You have a hang on V2 SDK because you don't have ConfigureAwait(false) on the async call inside repository. When you add it, the hang does not occur.
I've updated my samples to don't have any query-related stuff.
SDK V3, hangs:
repository.ReadDatabase().ConfigureAwait(false).GetAwaiter().GetResult();
[...]
internal async Task ReadDatabase()
{
try
{
var client = await clientFactory;
await client.ReadAsync().ConfigureAwait(false);
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
SDK V2, doesn't hang
repository.ReadDatabase().ConfigureAwait(false).GetAwaiter().GetResult();
[..]
internal async Task ReadDatabase()
{
try
{
var client = await _clientFactory;
await client.ReadDatabaseAsync(UriFactory.CreateDatabaseUri(_settings.DatabaseId)).ConfigureAwait(false);
}
catch (Exception e)
{
Trace.WriteLine(e);
throw;
}
}```Thanks for the catch! If you wrap your call in the RouteHandler in Task.Run, does it work correctly in your real project?
lukasz-pyrzyk
commented
4 years ago Well, in real app I have some kind of dynamic injecting and abstraction, but I quickly checked - With Task.Run it doesn't hang
ealsur
commented
4 years ago The linked PR was tested on the repro Github and it fixes the reported issue.
lukasz-pyrzyk
commented
4 years ago @ealsur thank you very much! If you have some kind of night builds, I will be happy to test it in the real app.
Describe the bug I work on the migration from SDK v2 to SDK v3 in the ASP.NET web project which runs on the .NET Framework 4.6.1. Because of the contract of the 3rd party library, I'm forced to do sync-over-async calls.
After migrating to SDK v3, I notice that my app hangs on communication to the database. First query to the database completes, but any other hangs.
I was thinking that some `ConfigureAwait(false) is missing any my thread cannot continue the asynchronous operation, but it's added to all async calls.
Tested code on gateway and direct mode. Screenshots and call stack from direct mode below. Is it normal that is uses
GatewayStoreClientunder the hood? The name suggest that Direct mode uses gateway client.To Reproduce I cannot give you repro right now, I will try to do it this week. Parts of the code:
Expected behavior All DB call, even with sync-over-async should be succeeded. SDK v2 worked correctly.
Actual behavior Only the first query to the database completes, any other is hanging for infinite amount of time.
Environment summary SDK Version: 3.4.1 OS Version: Windows 10
Additional context
List of all tasks:
Stack trace of the hanging thread: