search

microsoft / durabletask-mssql

Microsoft SQL storage provider for Durable Functions and the Durable Task Framework
MIT License
87 stars 31 forks source link

Add resiliency to SQL data reader read operations #151

Open cgillum opened 1 year ago

cgillum commented 1 year ago

I noticed the following failure in the CI:

[xUnit.net 00:00:06.69]     DurableTask.SqlServer.Tests.Integration.DataRetentionTests.InvalidTimerEventsAreDiscarded [FAIL]
  Failed DurableTask.SqlServer.Tests.Integration.DataRetentionTests.InvalidTimerEventsAreDiscarded [3 s]
  Error Message:
   Microsoft.Data.SqlClient.SqlException : Transaction (Process ID 92) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
  Stack Trace:
     at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at Microsoft.Data.SqlClient.SqlDataReader.Read()
   at DurableTask.SqlServer.SqlOrchestrationService.GetOrchestrationStateAsync(String instanceId, String executionId, CancellationToken cancellationToken) in /home/runner/work/durabletask-mssql/durabletask-mssql/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 607
   at DurableTask.SqlServer.Tests.Utils.TestInstance`1.WaitForStart(TimeSpan timeout) in /home/runner/work/durabletask-mssql/durabletask-mssql/test/DurableTask.SqlServer.Tests/Utils/TestInstance.cs:line [56](https://github.com/microsoft/durabletask-mssql/actions/runs/4107266240/jobs/7086524989#step:7:57)
   at DurableTask.SqlServer.Tests.Integration.DataRetentionTests.InvalidTimerEventsAreDiscarded() in /home/runner/work/durabletask-mssql/durabletask-mssql/test/DurableTask.SqlServer.Tests/Integration/DataRetentionTests.cs:line 218

We have transient failure handling for the ExecuteReaderAsync call, but not for any subsequent Read calls.

We should find a way to add transient failure handling to these individual Read operations as well to avoid letting failures unnecessarily propagate up to the caller.