App-vNext / Polly

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner. From version 6.0.1, Polly targets .NET Standard 1.1 and 2.0+.
https://www.thepollyproject.org
BSD 3-Clause "New" or "Revised" License
13.35k stars 1.22k forks source link

Does Polly support Retry on Aysnc Dapper Query calls? #384

Closed duydle closed 6 years ago

duydle commented 6 years ago

We just recently found that wrapping retry logic around async Dapper query call caused IIS to crashed. I wanted to ask if anyone has come across this before. Below is what I'm doing:

  1. I make my method call with async await var missingListSubmissionLists = await _dataOpsRepository.GetListsWithMissingListSubmissionsAsync(effectiveDate, dataOpsManagedOnly);

  2. Inside the method call, notice this is just a pass-through to DapperExtension.QueryAsyncWithRetry method: public async Task GetListsWithMissingListSubmissionsAsync(DateTime effectiveDate, bool dataOpsManagedOnly = true) { var data = new DataOpsListsToResolveOverview() { EffectiveDate = effectiveDate
    };

        using (var conn = new SqlConnection(_connectionString))
        {
            var queryParameters = new DynamicParameters();
            queryParameters.Add("@effectiveDate", effectiveDate);
            queryParameters.Add("@dataOpsManagedOnly", dataOpsManagedOnly);
    
            var missingListSubmissionLists = await DapperExtension.QueryAsyncWithRetry<MaestroList>(conn, "DataOps_GetListsWithMissingListSubmissions",
                                                                                            CommandType.StoredProcedure,
                                                                                            queryParameters).ConfigureAwait(false);
            data.MaestroLists = missingListSubmissionLists.ToList();
            return data;
        }
    }
  3. Inside QueryAyncWithRetry method which wraps RetryPolicy.RetryForTimeout call public static async Task<IEnumerable> QueryAsyncWithRetry( IDbConnection dbConnection, string sql, CommandType commandType, DynamicParameters paramList, int maxRetries = 3, int commandTimeout = 120) { try { //Need to verify retry when method is implemented var result = await (Task<IEnumerable>)RetryPolicy.RetryForTimeout(new Func< string, object, IDbTransaction, int?, CommandType?, Task<IEnumerable>>(dbConnection.QueryAsync), maxRetries, sql, paramList, null, commandTimeout, commandType ); return result; } catch (Exception ex) { _logger.Fatal(ex, $"DapperExtenions.QueryAsyncWithRetry failed on {sql}"); throw; } }

  4. Inside RetryForTimeout method that wraps my Retry Policy. Note that this is a generic function that executes method.DynamicInvoke based on the delegate method passed in.

private static Logger _logger = LogManager.GetCurrentClassLogger(); private static string timeout = "timeout expired"; private static string deadlock = "deadlock"; public static object RetryForTimeout(Delegate method, int maxRetries = 3, params object[] args) { var policy = Policy .Handle(ex => ExceptionMiner.FullMessageChain(ex).ToLower().Contains(timeout)) .Or(ex => ExceptionMiner.FullMessageChain(ex).ToLower().Contains(deadlock)) .Retry(maxRetries, (ex, retryCount) => { _logger.Error(ex, $"Attempting retry {retryCount} of maxRetry {maxRetries} for timeout/deadlock on method {method.Method.Name}. Parameter List Info: {GetPropertyValues(args)}"); });

        try
        {
            var result = policy.Execute(() => method.DynamicInvoke(args));
            return result;
        }
        catch (Exception e)
        {
            //log the exception after maxRetries are reached or there is no retry exception being handled
            _logger.Fatal(e, $"RetryPolicy.Retry failed on method {method.Method.Name}. Parameter List Info: {GetPropertyValues(args)}");
            throw;
        }
    }
  1. The result is it causes worker process to crash: Application: w3wp.exe Framework Version: v4.0.30319 Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FFE6475425F (00007FFE645F0000) with exit code 80131506.
duydle commented 6 years ago

After posting this issue, member of my team found this article which answers my question. http://www.thepollyproject.org/2017/06/09/polly-and-synchronous-versus-asynchronous-policies/

reisenberger commented 6 years ago

Yes, async executions currently require using async variants of the policies.

Closing. Do however open another issue or re-open, if you need further support.