dotnet / SqlClient

Microsoft.Data.SqlClient provides database connectivity to SQL Server for .NET applications.
MIT License
836 stars 277 forks source link

Continuous "Login failed for user '<username>'" until AppService Instance is Cycled Out. #1622

Closed JamiePed closed 1 month ago

JamiePed commented 2 years ago

Describe the bug

We've been experiencing issues while trying to recover from temporary drops in database availability. Different instances of the application (hosted in AppService instances) will continuously throw SqlExceptions until the instance is recycled (I.e. we scale up/down or forcibly drop the misbehaving instance). Restarting the App doesn't help.

The initial exception is thrown, understandably because the database isn't available. But every attempt to connect to the DB once it becomes available fails until we move the application to a different instance.

Exception message: Login failed for user '<user>'.
Stack trace:
Microsoft.Data.SqlClient.SqlException:
   at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod (Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject (Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest (Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection (Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen (Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Dapper.DapperDb+<OpenDbConnection>d__21.MoveNext (Organisation.Dapper, Version=4.0.23.0, Culture=neutral, PublicKeyToken=null: /home/vsts/work/1/s/src/Organisation.Dapper/DapperDb.cs:199)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Dapper.DapperDb+<OpenConnectionAsync>d__20.MoveNext (Organisation.Dapper, Version=4.0.23.0, Culture=neutral, PublicKeyToken=null: /home/vsts/work/1/s/src/Organisation.Dapper/DapperDb.cs:156)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Application.Data.WebhooksDatabase+<CreateConnectionAsync>d__3.MoveNext (Organisation.Application.Data, Version=1.1.128.0, Culture=neutral, PublicKeyToken=null: /src/Organisation.Application.Data/WebhooksDatabase.cs:39)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Application.Data.Repository.WebhooksRepository+<GetInstitutionEventEnabledAsync>d__7.MoveNext (Organisation.Application.Data, Version=1.1.128.0, Culture=neutral, PublicKeyToken=null: /src/Organisation.Application.Data/Repository/WebhooksRepository.cs:111)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Application.Data.CacheRepositories.InstitutionEventEnabledRepository+<>c__DisplayClass4_0+<<GetInstitutionEventEnabledAsync>b__0>d.MoveNext (Organisation.Application.Data, Version=1.1.128.0, Culture=neutral, PublicKeyToken=null: /src/Organisation.Application.Data/CacheRepositories/InstitutionEventEnabledRepository.cs:42)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Extensions.Caching.Extensions.MemoryCacheExtensions+<>c__DisplayClass0_1`1+<<GetOrCreateLazyAsync>b__1>d.MoveNext (Organisation.Extensions.Caching, Version=1.0.9.0, Culture=neutral, PublicKeyToken=null: /_/src/Organisation.Extensions.Caching/Extensions/MemoryCacheExtensions.cs:16)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Extensions.Caching.Extensions.MemoryCacheExtensions+<GetOrCreateLazyAsync>d__0`1.MoveNext (Organisation.Extensions.Caching, Version=1.0.9.0, Culture=neutral, PublicKeyToken=null: /_/src/Organisation.Extensions.Caching/Extensions/MemoryCacheExtensions.cs:18)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Application.Data.CacheRepositories.InstitutionEventEnabledRepository+<GetInstitutionEventEnabledAsync>d__4.MoveNext (Organisation.Application.Data, Version=1.1.128.0, Culture=neutral, PublicKeyToken=null: /src/Organisation.Application.Data/CacheRepositories/InstitutionEventEnabledRepository.cs:30)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Application.Services.Implementation.WebhookValidationService+<ValidateWebhookEventAsync>d__5.MoveNext (Organisation.Application.Services, Version=1.1.128.0, Culture=neutral, PublicKeyToken=null: /src/Organisation.Application.Services/Implementation/WebhookValidationService.cs:61)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Application.Controllers.ManagementController+<SendWebhookEventAsync>d__12.MoveNext (Application.Api, Version=1.1.128.0, Culture=neutral, PublicKeyToken=null: /src/Organisation.Application/Controllers/ManagementController.cs:162)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor+TaskOfIActionResultExecutor+<Execute>d__0.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker+<<InvokeActionMethodAsync>g__Logged|12_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker+<<InvokeNextActionFilterAsync>g__Awaited|10_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeFilterPipelineAsync>g__Awaited|20_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware+<<Invoke>g__AwaitRequestTask|6_0>d.MoveNext (Microsoft.AspNetCore.Routing, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Organisation.Application.Middleware.ExceptionLoggingMiddleware+<InvokeAsync>d__3.MoveNext (Application.Api, Version=1.1.128.0, Culture=neutral, PublicKeyToken=null: /src/Organisation.Application/Middleware/ExceptionLoggingMiddleware.cs:31)

To reproduce

This is difficult to reproduce, the problem doesn't happen every time on every instance. Sometimes one of the 3 instances of the app will fall over, sometimes all 3. Sometimes other applications are affected, sometimes they won't be.

public async Task<IDbConnection> CreateConnectionAsync()
{
    var connection = new SqlConnection(_options.WebhooksDb);

    await DapperDb.OpenConnectionAsync(connection,
        statusNotificationFunc: (attempt, success, text) =>
        {
            LogConnectionNotification(nameof(CreateConnectionAsync), attempt, success, text);
        }
    );

    return connection;
}

public static async Task OpenConnectionAsync(
    IDbConnection connection
    , int maxRetries = 5
    , int retryDelayFactorMilliseconds = 1000
    , StatusNotification statusNotificationFunc = null
)
{
    await OpenDbConnection(
        connection,
        dbConnection =>
        {
            if (dbConnection is DbConnection conn)
                return conn.OpenAsync();

            return Task.CompletedTask;
        },
        maxRetries,
        retryDelayFactorMilliseconds,
        statusNotificationFunc
    );
}

internal static async Task OpenDbConnection(
    IDbConnection connection
    , Func<IDbConnection, Task> connectionOpenTask
    , int maxRetries = 5
    , int retryDelayFactor = 1000
    , StatusNotification statusNotificationFunc = null
    )
{
    if (connection == null)
    {
        const string messageText = "The DapperDb has been disposed.";

        statusNotificationFunc?.Invoke(0, false, messageText);
        throw new InvalidOperationException(messageText);
    }

    if (connection.State == ConnectionState.Closed || connection.State == ConnectionState.Broken)
    {
        using (new SemaphoreWaiter(Semaphore))
        {
            if (connection.State == ConnectionState.Broken)
            {
                connection.Close();
            }

            // Implement retries for opening connections to take into account Azure SQL transient failures.
            // This comes out of the box in EF Core but we need to implement it ourselves for dapper.
            // See https://github.com/aspnet/EntityFrameworkCore/blob/master/src/EFCore.SqlServer/Storage/Internal/SqlServerTransientExceptionDetector.cs
            for (var i = 1; i <= maxRetries; i++)
            {
                try
                {
                    await connectionOpenTask(connection);
                    statusNotificationFunc?.Invoke(i, true, "Connection Opened");
                    break;
                }
                catch (SqlException ex)
                {
                    var isTransientError = IsTransient(ex);

                    if (!isTransientError || i == maxRetries)
                    {
                        statusNotificationFunc?.Invoke(i, false, $"Permanent {nameof(SqlException)} Failure: {ex.Message}");
                        throw;
                    }

                    statusNotificationFunc?.Invoke(i, false, $"Transient {nameof(SqlException)} Failure: {ex.Message}");

                    Thread.Sleep(retryDelayFactor * i);
                }
                catch (Exception ex)
                {
                    statusNotificationFunc?.Invoke(i, false, $"Permanent {ex.GetType().Name} Failure: {ex.Message}");
                    throw;
                }
            }
        }
    }
    else
    {
        statusNotificationFunc?.Invoke(0, true, "Connection already Open");
    }
}

Scale or Patch the DB while constantly opening connections (for different requests). Exception is thrown while the database is not available Database becomes available Instance will continue to throw this exception until

Expected behavior

We get errors during downtime but the app is able to recover as soon as the DB beocmes available

Further technical details

Microsoft.Data.SqlClient version: 4.1.0 .NET target: net6.0 SQL Server version: Azure SQL Operating system: Linux Docker Container hosted on App Service

Additional context So far all I can guess is that the Connection Pool is storing an invalid connection. The method call LogConnectionNotification is only called a handful of times as shown here image

But the Exception, with the stack above is thrown thousands of times, image

So we attempt to create the connection, it errors, but ends up in the pool so whenever we create a new one we continue to pick up that same connection.

Any help anyone can give would be super useful thanks!

JRahnama commented 2 years ago

@JamiePed can you try by setting ConnectRetryCount = 0 in your connection string?

JamiePed commented 2 years ago

Hey @JRahnama we can try adding that to the connection string - how would disabling retries help?

We're struggling to replicate this issue so it'll be hard to validate whether or not this helps, it seems to be dependent on the health state of SQL Server and the state of the connection during a given request which we've not managed to force in a test environment yet.

We think this is something to do with connection pooling, and the connection not being forcibly removed from the connection pool when it gets into this state - would this option help with that?

Rashik004 commented 1 year ago

@JamiePed , did you find any workaround or resolution? we've been facing the same issue and we've the same technical details as yours (except we're running our apps as webapps) and the version of Microsoft.Data.SqlClient is 5.0.1

cheenamalhotra commented 1 year ago

We think this is something to do with connection pooling, and the connection not being forcibly removed from the connection pool when it gets into this state

Are you able to capture exception and call 'ClearPool' with SqlConnection instance? Make sure you close connection so it goes back into pool, and then call ClearPool.

But it's strange why error doesn't go away.. do try capturing traces, as issues like these are very tricky to catch.

cheenamalhotra commented 1 month ago

Closing as stale