pieceofsummer / Hangfire.Console

Job console extension for Hangfire
MIT License
430 stars 79 forks source link

Occasional exception in production when progressbar is disposed: Invalid operation. The connection is closed. #126

Open sommmen opened 9 months ago

sommmen commented 9 months ago

I have multiple jobs running and occasionally i run into the following issue:

Invalid operation. The connection is closed.
   at Microsoft.Data.SqlClient.SqlConnection.GetOpenTdsConnection()
   at Microsoft.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
   at Microsoft.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
   at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass40_0`1.<UseTransaction>b__0(DbConnection connection)
   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)
   at Hangfire.SqlServer.SqlServerStorage.UseTransaction[T](DbConnection dedicatedConnection, Func`3 func, Nullable`1 isolationLevel)
   at Hangfire.SqlServer.SqlServerStorage.UseTransaction(DbConnection dedicatedConnection, Action`2 action)
   at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit()
   at Hangfire.Console.Storage.ConsoleStorage.AddLine(ConsoleId consoleId, ConsoleLine line)
   at Hangfire.Console.Server.ConsoleContext.AddLine(ConsoleLine line)
   at Hangfire.Console.Progress.DefaultProgressBar.SetValue(Double value)
   at Hangfire.Console.Progress.DefaultProgressBar.SetValue(Int32 value)
   at Hangfire.Console.Progress.ProgressEnumerable`1.Enumerator.Dispose()
   ....

Happens on some progressbars:

foreach (var x in current.WithProgress(Logger.WriteProgressBar("Batch")))
{
  ...

Seems to be an issue where somehow the connection gets closed too early? Any help would be awesome.

Packages:

        <PackageReference Include="Hangfire.AspNetCore" Version="1.8.6" />
        <PackageReference Include="Hangfire.Console" Version="1.4.2" />
        <PackageReference Include="Hangfire.SqlServer" Version="1.8.6" />

I am also now running the identitystream fork to see if it helps, but it does not for this issue. <PackageReference Include="IdentityStream.Hangfire.Console" Version="2.0.0" />

bau-wd commented 4 months ago

same issue here... it seems that the exception gets thrown when the hangfire job itself finishes before the latest progressBar value gets written ...

EDIT: This is my exception:

System.InvalidOperationException: ExecuteNonQuery requires an open and available Connection. The connection's current state is open.
 at Microsoft.Data.SqlClient.SqlCommand.ValidateCommand(Boolean isAsync, String method)
   at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Hangfire.SqlServer.SqlCommandBatch.ExecuteNonQuery()
   at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<Commit>b__17_0(DbConnection connection, DbTransaction transaction)
   at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass42_0`1.<UseTransaction>b__0(DbConnection connection)
   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)
   at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit()
   at Hangfire.Console.Storage.ConsoleStorage.AddLine(ConsoleId consoleId, ConsoleLine line)
   at Hangfire.Console.Server.ConsoleContext.AddLine(ConsoleLine line)
   at Hangfire.Console.Progress.DefaultProgressBar.SetValue(Double value)