pieceofsummer / Hangfire.Console

Job console extension for Hangfire
MIT License
436 stars 80 forks source link

Jobs are failing due to System.Data.SqlClient.SqlException Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. #91

Open iSatishYadav opened 4 years ago

iSatishYadav commented 4 years ago

It’s a wonderful extension and I totally love it. However, recently under load, getting a lot of time out exceptions due to this.

Stack trace:



System.Data.SqlClient.SqlException

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Hangfire.SqlServer.SqlCommandBatch.ExecuteNonQuery()
   at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.<Commit>b__7_0(DbConnection connection, DbTransaction transaction)
   at Hangfire.SqlServer.SqlServerStorage.<>c__DisplayClass26_0.<UseTransaction>b__0(DbConnection connection, DbTransaction transaction)
   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)
   at Hangfire.SqlServer.SqlServerStorage.UseTransaction[T](DbConnection dedicatedConnection, Func`3 func, Nullable`1 isolationLevel)
   at Hangfire.SqlServer.SqlServerStorage.UseTransaction(DbConnection dedicatedConnection, Action`2 action)
   at Hangfire.SqlServer.SqlServerWriteOnlyTransaction.Commit()
   at Hangfire.Console.Storage.ConsoleStorage.InitConsole(ConsoleId consoleId)
   at Hangfire.Console.Server.ConsoleServerFilter.OnPerforming(PerformingContext filterContext)
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)````
pieceofsummer commented 4 years ago

Could it be you’re using some filters that block SQL connections during execution (e.g. DisableConcurrentExecution) so your connection pool is exhausted? Also ensure your SQL server has enough resources.

Saad-Alothman commented 4 years ago

so if a job is using DisableConcurrentExecution, it will cause issues ?

Thank you for sharing the plugin with the rest of world!

iSatishYadav commented 4 years ago

Could it be you’re using some filters that block SQL connections during execution (e.g. DisableConcurrentExecution) so your connection pool is exhausted? Also ensure your SQL server has enough resources.

No, I’m not using any filters.

luizfbicalho commented 3 years ago

I have this same problem, looks like if the job changed states many times, then it's more likely to happen this error