search

jobbrIO / jobbr-server

Jobbr is a non-invasive .NET JobServer
https://jobbr.readthedocs.io
GNU General Public License v3.0
50 stars 5 forks source link

Exception thrown in the Storage component can take down Jobbr #105

Closed eXpl0it3r closed 1 year ago

eXpl0it3r commented 1 year ago

Steps to Reproduce

  1. Start Jobbr with the MSSQL storage component
  2. Shutdown MSSQL
  3. At the same time, have Jobbr fetch JobRuns

Actual Behavior

JobRuns fetching fails and takes down the whole of Jobbr

Expected Behavior

JobRuns fetching fails, Jobbr continues running, oven with no access to MSSQL

Additional Information

Sample stack trace:

Exception Info: System.Data.SqlClient.SqlException (0x80131904): SHUTDOWN is in progress.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at ServiceStack.OrmLite.OrmLiteResultsFilterExtensions.ConvertToList[T](IDbCommand dbCmd, String sql) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.OrmLite/src/ServiceStack.OrmLite/OrmLiteResultsFilterExtensions.cs:line 102
   at ServiceStack.OrmLite.OrmLiteReadCommandExtensions.SqlList[T](IDbCommand dbCmd, String sql, Dictionary`2 dict) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.OrmLite/src/ServiceStack.OrmLite/OrmLiteReadCommandExtensions.cs:line 545
   at ServiceStack.OrmLite.OrmLiteExecFilter.Exec[T](IDbConnection dbConn, Func`2 filter) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.OrmLite/src/ServiceStack.OrmLite/OrmLiteExecFilter.cs:line 83
   at Jobbr.Storage.MsSql.MsSqlStorageProvider.GetJobRuns(Action`1 sql, Int32 page, Int32 pageSize, String jobTypeFilter, String jobUniqueNameFilter, String query, Boolean showDeleted, String[] sort)
   at Jobbr.Server.Storage.JobbrRepository.GetJobRunsByStateRange(JobRunStates minState, JobRunStates maxState)+MoveNext()
   at Jobbr.Server.Storage.JobbrRepository.GetRunningJobs()+MoveNext()
   at Jobbr.Server.Storage.JobbrRepository.GetRunningJobs(Int64 triggerJobId, Int64 triggerId)+MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at Jobbr.Server.Scheduling.Planer.RecurringJobRunPlaner.Plan(RecurringTrigger trigger)
   at Jobbr.Server.Scheduling.DefaultScheduler.EvaluateRecurringTriggers()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
eXpl0it3r commented 1 year ago

Also happens if there's a timeout:

Exception Info: System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at ServiceStack.OrmLite.OrmLiteResultsFilterExtensions.ConvertToList[T](IDbCommand dbCmd, String sql) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.OrmLite/src/ServiceStack.OrmLite/OrmLiteResultsFilterExtensions.cs:line 102
   at ServiceStack.OrmLite.OrmLiteReadCommandExtensions.SqlList[T](IDbCommand dbCmd, String sql, Dictionary`2 dict) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.OrmLite/src/ServiceStack.OrmLite/OrmLiteReadCommandExtensions.cs:line 545
   at ServiceStack.OrmLite.OrmLiteExecFilter.Exec[T](IDbConnection dbConn, Func`2 filter) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.OrmLite/src/ServiceStack.OrmLite/OrmLiteExecFilter.cs:line 83
   at Jobbr.Storage.MsSql.MsSqlStorageProvider.GetJobRuns(Action`1 sql, Int32 page, Int32 pageSize, String jobTypeFilter, String jobUniqueNameFilter, String query, Boolean showDeleted, String[] sort)
   at Jobbr.Server.Storage.JobbrRepository.GetJobRunsByStateRange(JobRunStates minState, JobRunStates maxState)+MoveNext()
   at Jobbr.Server.Storage.JobbrRepository.GetRunningJobs()+MoveNext()
   at Jobbr.Server.Storage.JobbrRepository.GetRunningJobs(Int64 triggerJobId, Int64 triggerId)+MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at Jobbr.Server.Scheduling.Planer.RecurringJobRunPlaner.Plan(RecurringTrigger trigger)
   at Jobbr.Server.Scheduling.DefaultScheduler.EvaluateRecurringTriggers()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()