search

SparkDevNetwork / Rock

An open source CMS, Relationship Management System (RMS) and Church Management System (ChMS) all rolled into one.
http://www.rockrms.com
574 stars 346 forks source link

[Alpha 1.16.3.1] Database Maintenance Job Times Out and Fails #5755

Closed mikedotmundy closed 7 months ago

mikedotmundy commented 7 months ago

Description

Our Database Maintenance Job has timed out every morning since updating to v16.3 Alpha. It is scheduled to run at 2:30AM EST, and it has been failing each time after running for about 17-19 minutes.

This job is running successfully in our production instance on v16.2, and takes anywhere from 8-15 min to complete.

Exception from when job runs automatically, at scheduled time:

Screenshot 2024-02-14 at 20 50 40

Exception Description:

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
UPDATING STATISTICS [__MigrationHistory]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoAllocation]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoBracket]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoBracketType]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoRequest]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoTier]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoType]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_LocationLayout]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_Question]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_Reservation]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationApprovalGroup]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationLinkage]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationLocation]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationLocationType]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationMinistry]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationResource]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationType]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationWorkflow]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationWorkflowTrigger]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_Resource]
UPDATING STATISTICS [_com_blueboxmoon_DataToolkit_SqlFilter]
UPDATING STATISTICS [_com_blueboxmoon_DataToolkit_SqlFilterResult]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Attachment]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Project]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectAssignee]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBlocker]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBoard]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBoardCard]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBoardColumn]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectType]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectTypeAssociation]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Task]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_TaskBlocker]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Watching]
UPDATING STATISTICS [_com_centralaz_RoomManagement_LocationLayout]
UPDATING STATISTICS [_com_centralaz_RoomManagement_Question]
UPDATING STATISTICS [_com_centralaz_RoomManagement_Reservation]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationLocation]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationMinistry]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationResource]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationType]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationWorkflow]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationWorkflowTrigger]
UPDATING STATISTICS [_com_centralaz_RoomManagement_Resource]
UPDATING STATISTICS [_com_intulse_PbxComponent_CallDetailRecords]
UPDATING STATISTICS [_com_intulse_PbxComponent_CommunicationNotes]
UPDATING STATISTICS [_com_intulse_PbxComponent_ConnectedExtensions]
UPDATING STATISTICS [_com_intulse_PbxComponent_Settings]
UPDATING STATISTICS [_com_intulse_PbxComponent_SmsMessages]
UPDATING STATISTICS [_org_mywell_MyWellGateway_ApplePayDomain]
UPDATING STATISTICS [_org_mywell_MyWellGateway_Schedule]
UPDATING STATISTICS [_org_mywell_MyWellGateway_ScheduleAccountAllocation]
UPDATING STATISTICS [_org_mywell_MyWellGateway_ScheduleImport]
UPDATING STATISTICS [_rocks_pillars_PCOSync_Account]
UPDATING STATISTICS [_rocks_pillars_PCOSync_AccountGroup]
UPDATING STATISTICS [_rocks_pillars_PCOSync_AccountPerson]
UPDATING STATISTICS [_tmp_Users]
UPDATING STATISTICS [AchievementAttempt]
UPDATING STATISTICS [AchievementType]
UPDATING STATISTICS [AchievementTypePrerequisite]
UPDATING STATISTICS [AdaptiveMessage]
UPDATING STATISTICS [AdaptiveMessageAdaptation]
UPDATING STATISTICS [AdaptiveMessageAdaptationSegment]
UPDATING STATISTICS [AdaptiveMessageCategory]
UPDATING STATISTICS [AnalyticsSourceAttendance]
UPDATING STATISTICS [AnalyticsSourceCampus]
UPDATING STATISTICS [AnalyticsSourceDate]
UPDATING STATISTICS [AnalyticsSourceFamilyHistorical]
UPDATING STATISTICS [AnalyticsSourceFinancialTransaction]
UPDATING STATISTICS [AnalyticsSourceGivingUnit]
UPDATING STATISTICS [AnalyticsSourcePersonHistorical]
UPDATING STATISTICS [AnalyticsSourceZipCode]
UPDATING STATISTICS [Assessment]
UPDATING STATISTICS [AssessmentType]
UPDATING STATISTICS [AssetStorageProvider]
UPDATING STATISTICS [Attendance]
UPDATING STATISTICS [AttendanceCheckInSession]
UPDATING STATISTICS [AttendanceCode]
UPDATING STATISTICS [AttendanceData]
UPDATING STATISTICS [AttendanceOccurrence]
UPDATING STATISTICS [Attribute]
UPDATING STATISTICS [AttributeCategory]
UPDATING STATISTICS [AttributeMatrix]
UPDATING STATISTICS [AttributeMatrixItem]
UPDATING STATISTICS [AttributeMatrixTemplate]
UPDATING STATISTICS [AttributeQualifier]
UPDATING STATISTICS [AttributeReferencedEntity]
UPDATING STATISTICS [AttributeValue]
UPDATING STATISTICS [AttributeValueHistorical]
UPDATING STATISTICS [AttributeValueReferencedEntity]
UPDATING STATISTICS [Audit]
UPDATING STATISTICS [AuditDetail]
UPDATING STATISTICS [Auth]
UPDATING STATISTICS [AuthAuditLog]
UPDATING STATISTICS [AuthClaim]
UPDATING STATISTICS [AuthClient]
UPDATING STATISTICS [AuthScope]
UPDATING STATISTICS [BackgroundCheck]
UPDATING STATISTICS [Badge]
UPDATING STATISTICS [BenevolenceRequest]
UPDATING STATISTICS [BenevolenceRequestDocument]
UPDATING STATISTICS [BenevolenceResult]
UPDATING STATISTICS [BenevolenceType]
UPDATING STATISTICS [BenevolenceWorkflow]
UPDATING STATISTICS [BinaryFile]
UPDATING STATISTICS [BinaryFileData]
UPDATING STATISTICS [BinaryFileType]
UPDATING STATISTICS [Block]
UPDATING STATISTICS [BlockType]
UPDATING STATISTICS [Campus]
UPDATING STATISTICS [CampusSchedule]
UPDATING STATISTICS [CampusTopic]
UPDATING STATISTICS [Category]
UPDATING STATISTICS [Communication]
UPDATING STATISTICS [CommunicationAttachment]
UPDATING STATISTICS [CommunicationRecipient]
UPDATING STATISTICS [CommunicationResponse]
UPDATING STATISTICS [CommunicationResponseAttachment]
UPDATING STATISTICS [CommunicationTemplate]
UPDATING STATISTICS [CommunicationTemplateAttachment]
UPDATING STATISTICS [ConnectionActivityType]
UPDATING STATISTICS [ConnectionOpportunity]
UPDATING STATISTICS [ConnectionOpportunityCampus]
UPDATING STATISTICS [ConnectionOpportunityConnectorGroup]
UPDATING STATISTICS [ConnectionOpportunityGroup]
UPDATING STATISTICS [ConnectionOpportunityGroupConfig]
UPDATING STATISTICS [ConnectionRequest]
UPDATING STATISTICS [ConnectionRequestActivity]
UPDATING STATISTICS [ConnectionRequestWorkflow]
UPDATING STATISTICS [ConnectionStatus]
UPDATING STATISTICS [ConnectionStatusAutomation]
UPDATING STATISTICS [ConnectionType]
UPDATING STATISTICS [ConnectionWorkflow]
UPDATING STATISTICS [ContentChannel]
UPDATING STATISTICS [ContentChannelAssociation]
UPDATING STATISTICS [ContentChannelCategory]
UPDATING STATISTICS [ContentChannelItem]
UPDATING STATISTICS [ContentChannelItemAssociation]
UPDATING STATISTICS [ContentChannelItemSlug]
UPDATING STATISTICS [ContentChannelType]
UPDATING STATISTICS [ContentCollection]
UPDATING STATISTICS [ContentCollectionSource]
UPDATING STATISTICS [ContentTopic]
UPDATING STATISTICS [ContentTopicDomain]
UPDATING STATISTICS [DataView]
UPDATING STATISTICS [DataViewFilter]
UPDATING STATISTICS [DataViewPersistedValue]
UPDATING STATISTICS [DefinedType]
UPDATING STATISTICS [DefinedValue]
UPDATING STATISTICS [Device]
UPDATING STATISTICS [DeviceLocation]
UPDATING STATISTICS [Document]
UPDATING STATISTICS [DocumentType]
UPDATING STATISTICS [EntityCampusFilter]
UPDATING STATISTICS [EntitySet]
UPDATING STATISTICS [EntitySetItem]
UPDATING STATISTICS [EntityType]
UPDATING STATISTICS [EventCalendar]
UPDATING STATISTICS [EventCalendarContentChannel]
UPDATING STATISTICS [EventCalendarItem]
UPDATING STATISTICS [EventItem]
UPDATING STATISTICS [EventItemAudience]
UPDATING STATISTICS [EventItemOccurrence]
UPDATING STATISTICS [EventItemOccurrenceChannelItem]
UPDATING STATISTICS [EventItemOccurrenceGroupMap]
UPDATING STATISTICS [ExceptionLog]
UPDATING STATISTICS [FieldType]
UPDATING STATISTICS [FinancialAccount]
UPDATING STATISTICS [FinancialBatch]
UPDATING STATISTICS [FinancialGateway]
UPDATING STATISTICS [FinancialPaymentDetail]
UPDATING STATISTICS [FinancialPersonBankAccount]
UPDATING STATISTICS [FinancialPersonSavedAccount]
UPDATING STATISTICS [FinancialPledge]
UPDATING STATISTICS [FinancialScheduledTransaction]
UPDATING STATISTICS [FinancialScheduledTransactionDetail]
UPDATING STATISTICS [FinancialStatementTemplate]
UPDATING STATISTICS [FinancialTransaction][: Line 0, : Line 0, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20]

Stack Trace:

 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Rock.Data.DbService.ExecuteCommand(String connectionString, String query, CommandType commandType, Dictionary`2 parameters, Nullable`1 commandTimeout) in \Rock\Data\DbService.cs:line 312
   at Rock.Jobs.DatabaseMaintenance.UpdateStatistics(Int32 commandTimeout) in \Rock\Jobs\DatabaseMaintenance.cs:line 500
   at Rock.Jobs.DatabaseMaintenance.Execute() in \Rock\Jobs\DatabaseMaintenance.cs:line 241
   at Rock.Jobs.RockJob.ExecuteInternal(IJobExecutionContext context) in \Rock\Jobs\RockJob.cs:line 162
   at Quartz.Core.JobRunShell.Run()

Exception when job is run manually:

Exception Description:

A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
UPDATING STATISTICS [__MigrationHistory]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoAllocation]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoBracket]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoBracketType]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoRequest]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoTier]
UPDATING STATISTICS [_com_bemaservices_HrManagement_PtoType]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_LocationLayout]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_Question]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_Reservation]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationApprovalGroup]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationLinkage]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationLocation]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationLocationType]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationMinistry]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationResource]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationType]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationWorkflow]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_ReservationWorkflowTrigger]
UPDATING STATISTICS [_com_bemaservices_RoomManagement_Resource]
UPDATING STATISTICS [_com_blueboxmoon_DataToolkit_SqlFilter]
UPDATING STATISTICS [_com_blueboxmoon_DataToolkit_SqlFilterResult]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Attachment]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Project]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectAssignee]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBlocker]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBoard]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBoardCard]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectBoardColumn]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectType]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_ProjectTypeAssociation]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Task]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_TaskBlocker]
UPDATING STATISTICS [_com_blueboxmoon_ProjectManagement_Watching]
UPDATING STATISTICS [_com_centralaz_RoomManagement_LocationLayout]
UPDATING STATISTICS [_com_centralaz_RoomManagement_Question]
UPDATING STATISTICS [_com_centralaz_RoomManagement_Reservation]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationLocation]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationMinistry]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationResource]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationType]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationWorkflow]
UPDATING STATISTICS [_com_centralaz_RoomManagement_ReservationWorkflowTrigger]
UPDATING STATISTICS [_com_centralaz_RoomManagement_Resource]
UPDATING STATISTICS [_com_intulse_PbxComponent_CallDetailRecords]
UPDATING STATISTICS [_com_intulse_PbxComponent_CommunicationNotes]
UPDATING STATISTICS [_com_intulse_PbxComponent_ConnectedExtensions]
UPDATING STATISTICS [_com_intulse_PbxComponent_Settings]
UPDATING STATISTICS [_com_intulse_PbxComponent_SmsMessages]
UPDATING STATISTICS [_org_mywell_MyWellGateway_ApplePayDomain]
UPDATING STATISTICS [_org_mywell_MyWellGateway_Schedule]
UPDATING STATISTICS [_org_mywell_MyWellGateway_ScheduleAccountAllocation]
UPDATING STATISTICS [_org_mywell_MyWellGateway_ScheduleImport]
UPDATING STATISTICS [_rocks_pillars_PCOSync_Account]
UPDATING STATISTICS [_rocks_pillars_PCOSync_AccountGroup]
UPDATING STATISTICS [_rocks_pillars_PCOSync_AccountPerson]
UPDATING STATISTICS [_tmp_Users]
UPDATING STATISTICS [AchievementAttempt]
UPDATING STATISTICS [AchievementType]
UPDATING STATISTICS [AchievementTypePrerequisite]
UPDATING STATISTICS [AdaptiveMessage]
UPDATING STATISTICS [AdaptiveMessageAdaptation]
UPDATING STATISTICS [AdaptiveMessageAdaptationSegment]
UPDATING STATISTICS [AdaptiveMessageCategory]
UPDATING STATISTICS [AnalyticsSourceAttendance]
UPDATING STATISTICS [AnalyticsSourceCampus]
UPDATING STATISTICS [AnalyticsSourceDate]
UPDATING STATISTICS [AnalyticsSourceFamilyHistorical]
UPDATING STATISTICS [AnalyticsSourceFinancialTransaction]
UPDATING STATISTICS [AnalyticsSourceGivingUnit]
UPDATING STATISTICS [AnalyticsSourcePersonHistorical]
UPDATING STATISTICS [AnalyticsSourceZipCode]
UPDATING STATISTICS [Assessment]
UPDATING STATISTICS [AssessmentType]
UPDATING STATISTICS [AssetStorageProvider]
UPDATING STATISTICS [Attendance]
UPDATING STATISTICS [AttendanceCheckInSession]
UPDATING STATISTICS [AttendanceCode]
UPDATING STATISTICS [AttendanceData]
UPDATING STATISTICS [AttendanceOccurrence]
UPDATING STATISTICS [Attribute]
UPDATING STATISTICS [AttributeCategory]
UPDATING STATISTICS [AttributeMatrix]
UPDATING STATISTICS [AttributeMatrixItem]
UPDATING STATISTICS [AttributeMatrixTemplate]
UPDATING STATISTICS [AttributeQualifier]
UPDATING STATISTICS [AttributeReferencedEntity]
UPDATING STATISTICS [AttributeValue]
UPDATING STATISTICS [AttributeValueHistorical]
UPDATING STATISTICS [AttributeValueReferencedEntity]
UPDATING STATISTICS [Audit]
UPDATING STATISTICS [AuditDetail]
UPDATING STATISTICS [Auth]
UPDATING STATISTICS [AuthAuditLog]
UPDATING STATISTICS [AuthClaim]
UPDATING STATISTICS [AuthClient]
UPDATING STATISTICS [AuthScope]
UPDATING STATISTICS [BackgroundCheck]
UPDATING STATISTICS [Badge]
UPDATING STATISTICS [BenevolenceRequest]
UPDATING STATISTICS [BenevolenceRequestDocument]
UPDATING STATISTICS [BenevolenceResult]
UPDATING STATISTICS [BenevolenceType]
UPDATING STATISTICS [BenevolenceWorkflow][: Line 0, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20, : Line 20]

Stack Trace:

at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[] buff, Int32 offset, Int32 len, Int32& totalRead)
   at System.Data.SqlClient.TdsParserStateObject.TryReadInt64(Int64& value)
   at System.Data.SqlClient.TdsParser.TryProcessDone(SqlCommand cmd, SqlDataReader reader, RunBehavior& run, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Rock.Data.DbService.ExecuteCommand(String connectionString, String query, CommandType commandType, Dictionary`2 parameters, Nullable`1 commandTimeout) in \Rock\Data\DbService.cs:line 312
   at Rock.Jobs.DatabaseMaintenance.UpdateStatistics(Int32 commandTimeout) in \Rock\Jobs\DatabaseMaintenance.cs:line 500
   at Rock.Jobs.DatabaseMaintenance.Execute() in \Rock\Jobs\DatabaseMaintenance.cs:line 241
   at Rock.Jobs.RockJob.ExecuteInternal(IJobExecutionContext context) in \Rock\Jobs\RockJob.cs:line 162
   at Quartz.Core.JobRunShell.Run()

Actual Behavior

Database Maintenance Job Times Out and Fails.

Expected Behavior

Database Maintenance Job should run successfully.

Steps to Reproduce

Issue Confirmation

Rock Version

v16.3

Client Culture Setting

en-US

nairdo commented 7 months ago

Thanks for this report @mikedotmundy . Can you share your job's settings (namely the timeout and other advanced settings) or a screenshot: image

mikedotmundy commented 7 months ago
Screenshot 2024-02-15 at 10 30 55
azturner commented 7 months ago

We're working with church experiencing this same issue but they are still on version 14.3. They are currently running DB in an elastic pool. Our plan is to move their DB out of the elastic pool, increase the timeout setting and temporarily increase their DTU's to see if we can get the job to complete successfully.

image

jasonhendee commented 7 months ago

We've determined this was an issue with an under-provisioned test environment compared to the production environment, so we're going to go ahead and close this issue.

Thank you, @mikedotmundy, for being so quick to test locally and verify this morning. And thank you, @azturner, for confirming this was an issue before Rock v16.3, most likely related to server provisioning in that case as well.

Another way to "fix" this would be to increase the Command Timeout value for this job, although that could be a bit of a moving target. It really just seems to boil down to a larger and more complex database needing either more time to complete this maintenance work or more server resources to get the work done more quickly.