Scheduled jobs are not processing

alimosabebishift commented 4 years ago

I have an issue with Hangfire , having both recurring and scheduled, Recurring jobs are okay and processing without any issue but I see none of Scheduled jobs going to processing state and I see it shows Enqueue 1 day ago, 30 mins ago,...... (apparently something is wrong with serialization but I cant find what) I had this log with older version

2020-10-13 00:01:51,074 [DelayedJobScheduler] ERROR Hangfire.Server.DelayedJobScheduler [LogException] - Error occurred during execution of 'DelayedJobScheduler' process. Execution will be retried (attempt #104) in 00:05:00 seconds. Newtonsoft.Json.JsonReaderException: After parsing a value an unexpected character was encountered: . Path 'Arguments', line 6, position 3. at Newtonsoft.Json.JsonTextReader.ParsePostValue(Boolean ignoreComments) at Newtonsoft.Json.JsonTextReader.Read() at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor1 creator, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) at Hangfire.Common.JobHelper.FromJson[T](String value) at Hangfire.SqlServer.SqlServerConnection.<>c__DisplayClass8_0.<GetJobData>b__0(DbConnection connection) at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func2 func) at Hangfire.SqlServer.SqlServerConnection.GetJobData(String id) at Hangfire.States.BackgroundJobStateChanger.GetJobData(StateChangeContext context) at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContext context) at Hangfire.Server.DelayedJobScheduler.<>c__DisplayClass10_0.b__0(IStorageConnection connection) at Hangfire.Server.DelayedJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action) at Hangfire.Server.DelayedJobScheduler.Execute(BackgroundProcessContext context) at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)

and I updated to new version to see if this fix the issue
here is my setup private IEnumerable GetHangfireServers() { GlobalConfiguration.Configuration .SetDataCompatibilityLevel(CompatibilityLevel.Version_170) .UseSimpleAssemblyNameTypeSerializer() .UseRecommendedSerializerSettings() .UseSqlServerStorage("HangFireServer", new SqlServerStorageOptions { CommandBatchMaxTimeout = TimeSpan.FromMinutes(5), SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5), QueuePollInterval = TimeSpan.Zero, UseRecommendedIsolationLevel = true, PrepareSchemaIfNecessary = true, // Default value: true EnableHeavyMigrations = true // Default value: false //DisableGlobalLocks = true }); yield return new BackgroundJobServer(); } app.UseHangfireAspNet(GetHangfireServers); app.UseHangfireDashboard("/hangfire", new DashboardOptions { Authorization = new[] { new HangFireAuthorizationFilter() } });

but still getting this in logs,

2020-10-23 00:02:34,922 [DelayedJobScheduler #1] ERROR Hangfire.Server.DelayedJobScheduler [LogException] - 5 state change attempt(s) failed due to an exception, moving job to the FailedState Newtonsoft.Json.JsonReaderException: After parsing a value an unexpected character was encountered: . Path 'Arguments', line 6, position 3. at Newtonsoft.Json.JsonTextReader.ParsePostValue(Boolean ignoreComments) at Newtonsoft.Json.JsonTextReader.Read() at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor1 creator, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Hangfire.Common.SerializationHelper.Deserialize(String value, Type type, SerializationOption option) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Hangfire.Common.SerializationHelper.Deserialize(String value, Type type, SerializationOption option) at Hangfire.Common.SerializationHelper.Deserialize[T](String value, SerializationOption option) at Hangfire.Common.SerializationHelper.Deserialize[T](String value) at Hangfire.Storage.InvocationData.DeserializePayload(String payload) at Hangfire.SqlServer.SqlServerConnection.<>c__DisplayClass8_0.<GetJobData>b__0(DbConnection connection) at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func2 func) at Hangfire.States.BackgroundJobStateChanger.GetJobData(StateChangeContext context) at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContext context) at Hangfire.Server.DelayedJobScheduler.EnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)

anyone ever had this issue? is there anyway to fix this without deleting all scheduled jobs? Thanks

odinserj commented 4 years ago

Thanks for all the details, the problem is really strange. Could you issue the following query to your SQL Server with actual job identifier to see the actual payload?

SELECT * FROM [HangFire].[Job] WHERE [Id] = <JOBID>

alimosabebishift commented 4 years ago

yeah , something like this

Scheduled job

Id 7451 StatId 14981 StateName Scheduled

InvocationData {"t":"Quorum4.Helpers.HangFireHelper, Quorum4","m":"SendReminderEmail","p":["System.String","System.Int32","System.String"]}

Arguments ["\"<!DOCTYPE html PUBLIC \\"-//W3C//DTD XHTML 1.0 Strict//EN\\" \\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\\"> \r\n<html xmlns=\\"http://www.w3.org/1999/xhtml\\\">\\r\\n\r\n\t<meta http-equiv=\\"Content-Type\\" content=\\"text/html; charset=utf-8\\" />\r\n\t<meta name=\\"viewport\\" content=\\"width=device-width, initial-scale=1.0\\"/>\r\n\tYour Message Subject or Title\r\n\t<style type=\\"text/css\\">\r\n\t\t/ Based on The MailChimp Reset INLINE: Yes. / \r\n\t\t/ Client-specific Styles /\r\n\t\t#outlook a {padding:0;} / Force Outlook to provide a \\"view in browser\\" menu link. /\r\n\t\tbody{width:100% !important; -webkit-text-size-adjust:100%; -ms-text-size-adjust:100%; margin:0; padding:0;} \r\n\t\t/ Prevent Webkit and Windows Mobile platforms from changing default font sizes./ \r\n\t\t.ExternalClass {width:100%;} / Force Hotmail to display emails at full width / \r\n\t\t.ExternalClass, .ExternalClass p, .ExternalClass span, .ExternalClass font, .ExternalClass td, .ExternalClass div {line-height: 100%;}\r\n\t\t/ Forces Hotmail to display normal line spacing. More on that: http://www.emailonacid.com/forum/viewthread/43/ / \r\n\t\t#backgroundTable {margin:0; padding:0; width:100% !important; line-height: 100% !important;}\r\n\t\t/ End reset /\r\n\r\n\t\t/ Some sensible defaults for images\r\n\t\tBring inline: Yes. /\r\n\t\timg {outline:none; text-decoration:none; -ms-interpolation-mode: bicubic;} \r\n\t\ta img {border:none;} \r\n\t\t.image_fix {display:block;}\r\n\r\n\t\t/ Yahoo paragraph fix\r\n\t\tBring inline: Yes. /\r\n\t\tp {margin: 1em 0;}\r\n\r\n\t\t/ Hotmail header color reset\r\n\t\tBring inline: Yes. /\r\n\t\th1, h2, h3, h4, h5, h6 {color: black !important;}\r\n\r\n\t\th1 a, h2 a, h3 a, h4 a, h5 a, h6 a {color: blue !important;}\r\n\r\n\t\th1 a:active, h2 a:active, h3 a:active, h4 a:active, h5 a:active, h6 a:active {\r\n\t\tcolor: red !important; / Preferably not the same color as the normal header link color. There is limited support for psuedo classes in email clients, this was added just for good measure. /\r\n\t\t}\r\n\r\n\t\th1 a:visited, h2 a:visited, h3 a:visited, h4 a:visited, h5 a:visited, h6 a:visited {\r\n\t\tcolor: purple !important; / Preferably not the same color as the normal header link color. There is limited support for psuedo classes in email clients, this was added just for good measure. /\r\n\t\t}\r\n\r\n\t\t/ Outlook 07, 10 Padding issue fix\r\n\t\tBring inline: No./\r\n\t\ttable td {border-collapse: collapse;}\r\n\r\n\t\t/ Remove spacing around Outlook 07, 10 tables\r\n\t\tBring inline: Yes /\r\n\t\ttable { border-collapse:collapse; mso-table-lspace:0pt; mso-table-rspace:0pt; }\r\n\r\n\t\t/ Styling your links has become much simpler with the new Yahoo. In fact, it falls in line with the main credo of styling in email and make sure to bring your styles inline. Your link colors will be uniform across clients when brought inline.\r\n\t\tBring inline: Yes. /\r\n\t\ta {color: #f15b29;}\r\n\r\n\r\n\t\t/*\r\n\t\t****\r\n\t\tMOBILE TARGETING\r\n\t\t****\r\n\t\t*/\r\n\t\t@media only screen and (max-device-width: 480px) {\r\n\t\t\t/ Part one of controlling phone number linking for mobile. /\r\n\t\t\ta[href^=\\"tel\\"], a[href^=\\"sms\\"] {\r\n\t\t\t\t\t\ttext-decoration: none;\r\n\t\t\t\t\t\tcolor: #f15b29; / or whatever your want /\r\n\t\t\t\t\t\tpointer-events: none;\r\n\t\t\t\t\t\tcursor: default;\r\n\t\t\t\t\t}\r\n\r\n\t\t\t.mobile_link a[href^=\\"tel\\"], .mobile_link a[href^=\\"sms\\"] {\r\n\t\t\t\t\t\ttext-decoration: default;\r\n\t\t\t\t\t\tcolor: orange !important;\r\n\t\t\t\t\t\tpointer-events: auto;\r\n\t\t\t\t\t\tcursor: default;\r\n\t\t\t\t\t}\r\n\t\t}\r\n\r\n\t\t/ More Specific Targeting /\r\n\r\n\t\t@media only screen and (min-device-width: 768px) and (max-device-width: 1024px) {\r\n\t\t/ You guessed it, ipad (tablets, smaller screens, etc) /\r\n\t\t\t/ repeating for the ipad /\r\n\t\t\ta[href^=\\"tel\\"], a[href^=\\"sms\\"] {\r\n\t\t\t\t\t\ttext-decoration: none;\r\n\t\t\t\t\t\tcolor: #f15b29; / or whatever your want /\r\n\t\t\t\t\t\tpointer-events: none;\r\n\t\t\t\t\t\tcursor: default;\r\n\t\t\t\t\t}\r\n\r\n\t\t\t.mobile_link a[href^=\\"tel\\"], .mobile_link a[href^=\\"sms\\"] {\r\n\t\t\t\t\t\ttext-decoration: default;\r\n\t\t\t\t\t\tcolor: orange !important;\r\n\t\t\t\t\t\tpointer-events: auto;\r\n\t\t\t\t\t\tcursor: default;\r\n\t\t\t\t\t}\r\n\t\t}\r\n\t\t\r\n\t\t@media only screen and (-webkit-min-device-pixel-ratio: 2) {\r\n\t\t/ Put your iPhone 4g styles in here /\r\n\t\t}\r\n\t\t\r\n\t\t/ Android targeting /\r\n\t\t@media only screen and (-webkit-device-pixel-ratio:.75){\r\n\t\t/ Put CSS for low density (ldpi) Android layouts in here /\r\n\t\t}\r\n\t\t@media only screen and (-webkit-device-pixel-ratio:1){\r\n\t\t/ Put CSS for medium density (mdpi) Android layouts in here /\r\n\t\t}\r\n\t\t@media only screen and (-webkit-device-pixel-ratio:1.5){\r\n\t\t/ Put CSS for high density (hdpi) Android layouts in here /\r\n\t\t}\r\n\t\t/ end Android targeting /\r\n\r\n\t\r\n\r\n\t\r\n\t \r\n\r\n\t\r\n\r\n\t\r\n\r\n\r\n\r\n<table cellpadding=\\"0\\" cellspacing=\\"0\\" border=\\"0\\" id=\\"backgroundTable\\">\r\n\t\r\n\t\t<td valign=\\"top\\"> \r\n\t\t\r\n\t\t<table cellpadding=\\"0\\" cellspacing=\\"0\\" border=\\"0\\" align=\\"center\\" style=\\"border-top:1px solid #BFBFBF;border-left:1px solid #BFBFBF;border-right:1px solid #BFBFBF;\\">\r\n\t\t\t\r\n\t\t\t\t<td width=\\"800\\" valign=\\"top\\" class=\\"col1\\" style=\\"height:100px;background-color:#fff;text-align:center;\\">\r\n\t\t\t\t\t
\r\n\t\t\t\t\t<a href=\\"[Corporation URL]\\" style=\\"border:none;\\">\r\n\t\t\t\t\t\t\r\n\t\t\t\t\t
\r\n\t\t\t\t\t<table cellpadding=\\"0\\" cellspacing=\\"0\\" border=\\"0\\" style=\\"margin:15px 15px;\\">\r\n\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t<td width=\\"800\\" align=\\"left\\" style=\\"padding:10px 10px;text-align:left;font-family:arial;font-size:12px;\\">\r\n\t\t\t\t\t\t\t\t<span style=\\"font-size:14px;font-weight: bold;\\">Our records show that we have not received your Online Proxy form titled WSCC175 2020 AGM.

\r\n\t\t\t\t\t\t\t\tYou have been identified as an owner of unit [Owner Unit Number] in the condominium WSCC175 - Mountford Condominiums.

\r\n\t\t\t\t\t\t\t\tMF Property Management Ltd., on behalf of WSCC175 - Mountford Condominiums, has created an easy way for you to provide your input on an important matter in your condo and to complete the regulated Proxy form.

\r\n\t\t\t\t\t\t\t\tOnline Proxies may not be tabulated by your property manager if submitted too late

\r\n\t\t\t\t\t\t\t\tPlease indicate if you will be attending the meeting in person/virtually or want to complete the proxy online.Details of the meeting can be found <a href=\\"[Link URL With Key]\\">here.

\r\n\t\t\t\t\t\t\t\t<table cellspacing=\\"0\\" cellpadding=\\"0\\"> \r\n\t\t\t\t\t\t\t\t <td align=\\"center\\" width=\\"300\\" height=\\"40\\" bgcolor=\\"#f15b29\\" style=\\"-webkit-border-radius: 5px; -moz-border-radius: 5px; border-radius: 5px; color: #ffffff; display: block;\\">\r\n\t\t\t\t\t\t\t\t\t<a href=\\"[Link URL With Key]\\" style=\\"font-size:16px; font-weight: bold; font-family:sans-serif; text-decoration: none; line-height:40px; width:100%; display:inline-block\\">\r\n\t\t\t\t\t\t\t\t\t<span style=\\"color: #ffffff;\\">\r\n\t\t\t\t\t\t\t\t\t Complete my online proxy now\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t \r\n\t\t\t\t\t\t\t\t
\r\n\t\t\t\t\t\t\t\tAlternatively go to https://proxy.shiftsuite.com and enter your unique proxy key : [Unique Key]

\r\n\t\t\t\t\t\t\t\t<table cellspacing=\\"0\\" cellpadding=\\"0\\"> \r\n\t\t\t\t\t\t\t\t <td align=\\"center\\" width=\\"300\\" height=\\"40\\" bgcolor=\\"#f15b29\\" style=\\"-webkit-border-radius: 5px; -moz-border-radius: 5px; border-radius: 5px; color: #ffffff; display: block;\\">\r\n\t\t\t\t\t\t\t\t\t<a href=\\"[Link URL To Attend In Person]\\" style=\\"font-size:16px; font-weight: bold; font-family:sans-serif; text-decoration: none; line-height:40px; width:100%; display:inline-block\\">\r\n\t\t\t\t\t\t\t\t\t<span style=\\"color: #ffffff;\\">\r\n\t\t\t\t\t\t\t\t\t I will attend in person/virtually\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t \r\n\t\t\t\t\t\t\t\t
\r\n\t\t\t\t\t\t\t\tFor information specific to your condo and the content within the proxy forms, please contact your property manager.

\r\n\t\t\t\t\t\t\t\tThank You,

\r\n\t\t\t\t\t\t\t\tThe Shiftsuite Team On Behalf of WSCC175 - Mountford Condominiums and MF Property Management Ltd.
\r\n\t\t\t\t\t\t\t\t
\r\n\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\r\n\t\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\r\n\t\t\t\r\n\t\t\t\t<td width=\\"800\\" valign=\\"top\\" class=\\"col1\\" style=\\"height:50px;background-color:#f7f7f7;text-align:center;font-family:arial;font-size:10px;padding-top:8px;padding-bottom:8px;\\">\r\n\t\t\t\t\t<a href=\\"http://www.shiftsuite.com/\\" style=\\"border:none;\\">\r\n\t\t\t\t\t\t<img src=\\"https://proxystaging.shiftsuite.com/Content/images/ShiftSuiteLogo.png\\" alt=\\"SHIFTSUITE\\" title=\\"SHIFTSUITE\\" />

\r\n\t\t\t\t\r\n\t\t\t\r\n\t\t\r\n\t\t\r\n\t\t\r\n\t\r\n \r\n\r\n\r\n\"","1415","\"Mountford Condominiums\""]

CreatedAt 2020-10-20 22:27:38.070 ExpireAt Null

Recurring Job

Id StateId StateName InvocationData
7442 14961 Succeeded {"t":"Quorum4.Helpers.HangFireHelper, Quorum4","m":"DailyUpdateDB","p":["System.String"]}
Arguments CreatedAt ExpireAt ["\"DailyUpdateDB\""] 2020-10-23 06:00:11.627 2020-10-24 06:02:04.437

alimosabebishift commented 4 years ago

Thanks for all the details, the problem is really strange. Could you issue the following query to your SQL Server with actual job identifier to see the actual payload?
SELECT * FROM [HangFire].[Job] WHERE [Id] = <JOBID>

Just to add , I can manually Enqueue any scheduled job and there is no issue

Plasma commented 2 years ago

I managed to encounter a Scheduled Job stuck in a Scheduled state in a development environment, digging into the source it was because the next JobId in the set table where key = schedule due to run no longer existed in the job table.

This meant that this line https://github.com/HangfireIO/Hangfire/blob/master/src/Hangfire.Core/States/BackgroundJobStateChanger.cs#L162 returned NULL because BackgroundJobId (from the set table) no longer existed, and it retried forever (as per the comment https://github.com/HangfireIO/Hangfire/blob/master/src/Hangfire.Core/States/BackgroundJobStateChanger.cs#L141-L146 ) and so never attempts to enqueue other jobs that do exist and are due to be enqueued.

This was using the Postgres storage system, but as a development environment, there may have been times of a hard-stop of the jobs/code/etc and resulted in a mismatch between the job and set tables that caused it to enter this state.

odinserj commented 2 years ago

@Plasma, I've rewritten that piece of code to avoid infinite loops entirely (change is available in 1.7.28), but please note that DelayedJobScheduler passes CancellationToken.None to this method, and in this case control is returned immediately when job does not exist without looping infinitely, so looks like there was another reason for your issue, and it's likely it's related to the Hangfire.Postgresql implementation you are using.

https://github.com/HangfireIO/Hangfire/blob/master/src/Hangfire.Core/States/BackgroundJobStateChanger.cs#L194-L198

HangfireIO / Hangfire

Scheduled jobs are not processing #1754