search

dotnet / orleans

Cloud Native application framework for .NET
https://docs.microsoft.com/dotnet/orleans
MIT License
10.07k stars 2.03k forks source link

Microsoft Orleans Reminders to Not Execute #3622

Closed sbrohi15 closed 6 years ago

sbrohi15 commented 6 years ago

Hi,

I am having serious issues in which 1 or more reminders in a batch of say 20-50 reminders are not executing during their expected fire time. I am currently using Orleans version 1.0.10. This behavior occurs while the services are running and am currently using a 2 node cluster.

I have several questions:

**1. What can I do to resolve this issue?

  1. Are there any versions of the Orleans framework which resolve this issue?
  2. Does version 1.1.3 resolve these issues?
  3. What causes reminders to not be executed?**

Thank you, Shazad Brohi

jdom commented 6 years ago

Oh, that's a very old version, a lot has changed since then, so I would know about reminders in particular. But I wanted to ask which backing store are you using for reminders? Because GrainTable is not meant to be reliable, in case you were using that.

sbrohi15 commented 6 years ago

@jdom I am using SQL Server as the backing store.

Which version of Orleans corrects these instability issues I am seeing?

jdom commented 6 years ago

Not really sure, maybe someone else can chime in. I do see several references to Reminder in the release notes for 1.1,X and 1.2.X.

Just to clarify, are you setting the backing store for the reminders (specifically) to SQL Server? Because using SQL Server for storage providers and/or membership doesn't automatically sets it as the backing store for reminders.

sbrohi15 commented 6 years ago

OrleansConfiguration.txt @jdom Yes, I am setting the backing store for reminders to SQL Server.

Attached is a copy of the OrleansConfiguration.xml file I am using to configure it. Let me know if there is anything incorrect with it.

jdom commented 6 years ago

I think that's correct. Just to be certain, when the silo runs, it should print which store it uses. Search for ReminderServiceType in the logs

sbrohi15 commented 6 years ago

@jdom I do see that set to SQL Server.

Do you know someone who can point me in the right direction here? There are issues with reminders not executing that are causing testing roadblocks for our QA team and we need to figure this out.

If a version upgrade will resolve this, what version do you or anyone else recommend upgrading to?

sergeybykov commented 6 years ago

@sbrohi15 A lot has been improved since the releases you are trying, and on the SQL integration. SQL is sensitive to the initialization scripts, and @veikkoeeva and @shayhatsor tweaked them quite a bit.

I would suggest you try moving to the latest version - 1.5.2. If that's not an option in the short term, at the very minimum share the silo logs. They may indicate what's failing.

sbrohi15 commented 6 years ago

@sergeybykov Thanks - that sounds like the best approach for now. If I have issues after this upgrade, then I will share the logs, but we have not found anything informative in the log files to point to why reminders are not being executed.

shayhatsor commented 6 years ago

I recall two bugs I've fixed that caused this issue, I don't know if your version contains the fixes. @jdom can probably help. The PRs are: #1757 and #1374.

jdom commented 6 years ago

According to release notes, looks like #1757 was included in 1.2.1, and I think #1374 was first included in 1.2.0.

Still, of course, if it's just as easy for you to jump to the latest version, that would be recommended. One thing to validate, is that if you are using storage providers and defaulting to binary serialization, then you should make sure you can still read those types correctly...

sbrohi15 commented 6 years ago

Thanks for the information all! I will weight trade-offs between upgrading to version 1.2.1 and 1.5.2 (latest).

veikkoeeva commented 6 years ago

@shayhatsor Might remember better, but storage-wise there shouldn't be disruption to data to upgrade. The queries have changed, but it's a matter of restarting. Configuration etc. has changed, so that might be the bigger trouble -- but worth it.

shayhatsor commented 6 years ago

@veikkoeeva, I think you're right. The fixes weren't on the SQL Server side.

sbrohi15 commented 6 years ago

@shayhatsor Can you please clarify on your last comment, "I think you're right. The fixes weren't on the SQL Server side."? Are you implying that the issues with reminders not executing has not been resolved for the SqlServer Reminder Service Type and that we still have issues, even with version 1.5.2?

shayhatsor commented 6 years ago

As far as I know there are no issues with the SQL Server reminder implementation. I meant that the issues were fixed in the C# code and not the SQL scripts.

sbrohi15 commented 6 years ago

@shayhatsor Thanks for clarifying.