search

dnnsoftware / Dnn.Platform

DNN (formerly DotNetNuke) is the leading open source web content management platform (CMS) in the Microsoft ecosystem.
https://dnncommunity.org/
MIT License
1.03k stars 751 forks source link

Scheduler server change causing multiple emails to be shot to users. #5863

Open alokagarwal1979 opened 1 year ago

alokagarwal1979 commented 1 year ago

Is there an existing issue for this?

What happened?

We have DNN mail job scheduled to run on one server only (in all we have 4 web servers in web farm).

We read in one of the bug fix - https://github.com/dnnsoftware/Dnn.Platform/issues/5575, that scheduler is adjusted to change server if one of the server in web farm goes unavailable for some reason. Scheduler server got change to another server on web server list. This cause the mail job to restart again, so the users who probably got the emails earlier with the previous mail job ran got the email again.

We have checked with the smtp team, users who has got duplicate emails are receiving it from different servers.

Can this behavior of server change be controlled some way so that it always run on the same one server, through some configurations?

Steps to reproduce?

We have DNN mail job scheduled to run on one server only (in all we have 4 web servers in web farm).

We read in one of the bug fix - https://github.com/dnnsoftware/Dnn.Platform/issues/5575, that scheduler is adjusted to change server if one of the server in web farm goes unavailable for some reason. Scheduler server got change to another server on web server list. This cause the mail job to restart again, so the users who probably got the emails earlier with the previous mail job ran got the email again.

We have checked with the smtp team, users who has got duplicate emails are receiving it from different servers.

Can this behavior of server change be controlled some way so that it always run on the same one server, through some configurations?

Current Behavior

As mentioned in write up.

Expected Behavior

It should always run on one scheduled job

Relevant log output

No response

Anything else?

No response

Affected Versions

9.13.0 (latest release)

What browsers are you seeing the problem on?

Chrome, Microsoft Edge

Code of Conduct

mitchelsellers commented 1 year ago

The only time that the logic to switch servers should happen is if the server is offline and no longer reporting. Can you confirm that the server had gone offline?

alokagarwal1979 commented 1 year ago

Closed mistakenly... Server is not offline as such, but to me it appears that it is the only case that is why switching is happening even though we have set scheduled job to run on one server only. Do we have any other case when servers in a job scheduled on dnn scheduler can switch to some other server, even though it is being set to run on one server?

alokagarwal1979 commented 1 year ago

Also wanted to mention that we are using Mandeeps LiveCampaign DNN Module for scheduling newsletters in our product. And have also raised support with them, in case it is issue at the product end. Regarding this their response till now is as below:-

"that looks like a DNN Bug. You should report that over at DNN GitHub. Unfortunately, this cannot be handled within the code. You'll need to either wait for a bug fix from DNN or revert back to an older version from backup. "

"That particular issue indicates that if a scheduled job is on a specific server which no longer exists then it will automatically be moved over to another server. You should still not have the same scheduled job running on multiple servers. "

Which is confusing as we can see scheduled mail job is running from multiple servers, even though it has been set to run on on single server, from DNN documentation I got this thread https://github.com/dnnsoftware/Dnn.Platform/issues/5575 only which says that it is possible only in the server is not available that time, then it can switch to some other server. Do let me know if I am missing any other scenario here?

GerardSmit commented 1 year ago

Server is not offline as such, but to me it appears that it is the only case that is why switching is happening even though we have set scheduled job to run on one server only.

If the server isn't offline, can you see in the event viewer if the application pool has been shut down due to time-out? You could increase the application pool time-out, so it doesn't shut down.

The mails being sent twice could be a problem in the DNN module. When I look at this problem, it looks like the database record (or wherever it's being stored) is not updated after the mail has been sent but when the scheduler has been finished, causing the next time the scheduler is being started after it crashed (or shut down due time-out) the mails being resend again.

For DNN contributors if this is the problem:

In the DNN Scheduler a thread is created:

https://github.com/dnnsoftware/Dnn.Platform/blob/bc86e16085ab925c839557df8c03a8af9e9421e2/DNN%20Platform/Library/Services/Scheduling/DNNScheduler.cs#L241

There is a separate method HostingEnvironment.QueueBackgroundWorkItem that delays the shutdown of an IIS AppPool, until the task is completed. Maybe we could use this method instead of a Thread?

alokagarwal1979 commented 1 year ago

t updated after the mail has been sent but when the scheduler has been finished, causing the next time the scheduler is being started after it crashed (or shut down due time-out) the mails being resend again.

@meetmandeep, can you please suggest on last paragraph above.

"The mails being sent twice could be a problem in the DNN module. When I look at this problem, ....."

meetmandeep commented 1 year ago

@alokagarwal1979 The module has built-in checks to prevent duplicates even if multiple Scheduler Threads are running or if Scheduler stops & restarts. I would look at your Scheduler Logs to see exactly what's going on.