[Proposal] Adding control-queue monitor with dedicated orchestrator to each control-queue.

[pasaini-microsoft]

Motivation:

Issue: Unpredictable time to detect (TTD) for any orchestration being stuck in control-queue.

• We have been facing issues where DTF orchestration used to get stuck at random. Given that customer load is not very regular in our service, it was challenging to understand upfront if the orchestration would be processed or will be stuck.
• More often customers used to reach out with incidents complaining their request not completing for long time. Motivation: motivation was to reduce the TTD for finding if orchestration can be stuck/waiting-forever in a control-queue irrespective of the cause.
• This is where heartbeat orchestration comes into picture, which evaluates the control-queue siting outside the system (let's say system is completely down and cannot emit any metric or emitting false/partial metrics).
• We built this system locally in our service, and it has been detecting the issue with 100% accuracy. We build monitors around it to raise ICM and mitigate the situation manually using TSGs.

Issue: Manual efforts in identifying the impacted worker and mitigation steps.

• Post having heartbeat orchestrations, next issue was it used to take manual steps and active engineering cycles to identify the impacted worker and trigger mitigation. Motivation: motivation was to make detection of impacted worker and perform mitigation steps automatically.
• This is where monitor for these heartbeat orchestration and owner of control-queue arrives.
• We built this system locally in our service, and it has been identifying the impacted worker predictably and auto-mitigates the issue.

With both these systems in place, we are able to run DTF with no orchestrations stuck, without any manual steps for detection and mitigation. To bring this capability to all users of DTF users, proposing below change.

Proposals:

Adding control-queue monitor with dedicated orchestrator to each control-queue.

This change consists of 2 main portions:

1. Addition of orchestration and its instances exactly one for each control-queue.

   • This orchestrator (aka ControlQueueHeartbeatTaskOrchestrator) is very simple and quick to run.
   • It just waits for some interval (configurable) and logs a heartbeat message and continue itself as new.
   • It validates if the context is correct, just in case partition count changes (it assumes partition count doesn't change, otherwise it may result in closing off the orchestrators).
   • It provides a way for user to provide a delegate (callback) to run with each heartbeat.
   • This one gets the partition-count information and creates instance-ids such that one is allotted to each control-queue.

2. Addition of a monitoring of these orchestrations from last processed time.

   • Now, with the instance id for each control-queue, it checks for last updated time of orchestration instance and uses it measuring against threshold.
   • If orchestration found stuck, appropriate message is logged.
   • This one does a bit more to find point in time owners of each control-queue, to pin the impacted taskhubworker owning the control-queue.
   • It also provides a way for user to provide delegate (callback) to run when it detect any anomaly, like stuck orchestration, fetching owner fails, fetching orchestration instance fails, or any of these times out).

[davidmrdavid]

Hi @pasaini-microsoft: Thanks for opening this PR. As we discussed internally, I'll loop in the other DTFx/DF maintainers for discussion regarding the scope of the PR, as well as the design. I'm personally most excited about the ability to create an instanceID targetting a specific partition, so at the very least I'd like to get that in.

It would help greatly if you could update the PR description to contain not just a list of the changes, but also a small motivation and background behind this change. For example, you can explain that you're using this utility in your own app to help you detect stuck orchestrations, and so on. Thanks!

[jviau]

I love the idea of monitoring partition queue processing, but I don't think via an orchestration is the right way to do this. This is a health check after all, and there is a health check ecosystem in .NET. We should evaluate leveraging that. Metrics (#785) may also be a good avenue for this feature.

https://www.nuget.org/packages/Microsoft.Extensions.Diagnostics.HealthChecks/9.0.0-preview.2.24128.4

[pasaini-microsoft]

I love the idea of monitoring partition queue processing, but I don't think via an orchestration is the right way to do this. This is a health check after all, and there is a health check ecosystem in .NET. We should evaluate leveraging that. Metrics (#785) may also be a good avenue for this feature.

https://www.nuget.org/packages/Microsoft.Extensions.Diagnostics.HealthChecks/9.0.0-preview.2.24128.4

Thanks jviau. The reason for using orchestration was to keep the detection as independent from control-queue processing system as possible. This helps detect if control-queue is stuck for any reason including if taskhubworker is just absent or some control-queue just missed to be owned by any worker. This basically helps avoid false negatives.