Azure / durabletask

Durable Task Framework allows users to write long running persistent workflows in C# using the async/await capabilities.

Apache License 2.0

1.47k stars 287 forks source link

Get instance id for desired control-queue(s) #1069

Open pasaini-microsoft opened 2 months ago

pasaini-microsoft commented 2 months ago

Motivation

1079

Issue: No way of targeting an orchestrator instance to a desired control-queue.

We have been facing issues where DTF orchestration used to get stuck at random. Given that customer load is not very regular in our service, it was challenging to understand upfront if the orchestration would be processed or will be stuck.
More often customers used to reach out with incidents complaining their request not completing for long time.
This is where we needed orchestration instances to observe health of each queue by targeting one instance for desired control-queue.
Motivation:
motivation was to reduce the TTD for finding if orchestration can be stuck/waiting-forever in a control-queue irrespective of the cause.

Issue: No way to load lightly loaded control-queues.

We have face a few situations where some of control-queues are overwhelmed with orchestration instances while the others are happily processing almost nothing.
Motivation:
motivation was to target new instances of orchestration instances to set control-queue which are not heavily loaded

Proposal

API to generate instance id for a set of control-queues.

This API receives set of control-queues and prefix for instance id.
Implementation detail is: Allow special way of creating instance id with a suffix unsigned integer after delimiter '!' and explicitly use that value to route to control-queue (say suffixNumber % partitionCount). If this pattern is not used, it would goes back to default (current) which is hash(instance-id)%partition-count.