Implement dynamic task queue routing

[mfateev]

Is your feature request related to a problem? Please describe. Once an activity or workflow task is scheduled its task queue name is fixed. In many cases for development/troubleshooting and other scenarios ability to route all or subset of tasks to a different task queue can be very useful. See this @robzienert forum post for some usage examples.

Describe the solution you'd like Add API to manage routing rules that forward tasks to other queues.

Describe alternatives you've considered Use SDK library as described by @robzienert in his post.

[Multiply]

We'd be very interested in this, assuming it hits the go-sdk.

Our primary headache is running all microservices locally, and we've been trying various solutions to intercept traffic and route it to local environments. This feature could solve that to some degree, at least for the things running inside Temporal.

[jlegrone]

We've written some tooling to propagate request-scoped routing rules that allow targeting new task queues for child workflows or activities, including when these are deeply nested in the call stack. This approach is heavily inspired by Envoy's http routing rules.

Right now we use this technique primarily to support local dev in a shared Temporal cluster. Developers start a version of the worker they're making changes to locally, register this worker on a unique task queue, then make requests with routing rules on the context which target their new worker version.

I think this approach is complementary to the one described by Rob, as we also have use cases that require writing code to map task queues to worker topology based on the contents of the request or the client's environment. These rules are universally enforced and encoded in the worker/client binaries however, while request scoped routing rules allow targeted, last mile customization and support debugging without needing to deploy new worker versions.

This is something we've been considering open sourcing. Happy to go into more detail, but for now here's the API we're using to define routing rules. You'll see lots of reserved fields; we've considered use cases beyond modifying the task queue, including routing requests to a new workflow or activity type.

syntax = "proto3";

package temporal.router.v1;

message RoutingRule {
    message StringMatchRule {
        oneof match {
            // Match full text of string
            string exact = 1;
            // Match if string has prefix
            string prefix = 2;
        }
    }
    message MatchCondition {
        // Temporal task queue. This is required.
        StringMatchRule task_queue = 1;
        // Temporal namespace. If unset, all namespaces will match.
        reserved "namespace"; // not available for non-worker clients
        // Other fields that apply to both workflows and activities
        reserved 2 to 50;
        // The workflow name (also referred to as workflow type)
        reserved "workflow_name";
        // Other fields that apply only to workflows
        reserved 51 to 100;
        // The activity name (also referred to as activity type)
        reserved "activity_name";
        // Other fields that apply only to activities
        reserved 101 to 150;
    }
    message Destination {
        // New task queue
        string task_queue = 1;
        reserved 2 to 50;
        // New workflow name
        reserved "workflow_name";
        reserved 51 to 100;
        // New activity name
        reserved "activity_name";
        reserved 101 to 150;
    }
    // A human-readable name for the routing rule. Useful for debugging.
    string name = 1;
    // A set of conditions that determine whether the rule should be applied. When multiple
    // conditions are supplied, only one must match in order for the rule to apply.
    repeated MatchCondition match = 2;
    // A destination that determines how the request should be modified.
    Destination destination = 3;
    // A weight from 1-100 that represents the percentage of traffic to which this rule
    // should be applied.
    reserved "runtime_weight";
}

message RoutingRules {
    // A list of rules that will be matched, in order, for outgoing requests.
    // The first rule in the list which matches the request will be used.
    repeated RoutingRule rules = 1;
}