Continue-as-new with a different WorkflowID

[dhiaayachi]

Is your feature request related to a problem? Please describe. Workflows that call continue-as-new frequently can lead to hot shards.

Describe the solution you'd like Many workflows rely on continue-as-new to process large datasets. Many of those workflows do not receive any external signals that require addressing them by WorkflowID.

The proposal is to support an option to continue-as-new to change WorkflowID. The whole chain of workflows still can be navigated through correspondent ids in the events as well listed using visibility queries.

[dhiaayachi]

Addressing Hot Shards Caused by Frequent Continue-As-New Calls

This issue proposes a solution to address hot shards caused by workflows frequently using continue_as_new.

Problem:

• Workflows that frequently call continue_as_new can result in hot shards. This happens because all executions with the same workflow ID are routed to the same shard, leading to imbalanced workload distribution.
• This issue primarily affects workflows processing large datasets that don't require external signals addressed by workflow ID.

Proposed Solution:

• Implement an option to change the WorkflowID when using continue_as_new.
• This would allow the creation of a new workflow with a different WorkflowID, thereby distributing the workload across shards.
• The lineage of the workflow executions could be tracked through correspondent IDs in the events and visibility queries.

Benefits:

• Improved Shard Distribution: Evenly distributes the workload across shards, alleviating hot shard problems.
• Performance Optimization: Reduces the load on individual shards, leading to better performance.
• Scalability Enhancement: Allows for more efficient scaling of the system as the workload grows.

Potential Challenges:

• Maintaining Lineage: Ensuring the ability to track the lineage of workflow executions, including the ability to perform queries based on previous WorkflowIDs.
• Implementation Complexity: This change requires updating the Temporal service to handle changing WorkflowID during continue_as_new.

References:

• Continue-As-New: Provides an overview of continue_as_new functionality.
• Temporal CLI command options reference: Details the command-line options for managing Temporal workflows.
• What is a Temporal Workflow?: Discusses workflow definitions, deterministic constraints, and versioning.

This proposal addresses a significant performance bottleneck, enabling better scalability and load balancing within Temporal workflows. It's important to carefully consider the implementation details and potential impact on existing features before implementing this change.

[dhiaayachi]

Thank you for your feature request! I understand you are looking for a way to support changing WorkflowID when calling continue-as-new. This is a very interesting feature and it could be beneficial for the workflows that do not receive any external signals that require addressing them by WorkflowID.

However, currently, Temporal does not support changing WorkflowID when calling continue-as-new. Instead, consider implementing this functionality using child workflows. A child workflow allows you to start a new workflow with a new WorkflowID that is not tied to the parent workflow. You can then continue to process the large dataset in the child workflow without having to continue the execution of the original workflow.

Please let me know if you have any other questions.

[dhiaayachi]

Thank you for your feature request.

Currently, Continue-As-New is not able to change the WorkflowID. You can work around this limitation by using a different workflow id for each iteration. This will prevent hot shards but it will require you to correlate workflow instances using their run IDs.

We appreciate you bringing this to our attention and will consider it for future releases.

[dhiaayachi]

Thank you for your feature request!

We understand that the ability to continue-as-new with a new workflow ID would be a valuable feature for workflows that process large datasets and don't require addressing by WorkflowID.

Currently, Temporal doesn't offer this capability out of the box. However, you can consider a workaround where you can implement your own logic to generate a new Workflow ID for each continue-as-new call based on the current state of the workflow or any other context you might have. This approach would give you the flexibility to manage workflow IDs as needed while still leveraging the power of continue-as-new for managing event history.

We value your feedback and will consider your suggestion for future releases.

[dhiaayachi]

Thank you for your feature request! We appreciate you taking the time to suggest this.

We understand that supporting an option to continue-as-new to change WorkflowID for workflows that do not receive external signals could be beneficial in managing hot shards.

At this time, Temporal doesn't directly support changing the WorkflowID via Continue-As-New. We're always exploring ways to improve our feature set, and we'll take your feedback into consideration for future development.

In the meantime, you could consider the following workaround:

• Use a unique identifier for each iteration: Instead of relying on WorkflowID, you could use a unique identifier (like a timestamp, counter, or other distinct value) to track the different iterations of your workflow. This identifier could be passed as a parameter to Continue-As-New, allowing you to track each iteration within the same WorkflowID.

Let us know if you have any other questions or if there's anything else we can help with.

[dhiaayachi]

Thank you for your feature request.

While we don't currently support continuing-as-new with a different WorkflowID, you can work around it by using different Workflow Types and creating a new workflow with the new WorkflowID in a separate function, triggered by the first workflow's continue-as-new. This lets you manage the workflow chain through event IDs and visibility queries.

Let us know if you have any further questions.