[PERF] Enhance Scalability and Simplify Workflow Execution with Proto.Actor

[sfmskywalker]

Objective

To improve the scalability and manageability of workflow execution, we propose refactoring the IWorkflowRuntime interface and fully adopting Proto.Actor as our primary execution runtime.

Current Challenges

• The existing IWorkflowRuntime interface is responsible for both workflow lookup and execution, leading to code duplication and complexity across its implementations (Default Runtime and ProtoActor Runtime).
• Managing distributed workflow instance cancellation is cumbersome, requiring complex orchestration and distributed messaging.

Proposed Changes

1. Decouple Workflow Matching and Execution:

   • Remove workflow matching responsibilities from IWorkflowRuntime.
   • Introduce a WorkflowMatcher service to handle finding workflows based on stimuli (renamed from "bookmark + trigger hashes").

2. Interface and Method Refinements:

   • Add a CreateWorkflowClient method to IWorkflowRuntime to return an IWorkflowClient, serving as a proxy to the workflow instance.
   • Transition to using Local Workflow Runtime and ProtoActor Workflow Runtime to reflect different operational contexts.

3. Terminology and Structural Adjustments:

   • Rename "bookmark + trigger payloads" to "stimuli" to clarify their role in triggering workflow actions in response to external input (stimuli).
   • Workflows can execute in "local context" (e.g. current HTTP context) and in "remote context" (e.g. Proto.Actor Workflow Runtime).
   • A workflow instance can transition from local context to remote context and vice versa through explicit operations that are yet to be defined.

4. Enhancements through Proto.Actor:

   • Lock-Free Execution: Utilize Proto.Actor’s capabilities for lock-free workflow execution, significantly increasing throughput.
   • Event Sourcing: Leverage Proto.Actor's event sourcing to ensure reliability and recoverability, reducing the need for the current WorkflowInboxMessages DB table.
   • Improved Pub/Sub: Leverage Proto.Actor for faster event broadcasting compared to traditional event buses.
   • Simplified Cancellation: Enable more straightforward cancellation of workflows via Proto.Actor, even during a burst of execution.
   • Performance Improvements: Maintain workflow state in memory within actors, allowing for quicker resumption and response to new stimuli.

Benefits

Adopting this refactoring will lead to a more scalable and efficient system with significantly simplified codebase and operational enhancements. The use of Proto.Actor will allow us to leverage advanced distributed system features, enhancing overall system responsiveness and reliability.

[glime-ai[bot]]

To address the challenges and improve the scalability and manageability of workflow execution, the proposed changes to the IWorkflowRuntime interface and the adoption of Proto.Actor as the primary execution runtime seem to be a strategic move. Here's a summary of the proposed changes and their benefits:

1. Decouple Workflow Matching and Execution: By introducing a WorkflowMatcher service, the system can separate concerns, leading to cleaner code and potentially easier maintenance.

2. Interface and Method Refinements: The addition of a CreateWorkflowClient method to IWorkflowRuntime will provide a clear and consistent way to interact with workflow instances, improving the developer experience.

3. Terminology and Structural Adjustments: The new terminology ("stimuli") and the concept of local and remote contexts will help clarify the workflow execution process and the role of different components within the system.

4. Enhancements through Proto.Actor: The benefits of using Proto.Actor include lock-free execution, event sourcing, improved pub/sub mechanisms, simplified cancellation, and performance improvements due to in-memory state management.

Overall, these changes are expected to result in a more scalable, efficient, and manageable system. The adoption of Proto.Actor will leverage its distributed system features, which should enhance the system's responsiveness and reliability.

This comment was generated by Glime.