search

camunda / camunda-bpm-platform

Flexible framework for workflow and decision automation with BPMN and DMN. Integration with Quarkus, Spring, Spring Boot, CDI.
https://camunda.com/
Apache License 2.0
4.1k stars 1.55k forks source link

Extensible Task Polling with Parallel Execution in TopicSubscriptionManager #3962

Open shahryarSafizadeh opened 10 months ago

shahryarSafizadeh commented 10 months ago

User Story

As a user of the Camunda External Task Client, I am experiencing performance issues with the sequential task polling in the TopicSubscriptionManager. To address this, I propose a two-part enhancement to the starter:

Background

I propose dividing the issue into two parts to incrementally address the problem:

Part 1: Allowing Custom Task Polling Logic

Objective:

Allow users to extend TopicSubscriptionManager and override the task polling logic, enabling them to provide their implementation for parallel task polling.

Benefits:
  1. Users gain flexibility to implement their own parallel task polling logic.
  2. Provides a foundation for solving the overall performance issue.

Part 2: Optimizing Task Polling Logic

Objective:

Enhance the current task polling logic, which currently executes sequentially and by a single thread, to a more scalable and performant approach.

Approach:
  1. Separate the fetch and lock part (single-threaded) from the execution of handlers (multi-threaded).
  2. Implement a multi-threaded logic for executing tasks to improve performance and scalability.
Benefits:
  1. Improved performance and scalability in scenarios where parallel processing is crucial.

Now, let's discuss and address Part 1 through a pull request. Once that is successfully implemented and merged, we can then move on to tackling Part 2.

Next Steps:

I am willing to contribute to the implementation of Part 1 and submit a pull request. Let's focus on that aspect initially, and once it's resolved, we can discuss and plan for the implementation of Part 2.

Looking forward to your guidance on how to proceed with the first part of the enhancement.

danielkelemen commented 8 months ago

Hi @shahryarsz,

Thank you for opening this issue! The feature you are describing definitely makes sense and I agree that a multi threaded option would surely improve the client's performance vs. the current sequential approach.

We will soon look into your changes and provide feedback!

I link some issues that are related just for cross reference: https://github.com/camunda/camunda-bpm-platform/issues/2984

-Daniel