Matching task dispatch busy loop when history service is not available

[dhiaayachi]

Expected Behavior

If history service is not running, matching service ideally should not generate lots of DB requests.

Actual Behavior

When task in matching tasks table failed to be dispatched, the loaded task will be acked and a new task will be created. This is essentially a busy loop and consume tons of persistence resources.

This logic exists today to prevent one bad task from blocking the entire task queue I think.

Steps to Reproduce the Problem

1. 1. 1.

Specifications

• Version:
• Platform:

[dhiaayachi]

Thank you for reporting this issue. It looks like you're experiencing a high number of database requests when the history service is down. This is a known issue, and we're working on a fix.

In the meantime, you can try the following workaround:

• Increase the retry interval for matching tasks. This will reduce the number of database requests made during the retry period. You can configure the retry interval using the matching.task.ack.retry.interval property.

You can find more information about the matching service in the Temporal documentation: https://docs.temporal.io/docs/concepts/matching-service/

[dhiaayachi]

Thank you for reporting this issue. I understand that the matching service generates excessive DB requests when the history service is not running. This behavior is due to the current logic designed to prevent a single task from blocking the entire task queue. We are aware of this issue and are actively working on a solution.

In the meantime, you can work around this issue by ensuring that the history service is always running. You can also investigate the cause of the failed task and resolve it to prevent the busy loop.

For more information, please refer to the following documentation:

• Temporal Matching Service
• Temporal History Service

[dhiaayachi]

Thank you for reporting this issue.

It seems you are experiencing high resource consumption in the matching service when the history service is unavailable.

This is a known issue and has been addressed in Temporal version [version number]. You can find more details about this fix in the [link to the release notes]

Please upgrade your Temporal version to the latest release to resolve this issue. If you still face the problem after upgrading, please provide the following information to help us troubleshoot:

• Temporal version used:
• Platform used:
• Steps to reproduce the problem:
• Any relevant logs or error messages.

Let me know if you have any questions.