Use raw history/matching client in history task executor

[dhiaayachi]

History task processing framework already has it's own retry policy re. task processing failures. Using retryable history/matching client for talking to remote service when processing tasks are unnecessary and will cause tasks to retry while holding the goroutine.

This issue is most obvious when those client requests are getting rate limited, which is a retryable error and is retried slowly (backoff.ThrottledRetry) by the history/matching retryable client while holding the goroutine.

[dhiaayachi]

Thanks for reporting this issue.

This behavior is related to the retryable client used for communicating with remote services during history or matching task processing.

The history/matching retryable client uses a backoff strategy (specifically, backoff.ThrottledRetry) for retrying rate-limited requests. This strategy is designed to prevent overwhelming the remote service, but it can lead to goroutine holding while waiting for the retry to complete.

Here are some potential solutions:

• Investigate the rate-limiting issue: Identify the source of the rate limits and see if there are ways to adjust the rate or duration of requests to avoid triggering them as frequently.
• Adjust Retry Policy: Consider adjusting the retry policy for the history/matching client to use a shorter backoff time or a different strategy, such as a backoff.ConstantRetry for rate-limited errors, while still protecting the remote service.

If you need further assistance, please provide the following information:

• The version of Temporal you are using.
• Your workflow code.
• The remote service you are communicating with and its rate-limiting configuration.

Let me know if you have any other questions.