[Feature request] Timeout for `deferCompletion()`

[hktechn0]

We want a feature setting a timeout for ProcessingContext.deferCompletion() to avoid stuck DecatonProcessor in any case.

Decaton supports async task processing with using deferCompletion() on processor. However, async processing usually makes a bug and it might miss to call complete(). Finally, it causes critical issue of consuming topics.

If decaton has a timeout for deferCompletion(), we can avoid such outage in most cases.

[kawamuray]

Thanks for your feedback :)

Yeah, after I saw several troubles that are caused by leaking a DeferredCompletion, it turns out that managing it properly and make sure to absolutely complete it is sometimes not an easy challenge and the consequence is really bad.

I'm +1 for adding kind of some support that lets users to configure forcing a DeferredCompletion to complete at some point even without an acknowledgement from the decaton processor.

Let me ask you few questions to consider the most suitable interface and implementation.

• When a DeferredCompletion completes forcefully, the corresponding record is considered as completed and the offset gets committed, so if it was not actually completed (failed) or the consumer instances crashes before it completes processing, it will lost. Is it fine for you?
• Is the timeout needs to be configured for each deferCompletion() call like deferCompletion(Duration) or, would it be sufficient to have single global timeout configured for each subscription?
• How long is it like, in practice you think you're going to set for the timeout?
• Do you think it will be useful if a user can select either to complete or retry the "expired" DeferredCompletion?

[hktechn0]

1. Yes, "pause" is more critical than "lost" for our workload. However, we want to track which task has been lost by timeout.
2. Global timeout for each subscription is enough for us.
3. It depends on topic, up to 1 min.
4. It seems very useful for retryable tasks.

[kawamuray]

Thanks for answering. I got your feedback and will take it into account when designing this feature :)

[kawamuray]

closing by https://github.com/line/decaton/pull/99