Allow TaskMetadata include the `retry` count for DecatonClient

[trinhnx]

Though DecatonClient is preferring over DecatonProcess (the docs say) but when I do deal with retry and retry times, Decaton lacks of this information because of the TaskMetadata does not have any properties to store retry count as the TaskMetadataProto so that we have to use DecatonProcess which is more verbose and more codes to write!

So could you consider to update TaskMetadata?

[ocadaruma]

Hi, let me ask few questions:

• Could you explain what DecatonProcess means here? (decaton-processor artifact?)
• Could you elaborate what you want to achieve actually? When you process your tasks in DecatonProcessor, you can access current retry count through ProcessingContext#metadata :https://github.com/line/decaton/blob/master/processor/src/main/java/com/linecorp/decaton/processor/TaskMetadata.java#L56

  • when I do deal with retry and retry times

[trinhnx]

@ocadaruma thank you for your feedback I will answer one by one

• DecatonProcess -> It was my typo mistake, it is DecatonProcessor from decaton-processor libs, version 3.1.0
• I want to implement the new Retry to support exponential back off (like DecatonTaskRetryQueueingProcessor) and hence we must access the retry count to increase the delay time. I see there is retryCount in metadata but i wonder that value is automatically increased by Decaton or we have to manual update it via specified context.getKey()) as in the DecatonTaskRetryQueueingProcessor In addition, i see that if the SubscriptionBuilder will override my DecatonProcessor for retry (it will use DecatonTaskRetryQueueingProcessor instead)

Please teach me if you have time! I appreciate your help a lot.

[ocadaruma]

Thanks for your explanation.

but i wonder that value is automatically increased by Decaton or we have to manual update it

retryCount is automatically increased by Decaton every time we call ProcessingContext#retry here: https://github.com/line/decaton/blob/master/processor/src/main/java/com/linecorp/decaton/processor/runtime/internal/DecatonTaskRetryQueueingProcessor.java#L58

I want to implement the new Retry to support exponential back off

I see. Then, Decaton's built-in retry feature may not be useable so you have to implement your own version of something like DecatonTaskRetryQueueingProcessor.

By the way, please note that implementing exponential-backoff (i.e. differentiate backoff duration depending on task) could be tricky. In Decaton, retry-backoff is roughly implemented like below:

• When ProcessingContext#retry is called, produce retry-task to the retry-topic with setting scheduled execution time to be now + backoff
• When a retry-task is consumed, delay the execution until the scheduled time

Let's say we have retry-taskA (retry-count = 1) and retry-taskB (retry-count = 2, so set longer backoff than A), and retry-taskB is produced (retried) bit earlier than taskA.

Then retry-taskA's execution could be delayed unintentionally long until retry-taskB got processed.

One possible approach to address this problem is preparing retry-topics per each retry-count, bit awkward and could require too many topics depending on the max retry count though.

[trinhnx]

@ocadaruma oc Thank you so much for your kind reply.

Then retry-taskA's execution could be delayed unintentionally long until retry-taskB got processed. So my understanding is that Decaton will honour the task order when it is pushed/consuming topic which lead to that effect (afaik it is based on Kafka mechanism too)

I also appreciate your suggestion but as you said, it is awkward and not so easy to control. I will sort out other way to suit my need.

Once again, thank you so much for your time