microsoft / durabletask-java

Java SDK for Durable Functions and the Durable Task Framework
MIT License
14 stars 7 forks source link

Incorrect retry count returned in case of Activity Function #167

Open kanupriya15025 opened 1 year ago

kanupriya15025 commented 1 year ago

I have an Activity Function which has a retry count set to 3. This is intentionally made to throw an exception and retry. Now, I am trying to get the retry count in the Activity Function but it always returns 0. Here is a sample code I am trying :

public String aepHTTPPollingOperator(@DurableActivityTrigger(name = "input", dataType = "string") String input,
                        ExecutionContext context) throws JsonMappingException, JsonProcessingException, Exception {

    System.out.println("Context Retry : " + context.getRetryContext().getRetrycount()); //always prints 0
    throw new RuntimeException("test exception")
}

This is how I am calling this activity :

RetryPolicy policy = new RetryPolicy(3, 30);
TaskOptions options = new TaskOptions(policy);
ctx.callActivity("ActivityName", activityInputQO.toString(), options, String.class));
cgillum commented 1 year ago

Ah, so unfortunately these are two different retry counts for two different retry features, and they are largely unaware of each other, which makes this understandably confusing.

In your activity function code, the API you're calling (I think) corresponds to a generic Azure Functions retry feature. It's not at all related to retry policies in Durable Functions. It's a local, in-memory, non-durable retry capability with a static configuration that works with a specific set of function triggers. The documentation is here.

Retry policies in orchestrations, on the other hand, only kick in when the orchestration actually receives a failure signal from the activity. When using the local retries, the orchestration is unaware of a failure until the local retries expire. Unfortunately, there's no way currently for an activity function to know whether it is retrying due to an orchestrator retry policy. This is why you're always getting for context.getRetryContext().getRetrycount() - I'm guessing you didn't configure the generic retry feature of Azure Functions for which this API was designed for.

In fact, if you use these two retry features together, you'd get multiple layers of retry.

kanupriya15025 commented 1 year ago

Got it. @cgillum Is there a way to find it out? Our aim is to send a failure metric once the activity fails and if we don't know what retry it is, we'll be sending a failure counter everytime it retries instead of a 1.

cgillum commented 1 year ago

We don't have a way to surface this information, unfortunately. We have a backlog item to surface retry count information to activities to help developers implement idempotency policies, but it hasn't yet reached the top of our current priority list.

kanupriya15025 commented 1 year ago

@cgillum Can you share that ticket so that I can track?