search

Azure / durabletask

Durable Task Framework allows users to write long running persistent workflows in C# using the async/await capabilities.
Apache License 2.0
1.47k stars 287 forks source link

Orchestration activities stuck not completing retry attempts, SB showing timer events for both. #1035

Open devinlstoen opened 5 months ago

devinlstoen commented 5 months ago

Hello, I'm trying to get a better understanding why these two orchestration are not completing their activity retries. The orchestrations are scheduled for 9 total retries of the RetryCreateSupportTicketActivity, but the logs are only showing that this activity has tried 6 times. The 9 retries are supposed to take ~9 hours to complete, but it looks like the last attempt was made on 1/25 19:03:32

efab5de3-46d8-4c43-8384-547f7b4d6f40

Here's what were seeing in Service Bus. There are two timer events corresponding to the orchestrations. They look to be stuck.

49f391bd-22e0-45b1-9573-5359d76ea3f7

cgillum commented 4 months ago

I don't think there's enough information here to provide help. Can you share the snippet of code that shows how you're configuring your retry policy?

devinlstoen commented 4 months ago

Sure, line 284 is where we are defining the number of retries for the RetryCreateSupportTicketActivity.

[cid:12af285b-55cc-48f6-a92e-c302d4004c4d]

From: Chris Gillum @.> Sent: Wednesday, February 7, 2024 10:56 AM To: Azure/durabletask @.> Cc: Devin Stoen @.>; Author @.> Subject: Re: [Azure/durabletask] Orchestration activities stuck not completing retry attempts, SB showing timer events for both. (Issue #1035)

I don't think there's enough information here to provide help. Can you share the snippet of code that shows how you're configuring your retry policy?

— Reply to this email directly, view it on GitHubhttps://github.com/Azure/durabletask/issues/1035#issuecomment-1932675947, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZGG27GMKBWLDDQOTYQRDLDYSPE4VAVCNFSM6AAAAABCTRPGPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGY3TKOJUG4. You are receiving this because you authored the thread.Message ID: @.***>

devinlstoen commented 4 months ago

Then on line 73 we're constructing the retry options for this activity:

[cid:22b01918-a9c8-41ed-add7-d0622f342f1c]

From: Devin Stoen @.> Sent: Wednesday, February 7, 2024 11:24 AM To: Azure/durabletask @.>; Azure/durabletask @.> Cc: Author @.> Subject: Re: [Azure/durabletask] Orchestration activities stuck not completing retry attempts, SB showing timer events for both. (Issue #1035)

Sure, line 284 is where we are defining the number of retries for the RetryCreateSupportTicketActivity.

[cid:12af285b-55cc-48f6-a92e-c302d4004c4d]

From: Chris Gillum @.> Sent: Wednesday, February 7, 2024 10:56 AM To: Azure/durabletask @.> Cc: Devin Stoen @.>; Author @.> Subject: Re: [Azure/durabletask] Orchestration activities stuck not completing retry attempts, SB showing timer events for both. (Issue #1035)

I don't think there's enough information here to provide help. Can you share the snippet of code that shows how you're configuring your retry policy?

— Reply to this email directly, view it on GitHubhttps://github.com/Azure/durabletask/issues/1035#issuecomment-1932675947, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZGG27GMKBWLDDQOTYQRDLDYSPE4VAVCNFSM6AAAAABCTRPGPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGY3TKOJUG4. You are receiving this because you authored the thread.Message ID: @.***>

jviau commented 4 months ago

@cgillum , I chatted with @devinlstoen offline and I think something of importance is they have not provided an IOrchestrationServiceInstanceStore or IOrchestrationServiceBlobStore to ServiceBusOrchestrationService (they supply null). I think this would be problematic for ServiceBus would it not? It looks like it at minimum expects a non-null IOrchestrationServiceBlobStore