cgillum
commented
6 years ago Thanks for opening the issue. Just to clarify, you have a parent orchestrator which calls a sub-orchestrator, and both are succeeding, correct? In spite of this, you're seeing repeated failures in your logs for the sub-orchestrator?
A few other questions (apologies if I missed any of these details from your original post):
- Can you share an instance ID, a region, and an approximate time range where you last saw this issue?
- Do the errors ever stop, or do they just keep happening over and over?
- Assuming this is repeatable, do you know if these HTTP errors are new or if they might be associated with old orchestration instances?
I would like to investigate this more deeply to see if this might be an issue with the framework. Thanks for any additional information you can share.
pluralmonad
I am using durable orchestration function(s) to send batches of HTTP requests to a 3rd party API.
The job has been experiencing intermittent failures coming back as
System.Net.Http.HttpRequestException.The job is scheduled using a timer trigger which sends a queue message to a function with
OrchestrationClient. That function checks for a running instance and starts a new instance if one does not already exist and handles a few other scenarios. It is the only function that starts the orchestration and it always checks for existing before starting it. I disabled all scheduled triggers for the orchestration and had been kicking it off manually through the portal while testing.The HTTP issues appear to be fixed as the job has now completed every time I have kicked it off and watched it through the streaming logs. Although the orchestration succeeds, and I can see that it actually has because the 3rd party data is all updated, the log continues to show the below exception/stacktrace message it was failing with earlier, but after the orchestration has completed successfully. Another indicator it is completing as that the final step is to update a table entity recording its execution status, which is happening as expected.
I want to apologize ahead of time as this is a lot of lines of text, but I wanted to show the entire exception/stacktrace message that is appearing in the log repeatedly AFTER it has already completed.
That is the exception I had been troubleshooting all day and I finally got the job to run to completion without error, but that log message block still shows up repeatedly after completion.
I have seen the top level orchestration show up as executing and completing while only running for less than 1 second and often only a handful of milliseconds (while indicating no work being done since there are no log statements from within the job code being printed). I thought that it might have been because I was reusing a static instanceId in order to affect singleton behavior, but I have since stopped that and now store
LastKnownInstanceIDin table storage and cycle it as needed using newGuids.I have some thoughts on maybe what could be throwing a wrench into the orchestrations and DTFX cleanly completing, but they seem unlikely so I was hoping to get some feedback before adding anymore detail to this already long winded Issue.
I can show code and share App Service details as needed. Thank you for taking the time to read the wall of text :)
P.S. One more detail that may be material to the issue is that the failure is coming from a sub-orchestration initiated by the top level orchestration. Both completing successfully, as discussed above, before those errors show up in the log.
P.P.S No calls to sub-orchestrations or activities are being made with retry.
Adding .csproj for assembly version reference,