Function hangs and doesn't even timeout unless terminated

[shyamal890]

Describe the bug Long running function hangs just like in https://github.com/Azure/azure-functions-durable-js/issues/544 but for some reason doesn't timeout. This means even after 6 hours when the timeout was 3 hours it didn't timeout.

Investigative information

Durable Functions extension version: 2.x durable-functions npm module version: 3.0.0-alpha.5 Language (JavaScript/TypeScript) and version: Typescript 4.0.0 Node.js version: v18.2.1

If deployed to Azure App Service

• Timeframe issue observed: 2023-09-22 21:10:00.330 (Function started) ended when I manually triggered termination at 2023-09-23 04:31:21.905
• Function App name: ***links-node-v2
• Function name(s): durableActScrapeWebsite
• Region: SouthEastAsia
• Orchestration instance ID(s): 271c8ea925fe4e3b936e2d45828e5906

If you don't want to share your Function App name or Functions names on GitHub, please be sure to provide your Invocation ID, Timestamp, and Region - we can use this to look up your Function App/Function. Provide an invocation id per Function. See the Functions Host wiki for more details.

To Reproduce NA

While not required, providing your orchestrator's source code in anonymized form is often very helpful when investigating unexpected orchestrator behavior.

Expected behavior Function should have terminated after timeout duration

Actual behavior Termination didn't happen even after timeout

Screenshots NA

Known workarounds NA

Additional context

• Development environment (ex. Visual Studio)
• Links to source
• Additional bindings used
• Function invocation IDs

[shyamal890]

Recent hung orchestration instance id - 983c33044cfa4df6b0669502b56bfea9

Attaching get status output, check how even with functionTimeout of 03:00:00 it takes more than 8 hours.

{
    "name": "durableScraping",
    "instanceId": "983c33044cfa4df6b0669502b56bfea9",
    "runtimeStatus": "Failed",
    "input": "{\"domain\":\"hubspot.com\",\"max_pages\":\"2000\",\"scrape_with\":\"auto\"}",
    "customStatus": null,
    "output": "Orchestrator function 'durableScraping' failed: Activity function 'durableActScrapeWebsite' failed: Timeout value of 03:00:00 was exceeded by function: Functions.durableActScrapeWebsite \n {\"$type\":\"Microsoft.Azure.WebJobs.Host.FunctionTimeoutException, Microsoft.Azure.WebJobs.Host\",\"Timeout\":\"03:00:00\",\"InstanceId\":\"8ecce615-99a4-4107-8326-4fd065458147\",\"MethodName\":\"Functions.durableActScrapeWebsite\",\"Handled\":false,\"ClassName\":\"Microsoft.Azure.WebJobs.Host.FunctionTimeoutException\",\"Message\":\"Timeout value of 03:00:00 was exceeded by function: Functions.durableActScrapeWebsite\",\"Data\":null,\"InnerException\":null,\"HelpURL\":null,\"StackTraceString\":\"   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryHandleTimeoutAsync(Task invokeTask, CancellationToken shutdownToken, Boolean throwOnTimeout, CancellationToken timeoutToken, TimeSpan timeoutInterval, IFunctionInstance instance, Action onTimeout) in D:\\\\a\\\\_work\\\\1\\\\s\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 663\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\\\\a\\\\_work\\\\1\\\\s\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 571\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\\\\a\\\\_work\\\\1\\\\s\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 527\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\\\\a\\\\_work\\\\1\\\\s\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 306\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\\\\a\\\\_work\\\\1\\\\s\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 352\\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in D:\\\\a\\\\_work\\\\1\\\\s\\\\src\\\\Microsoft.Azure.WebJobs.Host\\\\Executors\\\\FunctionExecutor.cs:line 108\",\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":null,\"HResult\":-2146233088,\"Source\":\"Microsoft.Azure.WebJobs.Host\",\"WatsonBuckets\":null}",
    "createdTime": "2023-09-27T09:05:00Z",
    "lastUpdatedTime": "2023-09-27T17:11:08Z"
}

[nytian]

As mentioned in the issue #544, these two issues are from the same reason so I would suggest close this one and keep the discussions in #544.