Closed sunnyhay closed 3 years ago
Hi @sunnyhay - I took a look at your app and found some issues which are probably impacting your solution. You should be able to see the same analysis I'm looking at if you check out the Diagnose and Solve section of the Azure Functions portal.
In particular, it looks like you're reusing the same storage account and task hub name across multiple apps:
App Name | Task Hub | Storage Account | Example Instance ID |
---|---|---|---|
mps-webhookservice-ppe | AEPIngestorHub | mpsblobaccountppe | b109c87d8d3c460eaaa83536f0ac8923 |
mps-ingestorservice-ppe | AEPIngestorHub | mpsblobaccountppe | b109c87d8d3c460eaaa83536f0ac8923 |
When you do this, both of your apps will compete for messages in the same queues, often causing things to not run correctly. For that reason, each app you deploy must use either a unique task hub name or use a different storage account. More information here: https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-task-hubs?tabs=csharp
Also, you may want to upgrade the version of the nuget package you're using from 1.8.3 to 1.8.5 (even better if you can move to 2.2.2, but there are some breaking changes between 1.x and 2.x).
Thanks for prompt response Chris. Good catch! We've resolved such conflicting issue before. Yes, both services are using the same task hub intentionally since webhook service is used to send message to ingestor service. The webhook service is receiving messages pushed by Adobe IO and sends external events to waiting ingestor service. And webhook service is a pure Azure function, not a Durable function. My team is gonna have a big release next week and so breaking changes of dependency upgrade will be considered later after that. Sorry I forgot to mention the service plan details. We're using Production S1 box for app service plan. Lemme know if there is any scale concern for this issue. And this is an intermittent problem and we have no clue to track so far.
Since our function is wrapped in Data factory web activity, it's also possible ADF encounters some problem failing the function launching. I'd like to know how to troubleshoot this scenario using any function related debugging knowledge if possible. For any IaaS case, it's simply logon the machine and checking the log. For such PaaS case, may I get any valuable suggestion? Thanks.
Here are the best resources for debugging in Azure Functions. The information here applies both to PaaS and self-hosted:
Description
Expected behavior
Actual behavior
Relevant source code snippets
Known workarounds
App Details
Screenshots
If deployed to Azure