Slow processing Issue when orchestrator is being called on azure functions v2

[tejashshrestha]

We are building a workflow system where we are using durable functions to process events through event hubs instance and then spin up orchestrator and sub orchestrator functions. The set up we have is pretty simple.

1. A orchestration client listens to event hubs and then spins up an orchestrator.
2. The orchestrator calls an activity trigger which just reads the input and stores it in table storage.
3. Upon completion the orchestration client stores the response in a table in table storage

We tested it on a consumption plan in UK west and we wrote 400 records to event hubs in one second. The total count that was written by activity function and the orchestration client was ~318. The most servers that were online were 4 at any point.

Here are the codes for orchestration client, orchestration master, activity function and host.json. We are using the durabletask binding version 1.8.5 and event hubs binding version 4.1.1

Code for StartTrigger Function [Orchestrator Client]

function.json { "bindings": [ { "type": "eventHubTrigger", "name": "myQueueItem", "direction": "in", "eventHubName": "MyEventHub", "connection": "AzureEventHubConnection" }, { "name": "starter", "type": "orchestrationClient", "direction": "in" } ] }

index.js

const df = require('durable-functions'); const utils = require('../utils'); module.exports = async function (context, myQueueItem) { context.log('Inside Start Function eventHub: ', myQueueItem); const client = df.getClient(context); const document = myQueueItem; try { const instanceId = await client.startNew('orchestrator_Master', undefined, document); console.log('Started Orchestration with ID : ', instanceId); const response = client.createCheckStatusResponse(context.bindingData.req, instanceId); //custom function to just log in table storage await utils.storeOrchestratorResponse({ instanceId, document }, response.body.statusQueryGetUri); } catch (error) { context.log('Error in StartFunc: ', error); await new Promise((res, rej) => { setTimeout(res('done'), Math.floor(Math.random(100, 2000) * 100)) }); await utils.errorInFile(error, 'orchestratorClient'); } };

Code for OrchestratorMaster function.json { "bindings": [ { "name": "context", "type": "orchestrationTrigger", "direction": "in" } ], "disabled": false }

index.js

const df = require('durable-functions'); module.exports = df.orchestrator(function* (context) { try { const inputData = context.df.getInput(); const instanceId = context.df.instanceId; console.log('Instance idL ', instanceId); yield context.df.callActivity('activityFunc', { inputData, instanceId }); } catch (error) { console.log('Error in MAster: ', error); } });

ActivityFunction function.js { "bindings": [ { "name": "name", "type": "activityTrigger", "direction": "in" } ], "disabled": false }

index.js const utils = require('../utils'); module.exports = async function (context, req) { try { const { inputData, instanceId } = req; console.log("Inside Activity: ", req); //custom function to just log in table storage await utils.logInDB(inputData.i, instanceId); } catch (error) { console.log('Error in Activity Func: ', error); await utils.errorInFile(error, 'queue'); } };

host.json { "version": "2.0", "functionTimeout": "00:09:59", "extensions": { "http": { "routePrefix": "", "dynamicThrottlesEnabled": true, "maxOutstandingRequests": 10000, "maxConcurrentRequests": 5000 }, "durableTask": { "HubName": "%HUB_NAME%", "partitionCount": 16 }, "queues": { "batchSize": 32, "newBatchThreshold": 16, "visibilityTimeout": "00:00:05" }, "eventHub": { "maxBatchSize": 64, "prefetchCount": 256, "batchCheckpointFrequency": 5 }, "extensionBundle": { "id": "Microsoft.Azure.Functions.ExtensionBundle", "version": "[1.*, 2.0.0)" } } }

Would really appreciate if you could please help us with this issue. Is there anything wrong we are doing in how we have architected this solution? If we are unable to resolve this, we will have to do a major rewrite of our codebase.

Appreciate your help on this. Thanks in Advance.

[lilyjma]

@tejashshrestha - apologies for the very delayed response. Functions runtime host v2 reached the end of life on December 13, 2022. I also noticed that you were using an older version of the extension bundle. If this issue is still relevant to you, please try using host v4 and newer version of extension bundle and let us know if you're still experiencing this problem. Thanks!

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.