Azure / azure-functions-durable-extension

Durable Task Framework extension for Azure Functions
MIT License
711 stars 263 forks source link

Function App doesn't scale when custom HubName is specified in app setting #2803

Open Skymogul opened 2 months ago

Skymogul commented 2 months ago

We have a Durable Function, .Net Core 8.0, isolated worker, in a consumption plan. We are using deployment slots. We have our production slot and a slot named stg. We set the Task Hub name using an app setting, named AzureFunctionsJobHost__extensions__durableTask__hubName and set as a slot setting so it does not move with the slot swap. We set the hub name this way in both slots.

We queue up over 15,000 activities. We would expect that the scale controller would add consumption workers as we are processing about 10/second/worker. In this configuration, the function app never scales beyond one worker.

We found that this was related to #111 . The fix ultimately committed there reads the hub name from the host.json, which means it doesn't work if you are setting the hub name via an app setting. I think we can agree that having to set the hub name in host.json would effectively disable the ability to swap slots.

The workaround we came up with was to set the hub name via app setting in the staging slot and set that as a slot setting, then not set it at all (let it be default) in the production slot. Once we did this, scale out worked as expected in the prod slot. However, the staging slot cannot scale out to multiple workers in this configuration.

jcageman commented 1 month ago

Could you try host.json as:

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "hubName": "%TaskHubName%",
      "storageProvider": {
        "maxQueuePollingInterval": "00:00:05"
      }
    },
    "queues": {
      "maxPollingInterval": "00:00:01.000"
    }
  }
}

And introducing an appsetting with the name "TaskHubName"? i am curious if that works out for you. It's a very similar approach though. Another idea, i am not entirely sure if those appsettings are case sensitive. AzureFunctionsJobHost__Extensions__DurableTask__HubNamecould also we worth a shot. I've been using that notation in another place.