Add alternative for DurableClientFactory when using Worker.Extensions.DurableTask

[RichardPoes]

Is your feature request related to a problem? Please describe. In our .NET 6 architecture, we use a DurableClientFactory in ProjectA, which is a public web API, to create a DurableClient which starts an orchestration in ProjectB. ProjectB is a C# functionApp running in InProcess-mode. ProjectA was able to do this using the package Microsoft.Azure.WebJobs.Extensions.DurableTask.ContextImplementation. But, it seems that Azure Functions .NET is moving away from WebJobs and moving towards Worker.

We chose for this solution, becuase ProjectA is a public WebAPI and that WebAPI first had to send exactly the same HTTP request to ProjectB, where only then we could start the orchestration. So for us it prevented a lot of code duplication.

Are there any existing GitHub discussions or issues filed that help give some context to this proposal? The feature for WebJobs was implemented in PR #1125. I reckon a similar approach can be taken here.

Describe the solution you'd like Basically exactly the same as what PR #1125 had implemented. Thus a IDurableTaskClientFactory interface and a DurableTaskClientFactory implementing it. And it would return a DurableTaskClient which would enable one to start an orchestration outside of Azure Functions.

Describe alternatives you've considered The alternative is going back to our old architecture, passing the same HTTP request around via our public facing WebAPI in projectA, which would then send basically the same HTTP request to our private orchestrated FunctionApp ProjectB. Right now I'm in the process of checking whether it is possible to start a Worker Orchestration using the DurableClientFactory from the WebJobs package.

Additional context Since I'm asking for an almost exact 1:1 copy of a pre-existing feature in WebJobs I don't think much more information is necessary. I am curious as to whether this was considered. If it was deemed a bad idea I'd like to know why.

[RichardPoes]

Update: I tried it out, indeed, using WebJobs DurableClientFactory to start a Worker TaskOrchestration does not work. Code can be found here. This is somewhat unexpected, because I even changed the hubName so that both clients are using the same hub to start an orchestration. The same data appears in local storage. Yet, only when DurableTaskClient is used, an orchestration will start.

It appeared that the messages in the queue are not the same. Only the data stored in the tables.

Moreover injecting a DurableTaskClient directly in WebApi to start an orchestration in another FunctionApp also did not work. This resulted in:

Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler."))

[jviau]

@RichardPoes to make sure I understand, you want to be able to enqueue orchestrations from a non-function app and have your durable function app pick them up? If so, I believe you want this package:

https://www.nuget.org/packages?q=Microsoft.DurableTask.Client.OrchestrationServiceClientShim.

.ConfigureServices(s => s.AddDurableTaskClient(b => b.UseOrchestrationService(...)));

[RichardPoes]

Seems like to be the case indeed. I will close the issue and reopen it if it turns out not to work by any chance. If I remember to I will also update the documentation here.

[gorillapower]

@RichardPoes to make sure I understand, you want to be able to enqueue orchestrations from a non-function app and have your durable function app pick them up? If so, I believe you want this package:

https://www.nuget.org/packages?q=Microsoft.DurableTask.Client.OrchestrationServiceClientShim.

.ConfigureServices(s => s.AddDurableTaskClient(b => b.UseOrchestrationService(...)));

@jviau what about communicating with Orchestrations/Entities across FunctionApp boundaries? Is it possible to do this as was possible with the IDurableClientFactory and the relevant DurableClientOptions

 DurableClientOptions opts = new DurableClientOptions()
  {
      ConnectionName = connectionName,
      IsExternalClient = true,
      TaskHub = "MyExternalTaskHub"
  };

[nicolaor]

We are currently migrating our solution from in-process to isolated mode. In our solution we have external clients (no azure functions) which are storing and reading state from durable entities. This worked by injecting a IDurableClientFactory and creating the client by specifying the required connection parameters like this:

_durableEntityClient = durableClientFactory.CreateClient(new DurableClientOptions
{
    ConnectionName = "ConnectionXY",
    TaskHub = "MyTaskHub",
    IsExternalClient = true
});

We were then able to signal an etity as follows:

_durableEntityClient?.SignalEntityAsync(new EntityId("MyEntity", entityId), "EntityMethod", param)

I don't see a way to do the same by using the OrchestratiionServiceClientShim mentioned above. Am I overlooking something or is this currently not possible ?

[RichardPoes]

Okay, I finally got around into trying it out and it seems that the explanation of @jviau is a bit lacking. I did not manage to get this to work. Specifically, I fail to see what implementation of IOrchestrationServiceClient I would need to use or how I actually would even find that.

I also feel like that everything is in place to make this work, even in .NET 8 as all the information needed to start an orchestration is stored in Azure Storage. So as long as we can access that and have a HubName (as before) we can do this.

Addendum: I do realize that Azure uses some gRPC magic under the hood, so there might be more going on here. I just am a bit sad that some feature got randomly killed, without being offered an alternative. And I do know there are other ways to make this work, but to me this seemed so neat and the cleanest solution.

[nicolaor]

I actually found a PR with the implementation for durable entities created 7 months ago but never merged. https://github.com/microsoft/durabletask-dotnet/pull/228#pullrequestreview-2068788381

[RichardPoes]

That PR adds support for using Durable Entities, but it does not allow to start up a Orchestration from another app other than the Function App doing the actual orchestration itself, no?

[cliedeman]

Including a working sample I have if anyone needs it - this is specifically for the durabletask-mssql

    public static WebApplicationBuilder AddDurableTaskClient(this WebApplicationBuilder app)
    {
        app.Services.AddSingleton<IConnectionInfoResolver, ConnectionInfoResolver>();
        app.Services.Configure<DurableTaskOptions>(opts =>
        {
            // Can only be singleton hub for mssql
            // opts.HubName;
            opts.StorageProvider = new Dictionary<string, object>
            {
                ["connectionStringName"] = "DB",
                ["taskEventLockTimeout"] = "00:02:00",
                ["type"] = "mssql",
            };
        });

        app.Services.AddDurableTaskSqlProvider();
        app.Services.AddDurableTaskClient(builder =>
        {
            builder.UseOrchestrationService();
        });

        app.Services.AddSingleton<IConfigureOptions<ShimDurableTaskClientOptions>, ConfigureShimDurableTaskClientOptions>();
        return app;
    }

    private class ConnectionInfoResolver(IConfiguration config) : IConnectionInfoResolver
    {
        public IConfigurationSection Resolve(string name)
        {
            return config.GetSection("ConnectionStrings").GetSection(name);
        }
    }

    private class ConfigureShimDurableTaskClientOptions(
        IDurabilityProviderFactory providerFactory) : IConfigureOptions<ShimDurableTaskClientOptions>
    {
        public void Configure(ShimDurableTaskClientOptions options)
        {
            options.Client = providerFactory.GetDurabilityProvider();
        }
    }

disclaimer: this runs.... but it seems safer to just use the web jobs client for external clients for now