Azure / azure-functions-dotnet-worker

Azure Functions out-of-process .NET language worker
MIT License
414 stars 181 forks source link

Get "Webhooks are not configured" exception after converting to Isolated Worker under kubernetes/AKS #2443

Open SOFSPEEL opened 4 months ago

SOFSPEEL commented 4 months ago

What version of .NET does your existing project use?

.NET 6

What version of .NET are you attempting to target?

.NET 8

Description

So I converted my Function App to be

"FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated", by following this article: https://learn.microsoft.com/en-us/azure/azure-functions/migrate-dotnet-to-isolated-model?tabs=net8

It's running under kubernetes so my DockerFile uses the following image:

FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4.0-dotnet-isolated8.0
ARG CONFIGURATION
RUN mkdir -p /home/site/wwwroot
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true

COPY bin/${CONFIGURATION}/net8.0/publish "/home/site/wwwroot"

I can run the function app without problems when I run locally (using Rider IDE), and also when running in an Azure Function app, but when I run under Kubernetes I get the following exception: I want to point out that the function here is a cosmos trigger function that calls into an Orchestration (i.e. the DurableTask stuff in the ex) and it is failing somewhere in the DurableTask code.

Note that I was able to run my function app without issue prior to upgrading to isolated worker.

fail: Function.ProcessingRequestCosmos[3]
      Executed 'Functions.ProcessingRequestCosmos' (Failed, Id=fadaef1d-59d4-4919-8770-bf1f1634aa44, Duration=26ms)
      Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ProcessingRequestCosmos
       ---> System.InvalidOperationException: Webhooks are not configured
         at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.GetWebhookUri() in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\HttpApiHandler.cs:line 1244
         at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.GetUniversalQueryStrings() in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\HttpApiHandler.cs:line 1141
         at Microsoft.Azure.WebJobs.Extensions.DurableTask.BindingHelper.DurableOrchestrationClientToString(IDurableOrchestrationClient client, DurableClientAttribute attr) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\Bindings\BindingHelper.cs:line 40
         at Microsoft.Azure.WebJobs.Host.Bindings.PatternMatcher.<>c__DisplayClass5_0`3.<New>b__0(Object src, Attribute attr, ValueBindingContext ctx) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Bindings\PatternMatcher.cs:line 40
         at Microsoft.Azure.WebJobs.Host.Bindings.BindToInputBindingProvider`2.ExactBinding.BuildAsync(TAttribute attrResolved, ValueBindingContext context) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Bindings\BindingProviders\BindToInputBindingProvider.cs:line 221
         at Microsoft.Azure.WebJobs.Host.Bindings.BindingBase`1.BindAsync(BindingContext context) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Bindings\BindingBase.cs:line 50
         at Microsoft.Azure.WebJobs.Binder.BindAsync[TValue](Attribute[] attributes, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Bindings\Runtime\Binder.cs:line 117
         at Microsoft.Azure.WebJobs.Script.Binding.FunctionBinding.BindStringAsync(BindingContext context) in /src/azure-functions-host/src/WebJobs.Script/Binding/FunctionBinding.cs:line 221
         at Microsoft.Azure.WebJobs.Script.Binding.ExtensionBinding.BindAsync(BindingContext context) in /src/azure-functions-host/src/WebJobs.Script/Binding/ExtensionBinding.cs:line 112
         at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.<>c__DisplayClass12_0.<<BindInputsAsync>b__1>d.MoveNext() in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 154
      --- End of stack trace from previous location ---
         at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.BindInputsAsync(Binder binder) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 158
         at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 74
         at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
         at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:line 20
         at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
         at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 581
         at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 527
         at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 306
         --- End of inner exception stack trace ---
         at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 352
         at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 108
fail: Host.Results[0]

Project configuration and dependencies

net8.0 enable true false Exe v4 211c0ca5-21c2-4920-80a1-dd5993fe82f7 Debug;Release;Local <_FunctionsSkipCleanOutput>true AnyCPU;x86 enable FnApp.ProcReqs DEBUG;TRACE TRACE;RELEASE true PreserveNewest PreserveNewest Never ..\..\..\.AzureToolsForIntelliJ\AzureFunctionsCoreTools\v4\4.40.0\Microsoft.Azure.DocumentDB.Core.dll receiptConfig.json PreserveNewest

Link to a repository that reproduces the issue

No response

epDugas commented 4 months ago

I was also experiencing this issue when running the image from: mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0

Exception was being thrown from the same place as in OP's stacktrace (HttpApiHandler.cs, GetWebhookUri method.) return this.webhookUrlProvider?.Invoke() ?? throw new InvalidOperationException("Webhooks are not configured");

Following mention of this exception message in a previous issue HttpStart failure: Webhooks are not configured, I was able to workaround by adding the following Env variable to my container: WEBSITE_HOSTNAME=localhost:80 Which I guess isn't an acceptable workaround for a deployed container (?), but was sufficient in my case because the container was only being used for local integration testing.

SOFSPEEL commented 4 months ago

@epDugas, thanks for your feedback, so when I attempt to set the host as you mentioned (and other variations)

then this startup exception shows up, in other words, without setting the WEBSITE_HOSTNAME, the code makes it thru initialization.

What is so weird about the exception above is why is the code even doing anything with a webhook when calling into an orchestator function.

system.Net.Http.HttpRequestException:
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at Microsoft.Azure.WebJobs.Script.WebHost.SimpleKubernetesClient+<UpdateSecrets>d__17.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/KeyManagement/SimpleKubernetesClient.cs:84)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.KubernetesSecretsRepository+<WriteAsync>d__16.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/KeyManagement/KubernetesSecretsRepository.cs:71)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<PersistSecretsAsync>d__47`1.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/KeyManagement/SecretManager.cs:731)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<ModifyFunctionSecretsAsync>d__30.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/KeyManagement/SecretManager.cs:465)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<AddOrUpdateSecretAsync>d__28.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/KeyManagement/SecretManager.cs:405)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<AddOrUpdateFunctionSecretAsync>d__23.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/KeyManagement/SecretManager.cs:275)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider+<GetOrCreateExtensionKey>d__7.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/WebHooks/DefaultScriptWebHookProvider.cs:82)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetExtensionWebHookRoute (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/WebHooks/DefaultScriptWebHookProvider.cs:67)
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetUrl (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/WebHooks/DefaultScriptWebHookProvider.cs:52)
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.Microsoft.Azure.WebJobs.Host.Config.IExtensionConfigProvider.Initialize (Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=3.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\DurableTaskExtension.cs:368)
   at Microsoft.Azure.WebJobs.Host.DefaultExtensionRegistryFactory.Create (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\DefaultExtensionRegistryFactory.cs:38)
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions+<>c.<AddWebJobs>b__3_0 (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:58)
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter+<>c__DisplayClass3_0.<RegisterDescriptor>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/DryIocAdapter.cs:162)
   at DryIoc.Registrator+<>c__DisplayClass27_0.<RegisterDelegate>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:4550)
   at DryIoc.Factory+<>c__DisplayClass26_0.<ApplyReuse>b__2 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6605)
   at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7849)
   at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7834)
   at DryIoc.Factory.ApplyReuse (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6604)
   at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6564)
   at DryIoc.Factory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6634)
   at DryIoc.DelegateFactory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7740)
   at DryIoc.Container.DryIoc.IResolver.Resolve (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:290)
   at lambda_method1067 (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
   at DryIoc.Factory+<>c__DisplayClass26_0.<ApplyReuse>b__2 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6605)
   at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7849)
   at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7834)
   at DryIoc.Factory.ApplyReuse (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6604)
   at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6564)
   at DryIoc.Factory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6634)
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:210)
   at DryIoc.Container.DryIoc.IResolver.Resolve (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:195)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:99)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:82)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService (Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService (Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions+<>c.<AddWebJobs>b__3_6 (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:100)
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter+<>c__DisplayClass3_0.<RegisterDescriptor>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/DryIocAdapter.cs:162)
   at DryIoc.Registrator+<>c__DisplayClass27_0.<RegisterDelegate>b__0 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:4550)
   at DryIoc.Factory+<>c__DisplayClass26_0.<ApplyReuse>b__2 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6605)
   at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7849)
   at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7834)
   at DryIoc.Factory.ApplyReuse (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6604)
   at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6564)
   at DryIoc.Factory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6634)
   at DryIoc.DelegateFactory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7740)
   at DryIoc.Container.DryIoc.IResolver.Resolve (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:290)
   at lambda_method1065 (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
   at DryIoc.Factory+<>c__DisplayClass26_0.<ApplyReuse>b__2 (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6605)
   at DryIoc.Scope.TryGetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7849)
   at DryIoc.Scope.GetOrAdd (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7834)
   at DryIoc.Factory.ApplyReuse (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6604)
   at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6564)
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7083)
   at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6554)
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:7083)
   at DryIoc.Factory.GetExpressionOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6554)
   at DryIoc.Factory.GetDelegateOrDefault (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:6634)
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:210)
   at DryIoc.Container.DryIoc.IResolver.Resolve (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:195)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:99)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:77)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService (Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+<UnsynchronizedStartHostAsync>d__57.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.33.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs:343)