search

Azure / azure-functions-dotnet-worker

Azure Functions out-of-process .NET language worker
MIT License
403 stars 166 forks source link

Runtime exception after running .net upgrade assistant from 6 to 8 and converting to isolated worker model #2444

Open Camios opened 2 months ago

Camios commented 2 months ago

Description

I had a .net 6 in-process model function app which has a HTTP Trigger and a CosmosDBInput binding. I ran the .net upgrade assistant to convert to a .net 8 isolated worker model. It claimed success but left me with a bunch of Cosmos binding related errors. I tried fixing them manually and got it running, but now I'm getting a really strange runtime exception "FunctionInputConverterException".

It seems to wrap the real failure to find assembly Microsoft.Bcl.AsyncInterfaces 5.0.0.0

Related complaint: The examples section for isolated model is very sparse and deserves to expanded to cover more examples.

As far as I can tell from the Usage section, it should support IEnumerable and if I don't specify Id or SqlQuery, then it will return all documents.

[Function("ListEditors")]
public static Task<IActionResult> Run(
  [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
  [CosmosDBInput(databaseName: "EditorList", containerName: "Items", Connection = "CosmosDbReadOnlyConnectionString")] IEnumerable<object> documents)

The CosmosDBInput binding is just supposed to return all items in the database's container but fails with below exception. I'm not using any variables from the route (which is a known bug)

[2024-05-07T03:39:50.885Z] Function 'ListEditors', Invocation id '7b5efa25-3d1d-4dfb-8db4-2c7789dca3b0': An exception was thrown by the invocation.
[2024-05-07T03:39:50.886Z] Result: Function 'ListEditors', Invocation id '7b5efa25-3d1d-4dfb-8db4-2c7789dca3b0': An exception was thrown by the invocation.
Exception: Microsoft.Azure.Functions.Worker.FunctionInputConverterException: Error converting 1 input parameters for Function 'ListEditors': Cannot convert input parameter 'documents' to type 'System.Collections.Generic.IEnumerable`1[[System.Object, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]' from type 'Microsoft.Azure.Functions.Worker.Grpc.Messages.GrpcModelBindingData'. Error:System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
[2024-05-07T03:39:50.888Z] File name: 'Microsoft.Bcl.AsyncInterfaces, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
[2024-05-07T03:39:50.888Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.CreateTargetObjectAsync(Type targetType, CosmosDBInputAttribute cosmosAttribute)
[2024-05-07T03:39:50.890Z]    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
[2024-05-07T03:39:50.891Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.CreateTargetObjectAsync(Type targetType, CosmosDBInputAttribute cosmosAttribute)
[2024-05-07T03:39:50.892Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.ToTargetTypeAsync(Type targetType, CosmosDBInputAttribute cosmosAttribute) in D:\a\_work\1\s\extensions\Worker.Extensions.CosmosDB\src\CosmosDBConverter.cs:line 85
[2024-05-07T03:39:50.893Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.ConvertFromBindingDataAsync(ConverterContext context, ModelBindingData modelBindingData) in D:\a\_work\1\s\extensions\Worker.Extensions.CosmosDB\src\CosmosDBConverter.cs:line 57
[2024-05-07T03:39:50.894Z]    at Microsoft.Azure.Functions.Worker.Context.Features.DefaultFunctionInputBindingFeature.BindFunctionInputAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Context\Features\DefaultFunctionInputBindingFeature.cs:line 94
[2024-05-07T03:39:50.895Z]    at Foo.Azure.BarFunctionApp.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in E:\dev\frosted-azure-deployment\Apps\BarArtifacts\Foo.Azure.BarFunctionApp\obj\Debug\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 31
[2024-05-07T03:39:50.896Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-07T03:39:50.897Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-07T03:39:50.898Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
Stack:    at Microsoft.Azure.Functions.Worker.Context.Features.DefaultFunctionInputBindingFeature.BindFunctionInputAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Context\Features\DefaultFunctionInputBindingFeature.cs:line 94
[2024-05-07T03:39:50.899Z]    at Foo.Azure.BarFunctionApp.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in E:\dev\frosted-azure-deployment\Apps\BarArtifacts\Foo.Azure.BarFunctionApp\obj\Debug\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 31
[2024-05-07T03:39:50.900Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-07T03:39:50.901Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-07T03:39:50.902Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89.
[2024-05-07T03:39:50.976Z] Executed 'Functions.ListEditors' (Failed, Id=7b5efa25-3d1d-4dfb-8db4-2c7789dca3b0, Duration=1210ms)
[2024-05-07T03:39:50.978Z] System.Private.CoreLib: Exception while executing function: Functions.ListEditors. System.Private.CoreLib: Result: Failure
Exception: Microsoft.Azure.Functions.Worker.FunctionInputConverterException: Error converting 1 input parameters for Function 'ListEditors': Cannot convert input parameter 'documents' to type 'System.Collections.Generic.IEnumerable`1[[System.Object, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]' from type 'Microsoft.Azure.Functions.Worker.Grpc.Messages.GrpcModelBindingData'. Error:System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
[2024-05-07T03:39:50.981Z] File name: 'Microsoft.Bcl.AsyncInterfaces, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
[2024-05-07T03:39:50.983Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.CreateTargetObjectAsync(Type targetType, CosmosDBInputAttribute cosmosAttribute)
[2024-05-07T03:39:50.985Z]    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
[2024-05-07T03:39:50.988Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.CreateTargetObjectAsync(Type targetType, CosmosDBInputAttribute cosmosAttribute)
[2024-05-07T03:39:50.990Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.ToTargetTypeAsync(Type targetType, CosmosDBInputAttribute cosmosAttribute) in D:\a\_work\1\s\extensions\Worker.Extensions.CosmosDB\src\CosmosDBConverter.cs:line 85
[2024-05-07T03:39:50.993Z]    at Microsoft.Azure.Functions.Worker.CosmosDBConverter.ConvertFromBindingDataAsync(ConverterContext context, ModelBindingData modelBindingData) in D:\a\_work\1\s\extensions\Worker.Extensions.CosmosDB\src\CosmosDBConverter.cs:line 57
[2024-05-07T03:39:50.995Z]    at Microsoft.Azure.Functions.Worker.Context.Features.DefaultFunctionInputBindingFeature.BindFunctionInputAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Context\Features\DefaultFunctionInputBindingFeature.cs:line 94
[2024-05-07T03:39:50.998Z]    at Foo.Azure.BarFunctionApp.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in E:\dev\frosted-azure-deployment\Apps\BarArtifacts\Foo.Azure.BarFunctionApp\obj\Debug\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 31
[2024-05-07T03:39:51.000Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-07T03:39:51.003Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-07T03:39:51.005Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
[2024-05-07T03:39:51.007Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack:    at Microsoft.Azure.Functions.Worker.Context.Features.DefaultFunctionInputBindingFeature.BindFunctionInputAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Context\Features\DefaultFunctionInputBindingFeature.cs:line 94
[2024-05-07T03:39:51.009Z]    at Foo.Azure.BarFunctionApp.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in E:\dev\frosted-azure-deployment\Apps\BarArtifacts\Foo.Azure.BarFunctionApp\obj\Debug\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 31
[2024-05-07T03:39:51.011Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-07T03:39:51.013Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 48
[2024-05-07T03:39:51.016Z]    at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
[2024-05-07T03:39:51.018Z]    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88.

Steps to reproduce

  1. Install and run Azure Cosmos DB Emulator. Wait a long time for the web page to load the UI. Note the primary connection string.
  2. Run the function app in a local debug session
  3. Note the console window shows the address or the function app's http trigger
  4. use postman to send request to the url of the function app
  5. Get exception
Camios commented 2 months ago

I tried generating a new solution for minimal reproduction, but I don't get the same problem. I copied the classes out of the problematic solution into the new solution and it doesn't have the same problem.

liliankasem commented 2 months ago

Just to clarify, a new solution with this function is working just fine as expected?

It seems like the migration tool struggled here, so maybe there are improvements to make to that. However without a repro, it will be difficult to investigate this. It could be an issue with the project file, and/or nuget references - are you on all the latest packages for the worker, worker sdk, and cosmos extension?

Re: samples - we have more scenarios documented here: https://github.com/Azure/azure-functions-dotnet-worker/blob/main/samples/Extensions/CosmosDB/CosmosInputBindingFunctions.cs

Camios commented 2 months ago

Hi @liliankasem

Just to clarify, a new solution with this function is working just fine as expected?

Yes

I found the upgrade assistant failed in the many ways but it was its oversight of not adding one or both of these packages to the project which is causing the runtime exception about missing file Microsoft.Bcl.AsyncInterfaces:

Here's a longer list of things the upgrade assistant didn't do to my existing Azure Function App project (mix of problems it left behind and things it doesn't do compared to new project):

The upgrade assistant should be doing "best effort" to convert and showing a warning if there's a problem.

liliankasem commented 2 months ago

Thank you for the detailed write up! I'm going to consider the original issue resolved and repurpose this issue to track migration assistant enhancements - sound fair?

Camios commented 1 month ago

From my perspective:

  1. I'm not blocked by the runtime exception anymore and the focus turns more to addressing improvements to the upgrade assistant.
  2. I did have some outstanding questions (I bolded them in my previous command) and I was hoping you could answer promptly to help my migration.

Re: samples - we have more scenarios documented here: https://github.com/Azure/azure-functions-dotnet-worker/blob/main/samples/Extensions/CosmosDB/CosmosInputBindingFunctions.cs

Thanks for the link, I just found the Examples on learn.microsoft.com were insufficient and ideally expanded. And if it doesn't exist then linking to the github samples from learn.microsoft.com page would be helpful too.

liliankasem commented 1 month ago

Had Dependency injection in startup.cs with [assembly: FunctionsStartup(typeof(Startup))] but I think it should have converted to HostBuilder.ConfigureServices in the newly added prrogram.cs and removed startup.cs?

Correct, DI should be moved to Program.cs and Startup.cs should be removed.

It went from using Newtonsoft.Json serializers to System.Text.Json but didn't convert any of the POCO models that were using Newtonsoft-specific attributes. This annoyingly causes runtime exceptions. And there's https://github.com/Azure/azure-functions-dotnet-worker/issues/2131 and System.Text.Json has functional differences which might be breaking (like a lack of property-level JsonProperty DefaultValue handling)

You should be able to configure a Newtonsoft serializer (example). There is unfortunately a bug with this when using AspNetCore .ConfigureFunctionsWebApplication. But you should still be able to do this using .ConfigureFunctionsWorkerDefaults if you don't need AspNetCore. The Cosmos extension uses the worker config serializer (as shown in the example linked) as the serializer for the CosmosClientOptions.

New project created a mix of files in the Properties folder, but they weren't added during the upgrade to the existing project. Does it need them?

I think you only need launchSettings.json for VS, the rest can probably be removed.

The new project also added which wasn't in the existing project wasn't added to the upgraded project. Should the upgrade assistant be adding AspNetCore framework reference to upgraded Function Apps too?

<FrameworkReference Include="Microsoft.AspNetCore.App" /> is only required if you want to use the new HTTP extension Worker.Extensions.Http.AspNetCore (same for ConfigureFunctionsWebApplication), otherwise you don't need it.

Camios commented 1 month ago

It went from using Newtonsoft.Json serializers to System.Text.Json but didn't convert any of the POCO models that were using Newtonsoft-specific attributes. This annoyingly causes runtime exceptions. And there's https://github.com/Azure/azure-functions-dotnet-worker/issues/2131 and System.Text.Json has functional differences which might be breaking (like a lack of property-level JsonProperty DefaultValue handling)

You should be able to configure a Newtonsoft serializer (example). There is unfortunately a bug with this when using AspNetCore .ConfigureFunctionsWebApplication. But you should still be able to do this using .ConfigureFunctionsWorkerDefaults if you don't need AspNetCore. The Cosmos extension uses the worker config serializer (as shown in the example linked) as the serializer for the CosmosClientOptions.

When debugging the Cosmos input binding's JSON deserializing, I saw Microsoft.Azure.Functions.Worker.CosmosDBConverter has a hard-coded JsonSerializerOptions when deserializing which would prevent any settings override in Program.cs.

Is that CosmosDBConverter only used by Function Apps with the ASP.NET Core integrations (is that the bug you refer to)? Is a different DB converter used when the Function app doesn't use ASP.NET Core integration?

liliankasem commented 1 month ago

CosmosDBConverter has a hard-coded JsonSerializerOptions when deserializing which would prevent any settings override in Program.cs.

Yes it does have a hardcoded JsonSerializerOptions; what I was trying to convey is that there is a difference between that serializer and the cosmos client serializer that can be configured. The one that is hardcoded in the Converter is used when binding to a POCO. And the other is used by the CosmoClient, this one can be configured via workerOptions.Serializer.

Wanting the latter one to be configurable is perfectly valid feedback, and likely was an oversight. Please open a new issue to help us triage and address that.

Is that CosmosDBConverter only used by Function Apps with the ASP.NET Core integrations (is that the bug you refer to)? Is a different DB converter used when the Function app doesn't use ASP.NET Core integration?

No, this converter is used for all Cosmos input bindings.