Unable to load assembly after upgrading Microsoft.NET.Sdk.Functions to v1.0.33+

[ramondeklein]

I have created an Azure Functions project using .NET Core v2 and attempted to migrate it to Azure Functions v3. After upgrading it complains that it cannot find Microsoft.Data.Edm, Version=5.8.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35 anymore. When I look in the bin folder of my v2.1 version, then this file is copied to that location. My v3.1 version lacks a lot of files, including Microsoft.Data.Edm.dll. It seems the file is copied to the bin folder, but it is being removed afterwards. Does any one know what is happening?

It seems the problems start happening when the Microsoft.NET.Sdk.Functions package is upgraded to a version higher than 1.0.31. To illustrate the problem I have created a very simple Azure Function solution that contains both the V2 and the V3 version. It can be found at https://github.com/ramondeklein/AzureFunctionsWithEdm.

When running the AzureFunctionsWithEdm2 the call http://localhost:7071/api/EdmFunction returns OK, but with AzureFunctionsWithEdm3 it fails, because it cannot load the Microsoft.Data.Edm assembly. When the Microsoft.NET.Sdk.Functions package is upgraded to 1.0.33, then the V2 also fails to work.

[ramondeklein]

It is possible to make it work with v1.0.33 by patching the build\ExtensionsMetadataGeneratorVersion.props (directly in the .nuget cache folder). When the following line:

    <ExtensionsMetadataGeneratorVersion>1.1.4</ExtensionsMetadataGeneratorVersion>

is changed to:

    <ExtensionsMetadataGeneratorVersion>1.1.2</ExtensionsMetadataGeneratorVersion>

then it works again. I understand that this isn't a proper workaround, but was done to diagnose this issue.

When I check commit https://github.com/Azure/azure-functions-host/commit/1ed2a54159eb3c9aaff11a9289be2bf1ce91dcb5, then it seems that a "feature" has been added (by @fabiocav) to remove certain assemblies during the build causing this issue.

[ramondeklein]

Proper workaround

It seems that the RemoveRuntimeDependencies task removes this assemblies (source). It doesn't run when the _FunctionsSkipCleanOutput variable is set to true. Adding the following line effectively disables this task and makes it work for 1.0.33 and later too.

  <PropertyGroup>
    <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>
  </PropertyGroup>

[TehWardy]

I've had a similar issue which I initially reported https://github.com/dotnet/core/issues/4390 but it was with the roslyn api's I was using in the functions code.

It appears that upgrading azure functions apps is not really an option. Having now done that and attempted to revert it's all gone horribly wrong.

could this be related to the previous issue I reported about newtonsoft json being uber strict with its referenced version? In short ... azure functions appears to be extremely picky about what stuff it will work with and what versions you can use with what.

[TehWardy]

explicitly referencing 1.0,34 for me results in this compile time warning ...

NU1603 Microsoft.Azure.WebJobs.Extensions 3.0.5 depends on Microsoft.Azure.WebJobs.Host.Storage (>= 3.0.11) but Microsoft.Azure.WebJobs.Host.Storage 3.0.11 was not found. An approximate best match of Microsoft.Azure.WebJobs.Host.Storage 3.0.13 was resolved.

[ramondeklein]

@TehWardy I'm not sure if that's the same issue, but I think it's related that Azure Functions runs .NET Core in-process. The host executable already loads some assemblies and if your app requires different versions, then I guess you're out-of-luck.

If you're targetting .NET Core v3, then you shouldn't reference 1.0.34, but stick with the v3 versions. If you can't do that, then you need to revert to Azure Functions v2.

The in-process hosting makes things a bit complicated. Fortunately, I could already test it locally without needing to redeploy over and over again.

[mhoeger]

cc: @brettsam

[TehWardy]

All I know is if I compile the same stack of code without the Azure Functions Wrapper for my execution (so just make it a console app) ... everything compiles and runs fine, so there's something about assembly resolution / reference versioning in the Azure Functions framework that's causing a compatibility limit that seems to break my stacks ability to compile without warnings and in some cases errors.

I'm literally changing nothing else to produce this stack behaviour.

I did have an example issue I could link to but I guess it's been deleted, it was the Newtonsoft.Json lib ... Azure functions forces us to use an outdated version of it, in more recent versions though after upgrading i'm finding that the Microsoft dependency stack that's behind the stuff I reference seems to not agree with itself a lot.

For now i've given up and i'm just running with a console app, but that isn't going to scale longer term so I hope this gets resolved before I need this to be an azure function.

[brettsam]

Using <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput> is the right solution here. We unfortunately don't have this documented -- I'm going to try to figure out where we can put that.

As @ramondeklein says, your function gets loaded into a host process when you hit F5. It launches a separately-downloaded ASP.NET app (brought with the Functions Core Tools) and loads your assembly into that process. The same thing happens when you deploy to the running service -- your assembly is loaded into a running process.

This "clean output" feature in the SDK helps to remove as much of the assembly duplication as possible -- there's no reason to deploy an assembly if you'll get it "for free" from the hosting environment. It significantly reduces deployment size, startup time, etc.

However, you need this assembly deployed as it's a different version than the one the Host is using. The "clean output" doesn't realize that and removes this file anyway.

@fabiocav -- can we make this feature more specific to a version of the assembly and only remove the ones that that match? Or is that too hard to maintain?

[ramondeklein]

@brettsam I'm not sure if Azure Functions is bound to a specific set of assemblies and if the versions of those assemblies are fixed. If so, then if the assembly is already on the host and the version is the same, then it's safe to remove it. When the version is not the same, then a problem may occur and it would be good to either disable this "cleaning" and/or issue a warning about the version mismatch in the assembly.

It took me quite some time to figure out what was going wrong here. Without access to the Azure-Functions repo (open-source FTW) I would have never figured out what was wrong.

For the mean-time it might be a good idea to show an informational message about the cleaning process during build-time and a link to a Github page that explains what it is doing.

[moxplod]

I had this issue as well. Good to find the workaround here.

[brettsam]

I'm going to close this issue and use https://github.com/Azure/azure-functions-host/issues/5894 to track the fix.

[euangordon]

Faced exactly the same issue with a .Net Framework v1 Function using Microsoft.NET.Sdk.Functions V3.0.9.

The DLLs for SqlServerTypes were not being copied to my Azure Function when publishing.

Adding <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput> Made no difference.

Downgrading Microsoft.NET.Sdk.Functions to V3.0.3 resolved the issue.

[brettsam]

FYI -- if you're using a v1 or v2 function, please use the latest v1 of the Sdk package (instead of the latest v3). Does that help?

[euangordon]

FYI -- if you're using a v1 or v2 function, please use the latest v1 of the Sdk package (instead of the latest v3). Does that help?

Nearly all of my older functions are working fine on v1.0.7 of the Sdk, but I have one Function using Azure Cosmos and there was something in the dependency tree that required an update to the Sdk package beyond v1. For now, v.3.0.3 is working great, just v3.0.9 did not deploy the SqlServerTypes DLLs.

[SeanFeldman]

Using <_FunctionsSkipCleanOutput>true is the right solution here. We unfortunately don't have this documented -- I'm going to try to figure out where we can put that.

@brettsam, it's still not documented. Not to bash anyone here but the Functions team has to understand that the service as good as its documentation. Having to skim through closed GitHub issues to find something that should be in the documentation is not a good customer experience.

[SeanFeldman]

Also, why the _FunctionsSkipCleanOutput? .NET Core already has a somewhat similar mechanism to trim down assemblies. It does work for self-contained deployments though.

[Saibamen]

Adding <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput> inside csproj fixed this in Azure, but now I can't start my functions locally via Visual Studio. Error:

Missing value for AzureWebJobsStorage in local.settings.json. This is required for all triggers other than httptrigger, kafkatrigger, rabbitmqtrigger, orchestrationTrigger, activityTrigger, entityTrigger. You can run 'func azure functionapp fetch-app-settings <functionAppName>', specify a connection string in local.settings.json, or use managed identity to authenticate.

But my local.settings.json is inside bin\Debug\net6.0 folder and has all required values.

Debugging and running Azure Functions locally via VS was working before adding <_FunctionsSkipCleanOutput>

EDIT: Looks like something was wrong with my local machine. It works now after restarting Windows

[jairoortizbizagi]

Proper workaround

It seems that the RemoveRuntimeDependencies task removes this assemblies (source). It doesn't run when the _FunctionsSkipCleanOutput variable is set to true. Adding the following line effectively disables this task and makes it work for 1.0.33 and later too.

  <PropertyGroup>
    <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>
  </PropertyGroup>

This work for me! Thank you!