Build of the isolated worker doesn't respect package dependencies, nor nuget.config

[Kryptos-FR]

Summary

We are trying to make a custom binding that can use Google.Cloud.PubSub.V1. As part of the dependencies we are also referencing Microsoft.Extensions.Logging.Abstractions. However this fails at runtime with the following error:

A host error has occurred during startup operation '09f0e9cc-4aa8-4c13-a3d3-274686c7919b'.
System.Private.CoreLib: Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
Value cannot be null. (Parameter 'provider')
Host startup operation has been canceled

We have a couple of issues. For some of them, no workaround was found. The conclusion is that the isolated-process model is not even in beta and certainly not ready for production, despite Microsoft pushing for using it instead of the in-process model.

Nuget.config

We build our packages into a local Azure DevOps repository. To be able to restore properly we have added that source in a nuget.config file, such as:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <!--To inherit the global NuGet package sources remove the <clear/> line below -->
    <clear />
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
    <add key="Azure" value="https://our-domain.pkgs.visualstudio.com/_packaging/Azure/nuget/v3/index.json" />
  </packageSources>
</configuration>

Custom binding solution

Our custom binding is implemented in 3 projects in order to be used for both the in-process and the isolated-process execution models. We have:

CustomBinding.GooglePubSub.Core

This project implements the main controller that can communicate with Google PubSub. That project is supposed to be agnostic of the execution model. It has the following dependencies:

<Project Sdk="Microsoft.NET.Sdk">

  <ItemGroup>
    <PackageReference Include="Google.Cloud.PubSub.V1" Version="3.2.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
  </ItemGroup>

</Project>

CustomBinding.GooglePubSub

This project implements the in-process (web job) version. It has the following dependencies:

<Project Sdk="Microsoft.NET.Sdk">

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs" Version="3.0.39" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\Core\CustomBinding.GooglePubSub.Core.csproj" />
  </ItemGroup>

</Project>

CustomBinding.GooglePubSub.Isolated

And finally, the project for the isolated process has these dependencies:

<Project Sdk="Microsoft.NET.Sdk">

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Core" Version="1.16.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Abstractions" Version="1.3.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\Core\CustomBinding.GooglePubSub.Core.csproj" />
  </ItemGroup>

</Project>

In addition, it "loads" the dependencies thanks to the ExtensionInformation attribute:

using Microsoft.Azure.Functions.Worker.Extensions.Abstractions;
[assembly: ExtensionInformation("CustomBinding.GooglePubSub", "1.2.3.4")]

First issue: nuget.config is not followed

When building the consuming project, the build of the worker fails because it cannot find the package that is indirectly referenced thanks to the ExtensionInformation attribute. And this, despite having a correct nuget.config at the root directory. It seems that because it is built in a temp folder it doesn't bring along the content of the nuget.config file. That is the first issue that needs to be fixed by your team.

Workaround

We found a workaround for that particular issue. We add this to the consuming project in order to pretend to have a dependency so that it get restored first and find itself into the build maching .nuget cache.

<Project Sdk="Microsoft.NET.Sdk">
<!-- Required for building the worker -->
  <ItemGroup>
    <PackageReference Include="CustomBinding.GooglePubSub" Version="9.1.695.15404">
      <PrivateAssets>all</PrivateAssets>
      <ExcludeAssets>compile;runtime</ExcludeAssets>
    </PackageReference>
  </ItemGroup>
</Project>

Looking at the generated .csproj (in the Temp folder) during the build, we can inspect the project.assets.json file. And in there we, see that indeed the configFilePaths value is wrong as it is using the global nuget.configof the machine instead of our custom one.

"project": {
    "version": "1.0.0",
    "restore": {
    "configFilePaths": [
        "C:\\Users\\redacted\\AppData\\Roaming\\NuGet\\NuGet.Config",
        "C:\\Program Files (x86)\\NuGet\\Config\\Microsoft.VisualStudio.FallbackLocation.config",
        "C:\\Program Files (x86)\\NuGet\\Config\\Microsoft.VisualStudio.Offline.config"
      ],
      "originalTargetFrameworks": [
        "net6.0"
      ],
      "sources": {
        "C:\\Program Files\\dotnet\\library-packs": {},
        "https://api.nuget.org/v3/index.json": {}
      },

Second issue: transitive package dependencies are not respected

We believe the reason for the runtime crash is because the generated worker project doesn't have the right packages dependencies. While the generated .csprojlooks fine:

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>net6.0</TargetFramework>
        <LangVersion>preview</LangVersion>
        <Configuration>Release</Configuration>
        <AssemblyName>Microsoft.Azure.Functions.Worker.Extensions</AssemblyName>
        <RootNamespace>Microsoft.Azure.Functions.Worker.Extensions</RootNamespace>
        <MajorMinorProductVersion>1.0</MajorMinorProductVersion>
        <Version>$(MajorMinorProductVersion).0</Version>
        <AssemblyVersion>$(MajorMinorProductVersion).0.0</AssemblyVersion>
        <FileVersion>$(Version)</FileVersion>
        <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    </PropertyGroup>
    <ItemGroup>
        <PackageReference Include="Microsoft.NETCore.Targets" Version="3.0.0" PrivateAssets="all" />
        <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.2.0" />
        <PackageReference Include="CustomBinding.GooglePubSub" Version="9.1.695.15404" />
    </ItemGroup>
</Project>

When looking into the project.assets.json we can see that the right dependencies are supposed to be resolved.

      // ...
      "CustomBinding.GooglePubSub.Core/1.2.3.4": {
        "type": "package",
        "dependencies": {
          "Google.Cloud.PubSub.V1": "3.2.0",
          "Microsoft.Extensions.Logging.Abstractions": "8.0.0"
        },
        "compile": {
          "lib/net6.0/CustomBinding.GooglePubSub.Core.dll": {}
        },
        "runtime": {
          "lib/net6.0/CustomBinding.GooglePubSub.Core.dll": {}
        }
      },
      // ...
      "Microsoft.Extensions.Logging.Abstractions/8.0.0": {
        "type": "package",
        "dependencies": {
          "Microsoft.Extensions.DependencyInjection.Abstractions": "8.0.0"
        },
        "compile": {
          "lib/net6.0/Microsoft.Extensions.Logging.Abstractions.dll": {
            "related": ".xml"
          }
        },
        "runtime": {
          "lib/net6.0/Microsoft.Extensions.Logging.Abstractions.dll": {
            "related": ".xml"
          }
        },
        "build": {
          "buildTransitive/net6.0/Microsoft.Extensions.Logging.Abstractions.targets": {}
        }
      },

Yet, when we go back to the bin folder of the consuming project, we don't see Microsoft.Extensions.Logging.Abstractions.dll present under the bin\Debug\net8.0\.azurefunctions, and because of that it fails at runtime.

Workaround

We try explicitly referencing Microsoft.Extensions.Logging.Abstractions version 8.0.0 in all projects involved. Yet, it doesn't solve anything. So there is currently no workaround.

[fabiocav]

Thanks for opening this. Let me provide some information on the two issues you've flagged.

Issue 1 (Nuget.config outside of the probing path):

This is a known issue and being addressed with SDK updates actively being worked on. This is currently in PR (#1946) and tracked by #1863 . The workaround at the moment, in addition to what you've done, is to define a global Nuget.config, but this fix will be released soon.

Issue 2 (restricted host dependencies):

To provide some clarity, this is not an issue with transitive dependencies not being pulled in. This is an in-proc limitation, not a .NET Isolated issue.

The extensions build output is what the host runs (which is what generates the message you've shared), and the host does not support extensions using .NET 8 dependencies. When writing WebJobs custom bindings for Functions, this limitation needs to be observed and will apply to your in-proc applications as well. Your worker extensions do not have this limitation, and you're free to use any target there.

The project has the correct set of dependencies, but those restricted assemblies (provided by the host/runtime environment) are automatically removed at build time as they are unified at runtime. If you inspect the build output of your in-proc application, you'll notice that this assembly, unless explicitly preserved, will be removed there as well.

In summary:

Issue 1: Valid, fix in progress. Tracked by #1863 . Some workarounds available. Issue 2: A limitation of WebJobs extensions for Functions and consistent with the in-proc experience.

I want to better understand your statement about the Isolated model quality. Are there other issues also impacting your experiences?

[Kryptos-FR]

@fabiocav Thanks for replying. I don't understand your second point about .NET 8 dependencies, though. Microsoft.Extensions.Logging.Abstractions/8.0.0 is supposed to work for both .NET 6 and .NET 8 (at least according to the nuget package).

In fact I have the exact same loading error when all projects target .NET 6 (including the final worker project). In my previous message, all other libraries (esp. the custom binding) were also only targeting .NET 6. I changed the target for the worker to also be .NET 6 and the same error appears.

However, if I downgrade to use Microsoft.Extensions.Logging.Abstractions/6.0.0 instead, the error is gone.

Looking deeper into what are the difference between the two package versions, it appears that version 6.0 doesn't have a .targets file in the buildTransitive folder (or more specifically it has the special _._ one). However, the 8.0 version has one for .net 6 specifically. Could that be the issue?

[Kryptos-FR]

In addition, all the points you mentioned (esp. about this "When writing WebJobs custom bindings for Functions, this limitation needs to be observed and will apply to your in-proc applications as well") is documented nowhere. How are we suppose to get that?

In general, there is almost zero documentation on how the isolated process work, the architecture between the host process and the worker process, or how to write custom bindings that work for both models. I had to use classes that are all marked as obsolete with the message "Not ready for public consumption.". This includes FluentBindingRule<TAttribute> (used by BindToInput(), BindingFactory.GetTriggerBinding() and the related ITriggerBindingStrategy<TMessage, TTriggerValue>. That is not a great developer experience, when you have no idea if you are using the right code or if that code will still be valid in a subsequent release because everything was marked as "not-ready".

[Kryptos-FR]

I commented on #2092 by linking to a repository illustrating some usage. I guess it can also be linked here as it demonstrates the nuget.config issue as well.

https://github.com/Kryptos-FR/azure-functions-dotnet-worker-2092

[Kryptos-FR]

Hello, any update ? (as well as #2092)

[simonrivas]

Some update would definitely be appreciated!