SDK: Address no-network inner-build restore

[jviau]

The functions SDK for dotnet isolated performs an inner-restore/build/publish to discover and collect WebJobs extensions required to support worker extensions. This works fine for most cases; however, it is a headache when using secure/compliant builds. These secure builds often have network access only during restore phase and our design requires it during build phase. The end result is build failures due to no network access for the inner-build.

After having synced with the dotnet, msbuild, and nuget teams it is not feasible for us to participate in the restore target. We do not have enough information at restore time and rely on targets that are a part of build. The best option we came up with is to provide a tool, probably as part of func cli, which can generate this inner WorkerExtensions.csproj and have it checked in as part of source control. Customers can then include this project as part of their restore phase. This will be a tool only for customers with this strict network requirements.

Work Needed

1. Updated SDK build targets to recognize and use a pre-existing WorkerExtensions.csproj
2. Create a tool to generate said WorkerExtensions.csproj and to point the function worker csproj to this generate project.

Workaround

To workaround this issue today, you can avoid requiring network for restore by pre-restoring the necessary packages ahead of time.

1. Build function app locally
2. Copy WorkerExtension.csproj into your solution and ensure it is part of your builds restore phase
   • Can find the file path from the build logs. Or if using 1.17.3-preview* SDK, the project will be in intermediate output directory (obj) of your function app
   • If using central package management, replace Version with VersionOverride

The goal is to just have those set of PackageReference from WorkerExtension.csproj be restored ahead of time, so those packages are cached locally. This way the inner-build's restore will be 100% cache hits and never access the network.

[morrisonbrett]

Can you please post an example WorkerExtensions.csproj file here that I can add to my solution to work around this issue?

[jviau]

@morrisonbrett that won't work as we have to update our SDK build targets to recognize and use that pre-existing WorkerExtension.csproj

[morrisonbrett]

I'm just looking for a workaround until this is fixed properly. I tried adding a project named WorkerExtension.csproj to my solution, added all the "InProcess" AzureFunctions Nuget packages and even a Unit Test but it didn't work. My main projects keep pulling in their own WorkerExtension.csproj. Not only that, I can't seem to exclude it from my dotnet test and coverage run, so I've had to bump my coverage threshold down to accommodate it.

[jviau]

@morrisonbrett yes the function app will keep generating using its own WorkerExtension.csproj. There is no logic in the targets currently to detect a pre-existing one.

WorkerExtensionStartupCodeExecutor is not related to this project. That is a source-generated file, there are ways to exclude specific files/types from code-coverage. Not sure why this generated file is not automatically excluded.

https://learn.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2022

[morrisonbrett]

Thanks @jviau. I just tried and it did not work. I already have a coverlet.runsettings file in my project, so I added the <CodeCoverage><ModulePaths> section to it per the article referenced above. Same result, the coverage run is still including it.

Updated .runsettings file:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat Code Coverage">
        <Configuration>
          <Format>cobertura,opencover,lcov</Format>          
          <Exclude>[xunit*]\*</Exclude>
          <SingleHit>false</SingleHit>
          <UseSourceLink>true</UseSourceLink>
          <IncludeTestAssembly>false</IncludeTestAssembly>
          <SkipAutoProps>true</SkipAutoProps>
          <CodeCoverage>
            <ModulePaths>
              <Exclude>
                <ModulePath>*.WorkerExtensionStartupCodeExecutor</ModulePath>
              </Exclude>
            </ModulePaths>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

[EdwinOtten]

Hi there! I ran into problems because of this "inner-build restore", I'm sharing this info because it might save other people some time debugging.

The issue occured when a team member added the Microsoft.Azure.Functions.Worker.Extensions.DurableTask package to our project. Everything worked on their machine, but when I pulled their changes and attempted to build our project, it tried to restore WorkerExtensions.csproj. Which failed to resolve Microsoft.Azure.WebJobs.Extensions.DurableTask . Resulting in the build output below:

Determining projects to restore...
C:\Users\MyUser\AppData\Local\Temp\gt02a21z.5m5\WorkerExtensions.csproj : error NU1100: Unable to resolve 'Microsoft.Azure.WebJobs.Extensions.DurableTask (>= 2.13.0)' for 'net6.0'.
Failed to restore C:\Users\MyUser\AppData\Local\Temp\gt02a21z.5m5\WorkerExtensions.csproj (in 1.24 sec).
Done building project "WorkerExtensions.csproj" -- FAILED.
========== Build: 3 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

My workaround was:

1. Open the WorkerExtensions.csproj mentioned in the build output in Visual Studio
2. Right-click on the project and open the NuGet Package Manager
3. Look at the Package source, in my case there was none (this was the culprit)
4. Add NuGet v3 package source: https://api.nuget.org/v3/index.json
5. Right-click on the solution and click Restore packages
6. Now you should be able to build your project

So the problem for me was that WorkerExtensions.csproj in my local AppData didn't use the NuGet v3 package feed. I was under the impression that Visual Studio would be configured to use that by one by default. But apparently not? I'm curious why I experienced this issue, but my team members using Rider did not.

[jviau]

@EdwinOtten the WorkerExtensions.csproj will not inherit Visual Studios restore settings. VS restore settings only apply to projects listed directly in the solution file, this project is not part of that.

Do you have a global nuget.config? Check %appdata%\NuGet\NuGet.Config and %ProgramFiles(x86)%\NuGet\Config. This might be removing nuget v3 feed.

[EdwinOtten]

@jviau thanks for clarifying! 👍 I checked both:

• %appdata%\NuGet\NuGet.Config adds 2 sources (Microsoft Visual Studio Offline Packages and NuGet V3). The latter wasn't there when I first experienced the issue, but was added when I added it via the Package Manager GUI in VS2022.
• %ProgramFiles(x86)%\NuGet\Config adds 1 source (Microsoft Visual Studio Offline Packages)

FYI: I'm running Visual Studio 2022 Professional.

[alayala-MSFT]

Hi @jviau. In this thread you mention that the workaround in the meantime is to create a dummy project as mentioned here.

One piece of data I'd like to add is that this can workaround can still cause issues with staying compliant if the organization requires upgrades of newer packages that the WorkerExtensions.csproj project references.

For example, the below is a WorkerExtensions.csproj that is generated when using the latest (5.2.0) Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues.

This will generate <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage.Queues" Version="5.1.3" /> below which transitively relies on Azure.Identity >= 1.7.0.

The restore will take this version of Azure.Identity which is known to have a high severity vulnerability: https://www.nuget.org/packages/Azure.Identity/1.7.0

However, we cannot manually override this package for WorkerExtensions.csproj to use a newer version, so we're forced to use a version with a vulnerability.

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>net6.0</TargetFramework>
        <LangVersion>preview</LangVersion>
        <Configuration>Release</Configuration>
        <AssemblyName>Microsoft.Azure.Functions.Worker.Extensions</AssemblyName>
        <RootNamespace>Microsoft.Azure.Functions.Worker.Extensions</RootNamespace>
        <MajorMinorProductVersion>1.0</MajorMinorProductVersion>
        <Version>$(MajorMinorProductVersion).0</Version>
        <AssemblyVersion>$(MajorMinorProductVersion).0.0</AssemblyVersion>
        <FileVersion>$(Version)</FileVersion>
        <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    </PropertyGroup>
    <ItemGroup>
        <PackageReference Include="Microsoft.NETCore.Targets" Version="3.0.0" PrivateAssets="all" />
        <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.2.0" />
        <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage.Queues" Version="5.1.3" />

    </ItemGroup>
</Project>

[jviau]

@alayala-MSFT we should be publishing a new version of the queue extension to address that dependency. The other issue you opened #2147 will be used to track that work.

I also have this issue: #2221 to propose exposing a mechanism to allow for overriding this.

[mazuschlag]

@alayala-MSFT We had a similar issue with the WorkerExtensions.csproj being published in the root build directory with other build outputs, which caused compliance to get angry as that project had a transitive dependency on Azure.Identity 1.5.0. This was fixed by upgrading our Microsoft.Azure.Functions.Worker.Sdk version from 1.17.0 to 1.17.2. This caused the WorkerExtensions.csproj to be created in a temp directory rather than in our build outputs, satisfying compliance.

[alayala-MSFT]

@jviau I see #2221 was closed, so there will be no override. Is there an SLA for updating and releasing updates to extensions for vulnerable dependent packages?

[alayala-MSFT]

@mazuschlag Nice. So in this case, you didn't need to use the workaround solution mentioned above? Also what version of the Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues extension are you using?

[Tarun047]

Hey @jviau How about the projects using centralized package management? It seems that the generated WorkerExtensions.csproj for a simple use case with just a timer trigger outputs something like this

        <PackageReference Include="Microsoft.NETCore.Targets" Version="3.0.0" PrivateAssets="all" />
        <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.2.0" /> 

This breaks the CPVM for us, because we can neither control nor override this file. It would be great if it could detect if the project has centralized package management enabled and output something like this

        <PackageReference Include="Microsoft.NETCore.Targets" VersionOverride="3.0.0" PrivateAssets="all" />
        <PackageReference Include="Microsoft.NET.Sdk.Functions" VersionOverride="4.2.0" /> 

[jviau]

@Tarun047 I will look into that again - thought I had avoided that by suppressing Directory.Build.props and Directory.Build.targets from being imported in this project, so centralized package management wouldn't get enabled. Could you capture a binlog so I can see how central package management is being enabled for the generated csproj?

[lps2015]

Hello @jviau, I'm facing the same issue. Is the workaround solution (dummy WorkerExtension.csproj) ready now? Have you added logic to detect pre-existing WorkerExtension project?

[Tarun047]

Hey @lps2015 I am not facing the issue post upgrading to 1.17.3-preview2

[jviau]

@lps2015 have not looked into that yet. No ETA on when I will look into it either. The best workaround today is to maintain a separate project with identical packages to ensure they get cached while you have network access.

@Tarun047 - glad that fixed your issue. I had looked into some other similarly reported issues and the problem was: (1) using pre- 1.17.0 package (or 1.17.2) and (2) a custom CI system which placed the temp directory within scope of a Directory.Build.props, which brought in central package management.

This is fixed by using 1.17.3-* as importing of Directory.Build.* in WorkerExtension.csproj is supressed.

[lps2015]

Hey @jviau , I want to confirm whether the best workaround you proposed is currently available, or it’s still a proposal awaiting implementation. If it is available, which is the SDK version? I see your reply above that dummy WorkerExtensions.csproj does not work because you haven't added pre-existing project detecting logic.

[jviau]

@lps2015 the workaround does not require pre-existing project detection. It operates by having the dummy project hydrate the local nuget cache with the necessary packages. So when the generated WorkerExtension.csproj restores, it is purely cache-hits and never makes any network calls to your configured nuget sources.

[jinjli-msft]

@jviau Is there any solutions already ? Can you please share a sample project link how to fix it?

[jviau]

@jinjli-msft when using the latest SDK (1.17.3-preview2), you can build the app locally and then go to your intermediate output directory. You will see a WorkerExtension.csproj - copy the contents of that project and check it into your repository. Ensure it is restored (no need to build, but fine if you do) as part of your initial restore.

[alayala-MSFT]

Hi @jviau. I noticed that this WorkerExtensions.csproj targets .NET 6.0. Does this mean that our dummy project also requires targeting this framework rather than say .NET 8.0?

During a cloud build failure, i noticed in the nuget dgspec file, references to these packages for the .NET 6.0 runtime SDKs:

For context, our dummy project targets .NET 8.0 and so does our actual Azure Functions project

[alayala-MSFT]

@jviau Additionally, if matching the target framework is required here, are there any plans to move this intermediate WorkerExtensions.csproj to .NET 8.0 as that is the new LTS version of .NET?

[jinjli-msft]

Same issue with @alayala-MSFT . Since net6.0 will no longer be supported in December 24, our goal is also net8.0.

[jviau]

WorkerExtensions.csproj should target .NET6 - it targets the same TFM the host runtime uses and can be different than your function apps TFM (they are separate processes). We are working on upgrading that to .NET8 and that will be available in a later SDK.

[reynaldoburgos]

We are in the process of upgrading our project to Net8, and are using the SDK version 1.17.3-preview2; however, we're still experiencing issues where "error NU1008: Projects that use central package version management should not define the version on the PackageReference items but on the PackageVersion". We're using CPM, and need this autogenerated csproj to either not include strict versioning or to incorporate VersionOverride or set this variable to not block projects that use CPM.

<ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally>

[jviau]

We are in the process of upgrading our project to Net8, and are using the SDK version 1.17.3-preview2; however, we're still experiencing issues where "error NU1008: Projects that use central package version management should not define the version on the PackageReference items but on the PackageVersion". We're using CPM, and need this autogenerated csproj to either not include strict versioning or to incorporate VersionOverride or set this variable to not block projects that use CPM.

<ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally>

We discussed this offline, but this error was for when making the manual checked-in copy of WorkerExtensions.csproj. For this copy, it is okay to use VersionOverride. The csproj does not need to be identical with the generated csproj. It only needs to restore the same packages & versions to disk (and their transient dependencies). How it accomplishes that is flexible to what the individual repository needs.

[Porges]

Even aside from the no-network scenario (which is also relevant to my team), there's also the problem that we cannot upgrade package versions. With the current SDK (1.17.4) in our project it is pulling in 5 packages with CVEs registered against them, and there's no way to replace these.

Edit: there's a new bug about this here; #2604

[bondarenkod]

Could you test my workaround? You can find it at this link https://gist.github.com/bondarenkod/4b6a9bc7224e6c8ab09ae113b68de1f5

[santiagogun]

Is there any news on this topic?? I was able to build this around two weeks ago but now it seems that I am getting the infamous error of not being able to access my private feed

[bondarenkod]

Is there any news on this topic?? I was able to build this around two weeks ago but now it seems that I am getting the infamous error of not being able to access my private feed

I have a workaround. Check my post above.

[ntyrrell]

@bondarenkod I gave your workaround a go. Unfortunately, from what I can see, the NugetAuthenticate step doesn't specify the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS environment variable, but rather VSS_NUGET_ACCESSTOKEN.

We could modify the script to explicitly use this value, or add an additional step where we explicitly define this environment variable, but if that's the case, it might be worth including that in your example?

Just realised that we cannot actually set the environment variable directly because of limitations around the variables that we can define: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#variable-naming-restrictions

[eureka-gh]

For those who are stuck on forced internal feed, the most recent update from MFST CFS team is to use the latest version of Microsoft.Azure.Functions.Worker.Sdk as of today dotnet add package Microsoft.Azure.Functions.Worker.Sdk --version 1.17.4.

[Ashwad1996]

I had used this workaround and the issue was fixed but for the last 2 weeks, I again started getting the same error. Can someone please help?