Restore should fail when a non-existent fallback folder is provided

[KirillOsenkov]

MSBuild 15.7.179.6572

MSBuild of an SDK-style project with PackageReference fails because the generated .csproj.nuget.g.props file contains this line:

<NuGetPackageFolders Condition=" '$(NuGetPackageFolders)' == '' ">C:\Users\kirillo\.nuget\packages\;C:\Program Files (x86)\Microsoft SDKs\NuGetPackagesFallback\</NuGetPackageFolders>

The NuGetPackagesFallback folder doesn't exist on my machine, causing NuGet to fail the build with the exception:

C:\Program Files\dotnet\sdk\2.1.201\Sdks\Microsoft.NET.Sdk\build\Microsoft.PackageDependencyResolution.targets(172,5): error MSB4018: The "ResolvePackageDependencies" task failed unexpectedly.
NuGet.Packaging.Core.PackagingException: Unable to find fallback package folder 'C:\Program Files (x86)\Microsoft SDKs\NuGetPackagesFallback\'.
   at NuGet.Packaging.FallbackPackagePathResolver..ctor(String userPackageFolder, IEnumerable`1 fallbackPackageFolders)
   at Microsoft.NET.Build.Tasks.NuGetPackageResolver.CreateResolver(LockFile lockFile, String projectPath)
   at Microsoft.NET.Build.Tasks.ResolvePackageDependencies.get_PackageResolver()
   at Microsoft.NET.Build.Tasks.ResolvePackageDependencies.ResolvePackagePath(LockFileLibrary package)
   at Microsoft.NET.Build.Tasks.ResolvePackageDependencies.GetPackageAndFileDefinitions()
   at Microsoft.NET.Build.Tasks.ResolvePackageDependencies.ExecuteCore()
   at Microsoft.NET.Build.Tasks.TaskBase.Execute()
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext() [C:\monodevelop\main\external\RefactoringEssentials\RefactoringEssentials\RefactoringEssentials.csproj]

None of the nuget.config files on my machine contain anything related to the fallback folder.

I don't have any environment variables set that start with NuGet*.

My VS offline package cache is a similar folder, but without the fallback:

Why does NuGet decide to generate that non-existing folder into my .g.props files?

Where does that value come from?

[KirillOsenkov]

@nkolev92

[KirillOsenkov]

.binlog is available at \\kirillo7\public\Bugs\NuGetHomeIssue7012.binlog

[KirillOsenkov]

This might be related: https://developercommunity.visualstudio.com/content/problem/232735/the-folder-cprogram-files-x86microsoft-sdksnugetpa.html

[nkolev92]

Thanks @KirillOsenkov I can access the log.

I'll take a look when I get back to my desk.

The fallback folder is passed in to NuGet...but I guess we might not be validating whether it exists like we do with sources.

[jainaashish]

C:\Program Files (x86)\Microsoft SDKs\NuGetPackagesFallback\ is passed through dotnet sdk to NuGet and I'd expect this folder to be always present. Did you explicitly deleted this folder? and what's the dotnet sdk version (dotnet.exe --info)?

[KirillOsenkov]

Nope, I didn't delete the folder.

Where in the SDK is this set? The SDK folks have told me it's not them, it's NuGet.

C:\>dotnet --info
.NET Command Line Tools (2.1.300-preview1-008174)

Product Information:
 Version:            2.1.300-preview1-008174
 Commit SHA-1 hash:  b8df89a54f

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.300-preview1-008174\

Microsoft .NET Core Shared Framework Host

  Version  : 2.1.0-preview1-26216-03
  Build    : f2c3216183d20416568a4bbf5bb7d153e826f153

[nkolev92]

This is not the dotnet sdk, but UWP normally.

C:\Program Files (x86)\Microsoft SDKs\NuGetPackagesFallback\

The files section in the binary logger is equivalent to pp right? @KirillOsenkov Not sure why we can't find anyone passing it there.

[jainaashish]

This is a fallback folder similar to the MS VS offline packages, coming from C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.Fallback.config. This was added by UWP team somewhere in 15.5 or 15.6 which broke the world when they started having non-UWP packages in that fallback folder which were not extracted correctly. So they fixed it in 15.7 by removing this from fallback folders list and renaming NuGetPackagesFallback to UWPNuGetPackages. But seems like there is issue with clearing this file as well.

So, for now to workaround this, you can delete this config file C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.Fallback.config or just create an empty folder C:\Program Files (x86)\Microsoft SDKs\NuGetPackagesFallback\

[KirillOsenkov]

Yes, I have C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config with this contents:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="Microsoft Visual Studio Offline Packages" value="C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\"/>
  </packageSources>
</configuration>

The workaround to create an empty folder works, but I'm a trained professional. How do we expect regular users to deal with this failure?

Who would be a good person on the UWP team to investigate this further? If I wanted to file a bug about this, where could I file it? Thanks!

[tmeckel]

I had the same issue and resolved it by deleting the obj and bin directories. An analysis of the root cause (in my cause though) showed that the MSBuild environment creates obj\project.assets.json when a nuget restore is done. Inside this file references to the fallback folders, which were present at the time of the restore, are stored. If at a later time not all folders can be found, the build will fail. From my point of view not an ideal solution.

[jmoralesv]

Hi all, Today I came across this same issue while trying to recompile a .NET Core 2.1 solution I moved from an old hard disk to a new SSD for my development machine.

The old VS 2019 installation in the hard disk had a folder in this path: C:\Microsoft\Xamarin\NuGet\, which didn't exist on my installation with VS 2019. I saw that the obj\project.assets.json file still references the non existing folder path, although I recompiled my solution several times. I though the files in the obj folder would be regenerated every time I trigger a rebuild solution command in VS 2019, but that did not happen, it still uses the existing obj\project.assets.json file.

As soon as a removed the obj folder and its contents and recompile the solution, and after I created the the offending folder C:\Microsoft\Xamarin\NuGet\, which is referenced by this config file C:\Program Files (x86)\NuGet\Config\Xamarin.Offline.config, then the error went away and the solution compiled successfully, the Nuget packages were restored with no issues:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <fallbackPackageFolders>
    <add key="Xamarin Offline Packages" value="C:\Microsoft\Xamarin\NuGet\"/>
  </fallbackPackageFolders>
</configuration>

As I didn't have to modify the Xamarin.Offline.config, I guess I found a workaround for this issue. However, I think Visual Studio 2019 should regenerated the obj folder contents every time it sees a change on the solution files and folders.

Something about this issue is mentioned in this thread, although no solution was provided at that time: https://developercommunity.visualstudio.com/content/problem/479405/32-bit-x86-vs2019-installations-looking-for-xamari.html

[nkolev92]

The folder specified by C:\Program Files (x86)\NuGet\Config\Xamarin.Offline.config still didn't exist, so I'm not sure why do you expect that to change. It's the creation of the folder that fixed it, not the assets file regeneration, or am I missing something?

[jmoralesv]

The folder specified by C:\Program Files (x86)\NuGet\Config\Xamarin.Offline.config still didn't exist, so I'm not sure why do you expect that to change. It's the creation of the folder that fixed it, not the assets file regeneration, or am I missing something?

No, you're not missing something. However, I wonder why I have to create that folder manually. I think Visual Studio, the .NET Core SDK or something else is actually missing something.

[nkolev92]

The idea is whichever tool added the config should create the folder as well. NuGet does not really control this work. So there definitely is a bug on some part of the toolset. But for all we know it could've been fixed by now.

@rrelyea We have marked this as external. There's not much NuGet can do, other than add validation to fail at restore time if the fallback folder is not present. However, the current error message is very descriptive and might not need changes. Thoughts?

[nkolev92]

Closing this as non actionable from NuGet.

[KUTlime]

Thoughts?

@nkolev92 How about fix it? I came across this problem when I deleted the C:\Microsoft\Xamarin\NuGet\. Surprise, surprise,... The bug is still there. How about find a correct repo (a.k.a project) which is doing this problem and get rid of it?

I'm building a .NET Framework 4.7.2 application, completely unrelated to Xamarin a still I have these messages all over my repos.

[stephajn]

Fun bit of news, this is back in dotnet sdk 6.0. Yayyyy.....

[KirillOsenkov]

@nkolev92 I can confirm this again became a problem. Could you please reactivate this issue?

I'm seeing a hard crash in the ResolvePackageAssets task:

"C:\temp\console\console.csproj" (default target) (1) ->
(ResolvePackageAssets target) ->
  C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,
5): error MSB4018: The "ResolvePackageAssets" task failed unexpectedly. [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018: This is an unhandled exception from a task -- PLEASE OPEN A BUG AGAINST THE TASK OWNER. [C:\temp\conso
le\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018: NuGet.Packaging.Core.PackagingException: Unable to find fallback package folder 'C:\Program Files (x86
)\Microsoft Visual Studio\Shared\NuGetPackages'. [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at NuGet.Packaging.FallbackPackagePathResolver..ctor(String userPackageFolder, IEnumerable`1 fallba
ckPackageFolders) [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.NET.Build.Tasks.NuGetPackageResolver.CreateResolver(IEnumerable`1 packageFolders) [C:\
temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheWriter..ctor(ResolvePackageAssets task) [C:\
temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheReader.CreateReaderFromDisk(ResolvePackageAs
sets task, Byte[] settingsHash) [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheReader..ctor(ResolvePackageAssets task) [C:\
temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ReadItemGroups() [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ExecuteCore() [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() i
n C:\msbuild\src\Build\BackEnd\TaskExecutionHost\TaskExecutionHost.cs:line 564 [C:\temp\console\console.csproj]
C:\Program Files\dotnet\sdk\6.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(267,5)
: error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext() in C:\msbuild\src\
Build\BackEnd\Components\RequestBuilder\TaskBuilder.cs:line 825 [C:\temp\console\console.csproj]

Instead of crashing and printing the callstack we should emit a proper MSBuild error. Repro: manually add a non-existing fallback folder to the top of your project.assets.json:

{
  "version": 3,
  "targets": {
    "net5.0": {}
  },
  "libraries": {},
  "projectFileDependencyGroups": {
    "net5.0": []
  },
  "packageFolders": {
    "C:\\Users\\kirill\\.nuget\\packages\\": {},
    "C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\NuGetPackages": {},
    "C:\\Program Files\\dotnet\\sdk\\NuGetFallbackFolder": {}
  },

This is where it throws: https://github.com/NuGet/NuGet.Client/blob/b9657fd4665055429d15285c2c22ec081e4600b4/src/NuGet.Core/NuGet.Packaging/FallbackPackagePathResolver.cs#L60

This is the stack:

>   NuGet.Packaging.dll!NuGet.Packaging.FallbackPackagePathResolver.FallbackPackagePathResolver(string userPackageFolder, System.Collections.Generic.IEnumerable<string> fallbackPackageFolders) Line 60    C#
    Microsoft.NET.Build.Tasks.dll!Microsoft.NET.Build.Tasks.NuGetPackageResolver.NuGetPackageResolver(string userPackageFolder, System.Collections.Generic.IEnumerable<string> fallbackPackageFolders) Line 22  C#
    Microsoft.NET.Build.Tasks.dll!Microsoft.NET.Build.Tasks.NuGetPackageResolver.CreateResolver(System.Collections.Generic.IEnumerable<string> packageFolders) Line 70  C#
    Microsoft.NET.Build.Tasks.dll!Microsoft.NET.Build.Tasks.GetPackageDirectory.ExecuteCore() Line 54   C#
    Microsoft.NET.Build.Tasks.dll!Microsoft.NET.Build.Tasks.TaskBase.Execute() Line 35  C#
    Microsoft.Build.dll!Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() Line 564 C#

[KirillOsenkov]

The non-existing path that doesn't exist on my machine is: C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\NuGetPackages

[KirillOsenkov]

Correction, the actual task that crashes is:

[nkolev92]

Is the ask for NuGet to fail restore when this happens?

[KirillOsenkov]

Two things:

1. yes, if you detect a non-existing fallback folder, you should fail restore.
2. However this bug can happen when a restore was successful, then the folder is deleted, and then we're building the project where the non-existing folder is left over in the project.assets.json. In this case (and that's what I'm mostly asking) we should log an error instead of the task throwing an exception that propagates to MSBuild. MSBuild should never see exceptions from tasks, tasks should be logging errors instead.

Right now the task throws, and the user sees the crash stack with: This is an unhandled exception from a task -- PLEASE OPEN A BUG AGAINST THE TASK OWNER.

This is effectively what I'm doing now - opening a bug against the task owner ;)