NU1004 when restoring with lockedmode after nuget restore -ForceEvaluate returns successfully

[madelson]

Details about Problem

I recently spent a good deal of time debugging a strange NuGet error in one project which would only happen when restoring with locked mode on our CI server.

Ultimately, the problem had to do with the following invalid setup:

• 3 projects in one solution PackageA, PackageB, and PackageB.Tests. The Package projects are published as NuGet packages.
• PackageB has a project reference on PackageA, PackageB.Tests has a project reference on PackageB.
• PackageB.Tests also had an (invalid) <PackageReference> on the NuGet version of PackageA.

When I would run nuget restore -ForceEvaluate MySolution.sln, it would return successfully. PackageB.Test's package.lock.json file would have PackageA listed as a direct reference not as a project reference.

Under this setup, nuget restore MySolution.sln with <RestoreLockedMode>true</RestoreLockedModel> would fail with NU1004 complaining about ProjectB.Tests. This was quite difficult to debug because no error detail information was provided, and the problematic dependency was buried among many other non-problematic package dependencies. I ended up diagnosing by debugging NuGet.CommandLine itself.

Obviously, the setup in question was invalid and has been changed. However, I think there are 2 fixes that would have made this a better experience.

1. It would be great to have more error detail with NU1004. All it says is to run restore with --force-evaluate, which brings me to the second fix
2. I don't think restore with ForceEvaluate should have returned successfully in this case given that a later restore with locked mode was going to fail. It seems like a bug if a successful lock file generation can yield a lock file that isn't valid to be restored from.

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe): NuGet.exe

NuGet version (x.x.x.xxx): 5.4.0.6315, but also reproduced with latest master code as of 3/23/2020.

dotnet.exe --version (if appropriate): n/a

VS version (if appropriate): n/a

OS version (i.e. win10 v1607 (14393.321)):

Worked before? If so, with which NuGet version:

Detailed repro steps so we can see the same problem

See above.

[rrelyea]

Can you describe how to repro exactly? Perhaps even give us a small solution that repros the problem? I experimented with sdk style projects, and when restoring with: dotnet restore -- i got:

C:\Program Files\dotnet\sdk\3.1.200\NuGet.targets(124,5): error : An item with the same key has already been added. Key : (A, 1.0.0) [C:\temp\nu1004\nu1004.sln]

when doing the same with: nuget.exe restore, i got a slightly less useful:

An item with the same key has already been added

However, I don't know what kind of projects you have?

• csproj with packagesconfig?
• csproj with packagereference (i think this...but it would be great to have a small repro from you)
• sdk style csproj with packagereference
• other?

[madelson]

@rrelyea thanks for responding!

I don't know what kind of projects you have?

Sorry. We have sdk style csproj with packagereference.

when restoring with dotnet restore -- i

As I said in my issue, I am using nuget.exe for restore. We don't use dotnet yet because (a) we're still targeting .NET framework (in the process of converting) and (b) we have things set up so that each repo embeds a version of NuGet.exe that it is compatible with (a necessity for our build system given the various issues with SHA hashing and other cross-nuget-version consistency problems).

An item with the same key has already been added

Also seems like a bug, right? I'm not sure how the dotnet code path differs, but perhaps this is a different manifestation of the same bug?

Perhaps even give us a small solution that repros the problem?

Would you be willing to try your solution with nuget.exe -ForceEvaluate as mentioned in the issue and let me know if that repros it? If not I can try to reduce our solution down to something that I can share.

[madelson]

@rrelyea are you still waiting for anything from me?

[rrelyea]

@madelson - please try these steps. Seems like everything is working fine to me.

the first execution of nuget.exe with lockedmode, showed a message that sounded more verbose that you said:

NU1004: The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file.

Then running -forceevaluate afterwards fixed the lock files. then the last exection w/ lockedmode worked fine.

do i have something in the steps wrong? have things gotten better with the latest versions?

Thanks, Rob

if exist home9332 rd home9332 /s/q md home9332 cd home9332 dotnet new globaljson --sdk-version 3.1.200 dotnet new nugetconfig dotnet new classlib -o A dotnet pack a\a.csproj md packages copy a\bin\Debug\a.1.0.0.nupkg packages\ dotnet nuget add source packages\ -n packages9332 dotnet new classlib -o B dotnet add b\b.csproj reference a\a.csproj dotnet new classlib -o B.Tests dotnet add b.tests\b.tests.csproj reference b\b.csproj dotnet new sln dotnet sln add a\a.csproj dotnet sln add b\b.csproj dotnet sln add b.tests\b.tests.csproj c:\repos\nuget55.exe restore -uselockfile ECHO Hand add package ref into b.tests to A 1.0.0 pause REM Make sure nuget.exe is 5.5 pause nuget.exe restore -lockedmode nuget.exe restore -forceevaluate nuget.exe restore -lockedmode cd ..

[nkolev92]

Hey @madelson.

Any updates on @rrelyea's ask.

Thanks!

[madelson]

@nkolev92 these are the same steps I was following with my solution. I don't doubt that they work in the simple case. It seems to me that this problem must not surface under the minimal use-case I extracted out of our real-world more-complex use-case.

I unfortunately can't share the actual project that displayed the symptoms with you as it is in our private codebase, and I'm not sure I'll have time to recreated an equivalent example given that it involved dozens of internal packages.

[nkolev92]

@madelson Do you have any PrivateAssets on either of your project/package references?

Specifically I'm trying to figure out if you might be seeing: https://github.com/NuGet/Home/issues/8565

[madelson]

@nkolev92 do you mean dev dependency installs with <PrivateAssets>all</PrivateAssets> or similar? If so, then yes we have a few of those in the mix.

However, the project references are not marked this way.

[nkolev92]

Thanks for that update. When I look into #8565, I'll see if that bug is relevant with no ProjectReference private assets update this bug.

[Kiechlus]

We have simular issues using .Net 6. Our projects are of type Microsoft.NET.Sdk.Web. We use PackageReference with wildcard notations and ProjectReference. We do not use PrivateAssets. Running dotnet restore --locked-mode returns many NU1004. With --force-evaluate it works. However, we'd like to avoid --force-evaluate on the CI.

We tried many things like:

• Flush all nuget caches
• Run with force evaluate to have newest dependencies restored
• Have lock files in all projects of the solution or only in that with the main method
• Still running plain dotnet restore --locked-mode will always fail, both on windows and unix

We are not sure if ProjectReference is supported at all due to the following sentence in this blog post:

Currently, the lock file feature is only limited to PackageReference based projects.

I cannot share the project because it is full of private dependencies.

[nkolev92]

@Kiechlus project references are supported. When the blog is referring to PackageReference, it's saying that in contrast to packages.config.

NU1004 should be providing rich errors with the newer versions, can you maybe share those?

If you can find the minimal project that fails, it might be easy to work out a repro without your private packages.

[Kiechlus]

For me it looks like some problem with project references. That yellow is the main project which includes the others (marked with other colors).

If we add package.lock.json files into all projects within the solution, the number of errors increase.

[nkolev92]

@Kiechlus

I don't know if you'd be able to this, but maybe you can help us work out a more minimal repro?

The way, I'd approach it would be:

• Remove all but one reported project that's failing.
• Go to that project and start removing package references until it stops failing and just add the first one.

That could lead to basically something like: Project A -> Project B -> Package A with a floating version for example.

The repro doesn't have to be super granular as long as it's something we can recreate on our end.

[Kiechlus]

We'll try, may take a while

[BeyondEvil]

I'm seeing this issue as well.

Unfortunately I'm in the same situation and can't share much due to private repo.

[nkolev92]

@BeyondEvil Since the original issue was filled, the NU1004 error message should contain more concrete details about the failure. Maybe that gives you a hint as far what's going on.

For anyone willing to help with a repro, another idea is that the obj folder has a dgspec json file, which has the complete unresolved graph. You can try to anonymize that by find & replace all the private ids with the new ones.
That + the lock file should be enough to recreate things.

[BeyondEvil]

I wonder if the issue is that I'm creating the lock-files on a Mac (apple silicon/arm) but CI is running Ubuntu (linux-x64)?

I'm about to add a RuntimeIdentifiers property to see if that helps.

I did a dotnet restore --force-evaluate in CI and after that a git status and git diff, and the lock-files were changed. A bunch of both transitive and direct dependencies had been removed. I have no idea why...

[BeyondEvil]

Adding RuntimeIdentifiers did indeed cause updates to all the packages.lock.json-files, but it still fails on CI with the same issue.

[BeyondEvil]

MyProject.Infrastructure.csproj

```xml net8.0 ```

This is what packages.lock.json looks like on my machine:

packages.lock.json

This is what packages.lock.json looks like after being regenerated on CI (GH Actions Ubuntu-22.04) with dotnet restore --force-evaluation:

packages.lock.json

And this is the dgspec.json from the same CI run:

MyProject.Infrastructure.csproj.nuget.dgspec.json

ping @nkolev92 🙏 🙇

[nkolev92]

@BeyondEvil

The root difference is Twilio.AspNet.Core, and it looks like the 2nd run, the CI run is missing that. The nuget.dgspec.json confirms that as well.

I don't really have a clear indication as to why that direct reference is not used on the CI.

Can you check your repo and look whether there is any references to that package being removed? Binlog would tell me what's happening, but that effectively shares all of your projects structure and PRs.

[BeyondEvil]

@nkolev92

Why would that package not be used/be removed by dotnet restore in CI? 🤔

The only CI-specific setting we have is in the Directory.Build.props:

<Project>
    <PropertyGroup Condition="'$(GITHUB_ACTIONS)' == 'true'">
        <ContinuousIntegrationBuild>true</ContinuousIntegrationBuild>
    </PropertyGroup>
    <!-- <Target Name="DisableAnalyzersGlobally" BeforeTargets="CoreCompile">
        <PropertyGroup>
            <RunAnalyzers>false</RunAnalyzers>
            <RunAnalyzersDuringBuild>false</RunAnalyzersDuringBuild>
            <RunAnalyzersDuringLiveAnalysis>false</RunAnalyzersDuringLiveAnalysis>
            <EnableNETAnalyzers>false</EnableNETAnalyzers>
        </PropertyGroup>
    </Target>  -->
    <ItemGroup>
        <SourceRoot Include="$(MSBuildProjectDirectory)/" />
    </ItemGroup>
</Project>

Can you check your repo and look whether there is any references to that package being removed?

Any hints about what I'm looking for? A search reveals nothing afaict.

[nkolev92]

@BeyondEvil

I'd generate a binlog and open it in the msbuild binlog viewer https://msbuildlog.com

There you should see things like:

With the GetRestorePackageReferencesTask you should see something like:

If it's not happening and it's getting removed for example, you'd be able to see details like:

This was done by me adding, in the project file.

  <Target Name="RemoveTwilio" AfterTargets="CollectPackageReferences">
    <ItemGroup>
      <PackageReference Remove="Twilio.AspNet.Core" />
    </ItemGroup>
  </Target>