rrelyea
commented
5 years ago @kartheekp-ms - can you please investigate. @nkolev92 & @cristinamanum have experience with this codepath.
Open tebeco opened 5 years ago
rrelyea
commented
5 years ago @kartheekp-ms - can you please investigate. @nkolev92 & @cristinamanum have experience with this codepath.
rrelyea
commented
5 years ago would consider for 5.3 if this turns into a fix we should take.
nkolev92
commented
5 years ago I'm not sure I understand how values such as
"runtime.opensuse.13.2-x64.runtime.native.System.Security.Cryptography.OpenSsl": {
"type": "Transitive",
"resolved": "4.3.0",
"contentHash": "zWLOQ77Y4FV/6Vw2g+FYzprbQX5/xKvjoCLe4L9159Aw1bSboQoJ1KRZFNdexDooWRAIsLSdE0ZokkrVkwN8Yw=="
},are generated :(.
I checked for the package from NuGet.org I see:
"runtime.opensuse.13.2-x64.runtime.native.System.Security.Cryptography.OpenSsl": {
"type": "Transitive",
"resolved": "4.3.0",
"contentHash": "b3pthNgxxFcD+Pc0WSEoC0+md3MyhRS6aCEeenvNE3Fdw1HyJ18ZhRFVJJzIeR/O/jpxPboB805Ho0T3Ul7w8A=="
},Can you please delete your global packages folder and set
<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>
tebeco
commented
5 years ago Can you please delete your global packages folder and set
<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>
Yes of course. I think i already attempted similar on another repo so i probably made a typo Can you please confirm few question ?
dotnet nuget locals --clear all does do the tricks ?
=> should we add an option in the CLI for that before 3.0 lands ?
dotnet nuget locals --delete xxxWindows: %userprofile%.nuget\packages
tebeco
commented
5 years ago I'm not sure I understand how values such as .... are generated :(. I checked for the package from NuGet.org I see:
i can make a tool crawling my box if you tell me what / where / how to look at ^^
nkolev92
commented
5 years ago i suppose i only do that once on my laptop and never commit that to the CI ?
I'd disable the fallback folder period. It's going away in 3.0 anyways, and for your purposes it can mostly cause trouble depending on the machine.
i also assumes it is just to regenerate brand new Hash in lockfile once
The idea is that NuGet generates the content hash. There are 2 types of folders from which packages can be consumed in PR. The global packages folder and fallback folder. Fallback folders are not managed by the client so if there's an issue with those it's difficult to detect it :(
Do i need to ask all the team to do the same ?
I guess depending on the outcome of the investigation. Once your problems are addressed, we'll know more.
The dotnet nuget locals --clear all does do the tricks ? => should we add an option in the CLI for that before 3.0 lands ? dotnet nuget locals --delete xxx
Clear all will delete all the folders NuGet generates, so it does the trick. Removing only the global packages folder should be sufficient.
Assuming i did not override any env var where is the default location of that folder ? Windows: %userprofile%.nuget\packages Yup. That's correct.
should i delete the content only ? or the folder too and let nuget re-create it ? Either works :)
As far as the scripting part goes. Find all the nupkgs with a specific name (one of the offending packages) and then look at the .nupkg.metadata folder in the same directory. That value is the content hash.
Once you find the one that's different, we can figure what's wrong, whether it's a content hash generation bug or maybe just a different package somehow in that folder.
tebeco
commented
5 years ago I'd disable the fallback folder period
Ok, so i do commit in the DisableImplicitNuGetFallbackFolder in a props file of the repo ?
It's going away in 3.0 anyways, and for your purposes it can mostly cause trouble depending on the machine.
Can you be more specific on what's going away here ^^ ?
Clear all will delete all the folders NuGet generates, so it does the trick. Removing only the global packages folder should be sufficient.
Well i already did a dotnet nuget locals -c all, so i'm gonna try both this time + the props
tebeco
commented
5 years ago not sure where to look at to be honest ^^
C:\Users\NAME\.nuget>"C:\Program Files\Git\usr\bin\find.exe" . -type f | grep .nupkg.meta | grep System.Security.Cryptography.OpenSsl
(empty)
C:\Users\NAME\.nuget>cd ..\AppData
(empty)
C:\Users\NAME\AppData>"C:\Program Files\Git\usr\bin\find.exe" . -type f | grep .nupkg.meta | grep System.Security.Cryptography.OpenSsl
(empty)
C:\Users\NAME\AppData>cd "\Program Files\dotnet"
C:\Program Files\dotnet>"C:\Program Files\Git\usr\bin\find.exe" . -type f | grep .nupkg.meta | grep System.Security.Cryptography.OpenSsl
(empty)
C:\Program Files\dotnet>
(empty)
bddckr
commented
5 years ago I'm running dotnet nuget locals all --clear && dotnet restore --configfile NuGet.Config in a scheduled pipeline to raise PRs to update packages. My projects are using
<RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>When that pipeline is done I have a working build according to the (Azure DevOps) build agent, but as soon as I try to merge that branch to master, or just check out the branch locally on my machine and wait for Visual Studio to be done restoring and thus updating the lock file locally, I get the following diffs for most lock files:
I tried the same dotnet nuget locals all --clear && dotnet restore --configfile NuGet.Config locally after checking out the PR branch (so not using Visual Studio) and am getting the same diff.
Tbf I'm having a hard time following the previous discussion here, so if there's more to test I'm happy to do so, too!
tebeco
commented
5 years ago quick question, since i see --configfile NuGet.Config
should i do a :
dotnet new nugetconfig
and enforce a <clear /> and only add the https://api.nuget.org/v3/index.json ?
bddckr
commented
5 years ago I tried that already, doesn't help either for the hash mismatch.
tebeco
commented
5 years ago @nkolev92
Here is the github diff Is seems that the build looks green now
I guess the diff is because DisableImplicitNuGetFallbackFolder was set to false ?
Can you confirm me that on your latop dotnet restore --locked-mode works ?
Do i need to ask all the team to do the same ?
I guess depending on the outcome of the investigation. Once your problems are addressed, we'll know more.
Do i need to teamates to do a dotnet nuget locals -c all ?
Does it means dotnet build need to be replaced by dotnet restore --locked-mode && dotnet build --no-restore ? or dotnet build will be smart enought if the restore already ran with RestoreLockedMode set to true ?
It's going away in 3.0 anyways, and for your purposes it can mostly cause trouble depending on the machine.
Can you be more specific on what's going away here ^^ ?
tebeco
commented
5 years ago @bddckr
I create a bddckr branch on this repo, can you fork it ? https://github.com/dotnet-experimentations/LockFiles
then
dotnet nuget locals -c all
dotnet restore --force-evaluateand create a PR on branch bddckr on that repo so it can be used as a reference in this issue ?
bddckr
commented
5 years ago If I clone your repo I do not have any issues when doing dotnet restore --locked-mode. Doing dotnet nuget locals -c all && dotnet restore --force-evaluate also gives no issues!
I can't make this work for my own (private) repo, though - my own CI build always fails, stating there's a mis-match of one of the hashes (is there any way to get more details?) compared to what my own machine and the PR pipeline that does update the lock files say. I'm currently trying to come up with a new small reproduction project, but it's... just working in that one.
Here's something bizarre about my project, as far as I understand it at least:
dotnet restore --locked-mode and get the errors.dotnet restore --locked-mode --force-evaluate and get no errors.dotnet restore --locked-mode again and get no errors this time.git status, as clearly you just changed something to make it work! No changes according to git.obj folder's content for each project being created:
*.AssemblyInfo.cs*.AssemblyInfoInputs.cache*.assets.cacheI have no clue as to why those files need to be there in addition to the lock files, but if I commit them and keep them it gets rid of the NU1004s when running dotnet restore --locked-mode. This isn't needed in @tebeco's project either... 🤔
tebeco
commented
5 years ago In the repo linked, did you tried ONLY dotnet restore --force-evaluate ? never use --locked-mode on it, that could help spotting a diff
I can't make this work for my own (private) repo, though - my own CI build always fails, stating there's a mis-match of one of the hashes
That's why i opened this issue initially, i have the same issue in our private CI, so i managed to repro here.
Does your CI hae cache ? probably
Do you access nuget.org from the CI or do you have a proxy feed from your Corp <== if you have a cache here ... you need to find a way to clear it
Will try back in our CI.
you also said you ran, dotnet restore --locked-mode --force-evalute, why ? what is that supposed to do ? those 2 are not supposed to be "Opposite" ?
bddckr
commented
5 years ago In the repo linked, did you tried ONLY
dotnet restore --force-evaluate? never use--locked-modeon it, that could help spotting a diff
Yup. I did multiple clones, same for my own repo.
I can't make this work for my own (private) repo, though - my own CI build always fails, stating there's a mis-match of one of the hashes
That's why i opened this issue initially, i have the same issue in our private CI, so i managed to repro here. Does your CI hae cache ? probably
Nope! Just the Microsoft-hosted Azure DevOps agents that get thrown away/reset by them for each pipeline/job.
Do you access
nuget.orgfrom theCIor do you have aproxy feedfrom your Corp <== if you have a cache here ... you need to find a way to clear it
All defined in my NuGet.Config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="nuget" value="https://api.nuget.org/v3/index.json" />
<add key="myget-roslyn" value="https://dotnet.myget.org/F/roslyn/api/v3/index.json" />
<add key="myget-orleans" value="https://dotnet.myget.org/F/orleans-ci/api/v3/index.json" />
<add key="my-private-feed" value="https://pkgs.dev.azure.com/<org name>/_packaging/<feed name>/nuget/v3/index.json" />
</packageSources>
<fallbackPackageFolders>
<clear />
</fallbackPackageFolders>
</configuration>you also said you ran,
dotnet restore --locked-mode --force-evalute, why ? what is that supposed to do ? those 2 are not supposed to be "Opposite" ?
I didn't have the MSBuild property in the project files, so had to enforce the locked mode this way. You are probably right and I can drop it. I did try all of this, without any difference in the outcome.
tebeco
commented
5 years ago well, will wait for @nkolev92 for all previous question then ^^
nkolev92
commented
5 years ago Lots of question, so I'll try to address them 1 by 1 :)
@tebeco
Can you be more specific on what's going away here ^^ ?
The SDK will not install with a fallback folder starting in 3.0.
I guess the diff is because DisableImplicitNuGetFallbackFolder was set to false ?
Likely, yes. On my machine even that's up to date, but I have pretty much every preview ever installed.
Does it means dotnet build need to be replaced by dotnet restore --locked-mode && dotnet build --no-restore ? or dotnet build will be smart enought if the restore already ran with RestoreLockedMode set to true ?
dotnet restore --locked-mode & then dotnet restore will do an incremental check and bail early. It's up to you what you prefer to run. I'd consider setting the CI variable to enable locked mode and just calling dotnet build if at all possible :)
dotnet restore --locked-mode --force-evalute
As @tebeco suggested, try not to run that. It's a bug currently. https://github.com/NuGet/Home/issues/8222
@bddckr
Maybe you are hitting https://github.com/NuGet/Home/issues/7350? Maybe the package in question exists in a different feed?
What I'd recommend to try to eliminate random is noise is the following.
All of that should work locally.
Next on the CI.
Hope this helps. Let us know if you hit any issues.
tebeco
commented
5 years ago @nkolev92 i did have the same issue on our teamcity on premise CI i think i tried previous made here
i'll try that again this week and keep you in touch
What about other teamates ? since the fallback is disabled i guess they should be able to "restore" but i guess it will fail if
is it safe to force this also :
<RestoreNoCache>true</RestoreNoCache>
dotnet restore --locker-mode --no-cachedotnet restore --force-evaluate --no-cacheis that stupid ? i would loose "package cache if i do that i guess ?
tebeco
commented
5 years ago @nkolev92
sorry for all the questions, as i am deeply trying to understand how it work to properly helps about 30 developers to migrate over lockfiles ;)
better be sure how it works out, potential glitches etc ...
nkolev92
commented
5 years ago no-cache doesn't really do that.
It merely skips the http-cache. NuGet "treats" the global packages folder as an installation directory, not a cache that's managed and expires etc.
What about other teamates ? since the fallback is disabled i guess they should be able to "restore" but i guess it will fail if
- in the past they did a restore
- it used the fallback
- fallback is now in their global package folder
- they attempt force evaluate
1 and 3 could lead to issues. 2 would mean the package gets downloaded during the next restore.
We really need a best practices guidebook from our end (//cc @rrelyea, @anangaur).
Basically what I'd recommend. One person updates, the rest pull, run a clean to blow out the obj folder. Clean the global packages folder to avoid potential issues (note that it might work for some people, but I don't know if you want to make people do the whole dance we've been doing here.
tebeco
commented
5 years ago i would honestly recommend a complex scenario for the guidance
taking in account 2.x bug as 2.1 is LTS for a long time so i honestly suggest doing patch in sdk in all possible feature band to raise error if DisableImplicitNuGetFallbackFolder is still false
and prints out a URL to a detailed page explaining why it need to be true and who / when needs to `dotnet nuget locals -c all && git en - r */.lock.json && dotnet restore --force-evaluate" + Agent "maintainance"
else this will escaladante in a pill of issue here ^^
the existing blog post from @anangaur was honestly not enough and also mixing msbuild + cli options without explicit "combo equivalent" or workflow
keep in mind that since generating and using lockfile is not the default behavior people without advanced scenario won't use it so later guidance will probably need to actually create a repo and build script and/or azure pipeline yaml because that's why users are going to do in their CI. Just creating lockfile for "POC" repo is less likely to happen
also ... unrelated question
is it possible for a later version to make
dotnet restore just ... restore (locked mode)
and something like dotnet install doing a force evaluate separate commanda for separate concerns so there are less question around / a more uniform workflow / easiest command to type in the CLI
This is a pretty big source of confusion to be honest
if that a too big breaking change and fear for back compat then either use a major (dotnet 5 is in 1 year would be perfect)
or we could keep restore and find another one, the idea is to have 2 commands, not one command doing 2 different things (completely oppositive thing on top of it)
that was demonstrate in this thread by dotnet restore --locked-mode --force-evaluate
nkolev92
commented
5 years ago in sdk in all possible feature band to raise error if DisableImplicitNuGetFallbackFolder is still false
The issue with that is that the fallback folder is not necessarily invalid. So not sure something like this will happen.
keep in mind that since generating and using lockfile is not the default behavior people without advanced scenario won't use it so later guidance will probably need to actually create a repo and build script and/or azure pipeline yaml because that's why users are going to do in their CI. Just creating lockfile for "POC" repo is less likely to happen
Good point. The examples should be close to real-life rather than a super straightforward example. s
is it possible for a later version to make dotnet restore just ... restore (locked mode) and something like dotnet install doing a force evaluate separate commanda for separate concerns so there are less question around / a more uniform workflow / easiest command to type in the CLI
I think that ship has sailed. It'd be a breaking chance. It's kind of been a thing throughout the entirety of .NET Core lifetime.
The idea is that customers set restore locked mode property in their CIs, and don't really change the commands used to build.
tebeco
commented
5 years ago fair enought
will keep you in touch with our CI as @bddckr i stumbled open issue even with these steps so i'll be sure to triple check again
will also try with multiple teammates to see what happens.
i would have suggest raising an announcement to raise awareness but it seems the nuget organization does not have the same as aspnet or dotnet
and honestly ...thanks A LOT for your time here
nkolev92
commented
5 years ago We have the following repo: https://github.com/NuGet/announcements. @anangaur and @karann-msft will help decide whatever we come up with is appropriate for that or a blog.
I have an issue created about this whole effort https://github.com/NuGet/Home/issues/8289.
bddckr
commented
5 years ago Lots of question, so I'll try to address them 1 by 1 :)
Thanks a lot for finding the time!
dotnet restore --locked-mode --force-evalute
As @tebeco suggested, try not to run that. It's a bug currently.
8222
Got it.
Maybe you are hitting #7350? Maybe the package in question exists in a different feed?
To quote from that issue:
doesn't take into account the fact that the same package with different hashes were there on different feeds and there was indeed one feed which had the correct hash of the package but since it's not deterministic that which feed restore will use to restore this package, there will be inconsistency in validation and it's result
That definitely could be the case, though I'd be surprised to see a version then having a different content hash. I guess it could easily have one if that wasn't build with lock files deterministically either 😉 I have no idea if this is the case. Is there any way for me to check humanly possible? Looking at my lock file as is I don't think I can easily figure this out for the hundred+ packages referenced.
Is the better solution to only reference a private NuGet feed which is set up to include all the necessary feeds (including NuGet.org) as upstream sources? I'll try that next, it seems like the logical conclusion to the issue of "who's deciding which of those same-version packages to use?" as this gives my team a single answer: Our private feed.
What I'd recommend to try to eliminate random is noise is the following.
- Disable the implicit fallback folder.
- Disable LockedMode.
- Clear the global packages folder
- Run dotnet restore and commit the regenerated lock files.
- Now run locked mode restore and see what happens.
All of that should work locally.
<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder> in all projects, so that should follow this instruction. Also using <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile> only on executable projects here, as recommended in the NuGet docs (library projects are not using a lock file).RestoreLockedMode in my projects as I want my team to update lock files as necessary locally. On the CI I run with --locked-mode. So taking your advice here: I'm not passing --locked-mode to the restore command in the next steps.dotnet nuget locals global-packages --clear.dotnet restore now. (Not using --locked-mode as mentioned earlier.)
🚨 Issue: This runs fine (all projects listed as Restore completed), but there are no changes to the lock files, thus I can't commit anything. Let's assume this means from my local perspective everything is already OK.dotnet restore --locked-mode.
🚨 Issue: The two (executable) projects that are using RestorePackagesWithLockFile set to true are listed as
C:\Program Files\dotnet\sdk\3.0.100-preview8-013656\NuGet.targets(123,5): error NU1004: The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file. [C:\Path\To\MySolution.sln]
- Now locally still, delete the global packages folder.
- Run locked mode restore. Everything should still be working.
Ignoring the issues in the previous process:
dotnet nuget locals global-packages --clear.dotnet restore --locked-mode. Same issue as before (as expected I assume):
🚨 Issue: The two (executable) projects that are using RestorePackagesWithLockFile set to true are listed as
C:\Program Files\dotnet\sdk\3.0.100-preview8-013656\NuGet.targets(123,5): error NU1004: The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file. [C:\Path\To\MySolution.sln]
I know you did tell me there's the bug with running both --force-evaluate and --locked-mode but I figured it's worth stating what I can observe here nonetheless: Here's a process that removes the lock error locally, but still does not result in any change to the lock file. At least temporarily, see for yourself:
dotnet nuget locals global-packages --cleardotnet restore --force-evaluate --locked-mode (Just dotnet restore --force-evaluate doesn't give errors either, but later still gives the lock error. I assume this is expected as --force-evaluate basically just overrides the potential use of RestoreLockedMode being set to true in the project. Thus in my case it's the same as running just dotnet restore.)dotnet restore --locked-mode succeeds without the lock error now! 🎉dotnet restore (no locked-mode!)dotnet restore --locked-mode 🚨 fails again with the lock errors.--force-evaluate --locked-mode definitely does something different than just --force-evaluate:
--locked-mode afterwards, until just restore is run (without locked mode).--force-evaluate never helps to make locked mode restores work afterwards.It feels like the linked non-deterministic feed selection is the issue here, yet I wonder why I can reproduce all of this locally without ending up with any changes to the lock files. And yes I did check my .gitignore like 5 times during all of this 😉
Next on the CI.
- Make sure you have made the fallback folder changes
- For sanity, add a global packages folder clear statement
- Run only locked mode restores on the CI.
I'm following this on the CI, with the state of the lock files as is, as explained above. I.e. the one where it works locally for locked mode, but only when running --force-evaluate --locked-mode.
<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder> in all projects.dotnet nuget locals global-packages --clear in the CI pipeline before the next steps.dotnet restore --locked-mode.
🚨 Failing with the exact same lock error as locally (see above).I can confirm there are no changes to the lock files on the CI here either (when running without --locked-mode as a test).
bddckr
commented
5 years ago I'm thinking about running my CI without --locked-mode for now and instead failing the build after the restore task if any of the lock files is modified according to git. Since the files aren't modified at all while the lock error is given by the restore command in the local and CI case, it doesn't actually modify any of the files until a "real package change" was done by the team. Thus that should work.
nkolev92
commented
5 years ago
- Running dotnet restore --locked-mode. 🚨 Issue: The two (executable) projects that are using RestorePackagesWithLockFile set to true are listed as
This might be a bug. Normally running a clean restore and then running a --locked-mode restore on the same machine should work no problem.
Any chance you can provide a repro for us?
dotnet restore --force-evaluate --locked-mode (Just dotnet restore --force-evaluate doesn't give errors either, but later still gives the lock error. I assume this is expected as --force-evaluate basically just overrides the potential use of RestoreLockedMode being set to true in the project. Thus in my case it's the same as running just dotnet restore.
Your hunch is correct, it basically ignores the locked mode part right now.
If possible, if you can provide a repro, we'd be happy to investigate.
bddckr
commented
5 years ago I'm unable to recreate the issue in @tebeco's repo, and also unable to re-create using the same packages, the same distribution of settings (some .props, a NuGet.Config etc.) in a newly created project with and without using Microsoft.Build.CentralPackageVersions.
Copying my existing repo piece by piece into a new solution setup still gives the issue, so I'm kinda baffled as to what specifically is giving the issue here. It's not the package list and it's not the .props setup, nor Microsoft.Build.CentralPackageVersions.
I did end up going the other way round and taking my actual repo and removing pieces: First Microsoft.Build.CentralPackageVersions, then dumbing down my .props setup so everything is just in each .csproj file. This also doesn't help and the error is still happening.
The only way to make it work at all times (both locally and in the CI agent, as well as just fixing the local --locked-mode vs. just restore) is to do what I mentioned earlier:
- Spend some time doing some folder-wide diffs and notice the only difference is the
objfolder's content for each project being created:
*.AssemblyInfo.cs*.AssemblyInfoInputs.cache*.assets.cacheI have no clue as to why those files need to be there in addition to the lock files, but if I commit them and keep them it gets rid of the
NU1004s when runningdotnet restore --locked-mode.
It might not need all of these files, but those in particular being present from a regular restore run before then attempting restore --locked-mode always works.
nkolev92
commented
5 years ago Those files are not considered by NuGet, so unfortunately I don't have any pointers for you :(
If you can't share a project, I can help guide you to debug nuget.exe (easiest of the clients) to see what exactly the problem is.
Fixing https://github.com/NuGet/Home/issues/7696 will help these investigations.
bddckr
commented
5 years ago That would be much appreciated as I can't share this project sadly. I did increase the verbosity all the way. Each project that isn't using the lock file is going through fine, then when it comes to the two that use lock files, this is what I get locally:
Persisting no-op dg to C:\Some\Path\artifacts/build/obj/base/My.App/My.App.csproj.nuget.dgspec.json (TaskId:42)
19:42:27.487 1>C:\Program Files\dotnet\sdk\3.0.100-preview8-013656\NuGet.targets(123,5): error NU1004: The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file. [C:\Some\Path\My.Solution.sln]
Committing restore... (TaskId:42)
Assets file has not changed. Skipping assets file writing. Path: C:\Some\Path\artifacts/build/obj/base/My.App/project.assets.json (TaskId:42)
Writing cache file to disk. Path: C:\Some\Path\artifacts\build\obj\base\My.App\My.App.csproj.nuget.cache (TaskId:42)
Writing packages lock file at disk. Path: C:\Some\Path\src\My.App\packages.lock.json (TaskId:42)
Restore failed in 12.06 ms for C:\Some\Path\src\My.App\My.App.csproj. (TaskId:42)The two projects depend on the other projects via ProjectReference.
tebeco
commented
5 years ago i'll see if i can anonymize an empty shell of the repo at work gonna take time
nkolev92
commented
5 years ago @bddckr
Sure.
Checkout the repo
Follow the steps to build the client https://github.com/NuGet/NuGet.Client#steps-to-build-nuget-client-tools. Make sure you don't run the tests when you build. They are unnecessary for you.
Load the NuGet.sln with a version of VS 2019.
A copy from some instructions I have written before.
Put the path to your sln/project in there. (You can set the locked mode variable in your project or pass the parameter).
In your specific case, you are testing a PackageReference project, so you want a break point, here: https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Commands/RestoreCommand/RestoreCommand.cs#L481 You probably want to condition it on _request.Project.Name where the name is the project in question that fails.
Lastly follow https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.ProjectModel/ProjectLockFile/PackagesLockFileUtilities.cs#L69 and see where it fails.
File bugs if necessary You can find the bugs relating to lock in the following query: https://github.com/NuGet/Home/issues?q=is%3Aopen+is%3Aissue+label%3AArea%3ARepeatableBuild Of the ones in there, I think https://github.com/NuGet/Home/issues/8465 is the one that could be affecting you.
bddckr
commented
5 years ago It goes here and then fails here, because of 0 + 6 != 0.
Turns out due to my .props setup all ProjectReferences are changed to always include <PrivateAssets>all</PrivateAssets>. This in turn means the lock file of the project that references another one does not know about the other one's dependencies - which makes sense, I'm disabling the transitive flow here after all!
I am removing this to resolve the issue. The default of PrivateAssets is contentfiles;analyzers;build which works just fine for me. With this resolved my lock files now have the added knowledge of the dependencies of each referenced dependency and thus --locked-mode is happy.
Thanks a lot for helping me resolve this completely!
The idea was to enforce referencing projects to define a package dependency explicitly, instead of relying on a package reference being imported via the transitive dependency of the project they depend on.
This seems to have worked so far just nicely: a project only needed to add package reference for the ones it needs. If it needs a package that a referenced project already uses, one has to explicitly add a dependency nonetheless. The build output seems to have worked fine so far - I am not missing any "hidden" package outputs in the "root" projects despite basically using this in all projects:
<ItemDefinitionGroup>
<PackageReference>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<ProjectReference>
<PrivateAssets>all</PrivateAssets>
</ProjectReference>
</ItemDefinitionGroup>The lock files are the first time I'm hitting this issue now.
PackageReference and remove it for ProjectReference? Why do these behave differently like this with the lock mode?nkolev92
commented
5 years ago Thanks for investigating @bddckr This is super helpful!
Yeah, sounds like a bug. I'll investigate myself (get a repro) and confirm soon.
tebeco
commented
4 years ago @nkolev92 it seems to be persistent depending on computer, build agent included (my guess is that the cleanup might not have been done properly)
is there an MsBuild variable to use lockfile for the restore but only skip hash ? i do understand this is a security issue to avoid package tempering / man in the middle and spoofing ^^
i will continue to investigate here what's happening in details kinda like @bddckr did. Just have to convince affectés user to use their laptop ;)
nkolev92
commented
4 years ago is there an MsBuild variable to use lockfile for the restore but only skip hash ?
https://github.com/NuGet/Home/issues/7651
As far as the cleanup goes, another thought is ensuring that the same sources are being used for the same repo. Specifically [see].(https://github.com/NuGet/NuGet.Client/blob/638f34578d2221abc1259286ce2d4c132600cea7/NuGet.Config#L4)
nkolev92
commented
4 years ago Thanks to @bddckr who debugged to narrow down the bug, I have created a tracking issue https://github.com/NuGet/Home/issues/8565, which we hope to fix soon.
@tebeco I'd like to keep this discussion focused to your specific issue, figure out why the locking down is not working out, so hopefully that produces some learnings.
So for now, I'm marking this issue as discussions, until we figure out otherwise.
nkolev92
commented
4 years ago @rrelyea self assinging the discussions issue so it's tracked. Feel free to assign bugs produced as you see fit.
tebeco
commented
4 years ago Can change the title if it helps people
[DISCUSSION] 3.x, Lockfile, Latest major version and smoking barrels
this is kind of an edge case but it might help people as it cover lots of scenarios.
PS : trying to find some time today, as i will be working on updating the CI, to check if it's green now. If it's not it's gonna take lot of time but i'll probably try to remove all code a create empty shell project to see if i repro in order to make the repro public
jrusbatch
commented
2 years ago
- I'm using
<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>in all projects, so that should follow this instruction. Also using<RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>only on executable projects here, as recommended in the NuGet docs (library projects are not using a lock file).
I stumbled on this thread while trying to catch up on the latest state of this feature. I'm having trouble finding an explanation for the guidance that RestorePackagesWithLockFile should only be set on executable projects, and that libraries should not have a lock file. I recognize that this issue is a few years old at this point. Is that referring to antiquated docs that aren't available anymore?
nkolev92
commented
2 years ago I think people are referring to https://docs.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files#make-lock-file-part-of-your-source-repository
It contains a further explanation, about why lock files might not be as helpful in the library scenarios, but they are not without merit you can still use them.
We can consider tweaking the documentation if it doesn't read super well.
Details about Problem
NuGet product used dotnet.exe:
https://github.com/dotnet-experimentations/LockFiles/blob/master/azure-pipelines.yml#L28-L37
OS version :
Worked before? Never seen it working, tried at work, same behaior. See multiple issue, never made it work ^^
Detailed repro steps so we can see the same problem
dotnet restore=> it should generate lock filesdotnet restore -p:FakeCi=1it seems to work locally because the hash are from the same computerAttempts (not in this repo)
I've seen other issue with bugs in nuget 4.9.x, and 5.2.x Some of these seems to says that there were version of the sdk that were shipped with wrong hash in
NugetFallbackFolderSome were suggesting todotnet nuget locals --clear alldotnet restore --force --no-cacheAs you can see :
dotnet retoreon the CI only "restore"Logs shows that it seems to set the good
Ms Build props:RestorePackagesWithLockFileRestoreLockedModeVerbose Logs
Full logs + failure line is here : https://dev.azure.com/dotnet-experimentations/LockFiles/_build/results?buildId=9&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=2b20c7d0-7587-5b0f-aadc-b456e6f0b807&l=6914
Sample Project
https://github.com/dotnet-experimentations/LockFiles/