search

NuGet / Home

Repo for NuGet Client issues
Other
1.49k stars 251 forks source link

Enable repeatable package restores using a lock file #5602

Closed anangaur closed 5 years ago

anangaur commented 7 years ago

Discussions should happen on this issue. Please link other issues with similar asks to this one.

July 2018 - Announced feature December 2018 - Blog

bording commented 7 years ago

It appears that the current specification document is incorrect regarding how version ranges work with direct dependencies. It says that given the following package reference:

<PackageReference Include="My.Sample.Lib" Version="[4.0.0, 5.0.0]"/>

That it would pick the highest version, 5.0.0. This is not what I observe. Instead, that version range would resolve to the lowest version, 4.0.0.

Floating version numbers appear to be the only case where any sort of highest version logic is applied.

I also don't see anything indicating that highest would be expected in any of the documentation I've found:

https://docs.microsoft.com/en-us/nuget/consume-packages/dependency-resolution https://docs.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files

anangaur commented 7 years ago

Thanks @bording for reporting the discrepancy. This was an oversight from my end. I have corrected the same in the spec.

  1. If a range is specified - NuGet resolves to the lowest version specified in that range. E.g.

    Feed has only these versions for My.Sample.Lib: 4.0.0, 4.6.0, 5.0.0

    a. Range is specified:

    <PackageReference Include="My.Sample.Lib" Version="[4.0.0, 5.0.0]"/>

    NuGet resolves to the 4.0.0 here.

    b. Range is specified contd..

    <PackageReference Include="My.Sample.Lib" Version="[4.1.0, 5.0.0]"/>

    NuGet resolves to the 4.6.0 here.

isaacabraham commented 6 years ago

What happens if the NuGet.config file is on one machine but not on another, or is changed after the lock file is create - does the lock file still take effect?

anangaur commented 6 years ago

IMO, sources definition should not matter for a lock file. Irrespective of which source the packages are coming from, finally it should be the same package that goes into the build.

To make sure whether its the same package or not I am thinking about putting the hash of the package in the lock file. Open to other ideas.

isaacabraham commented 6 years ago

@anangaur Sorry, didn't explain myself very well :-) What happens in the following situation: -

  1. You "turn on" repeatable builds in NuGet.config (mechanism still TBD).
  2. Your team start using NuGet, lock file is generated etc.
  3. Someone removes the flag in the NuGet.config.

What happens to the lock file? Does it still get used or is it discarded and deleted?

anangaur commented 6 years ago

@isaacabraham Haven't put much thought on it but I am leaning towards generating the file by default and not as an option. Let me spend sometime on it and thn I will come back with the proposal.

mletterle commented 6 years ago

why not just have a command line option to create or consume a project wide project.assets.json file at restore time?

Something like:

nuget restore path\to\something.sln -AssetsOut something.assets.json

and later (say on a build server)

nuget restore \path\to\something.sln --AssetsIn something.assets.json

It would be up to the consumer to regenerate/edit the assets file and check it into source control and to configure their builds to use it.

This would seem to allow the requested functionality with the minimum amount of changes....

anangaur commented 6 years ago

Asset file is not just the restore graph but has multiple overloaded functionality and lot of contents irrelevant to a typical lock file. For example it contains all the warnings/errors applicable for the project restore. In addition to this, the assets file is not easy to understand or modify. (IMO even the lock file should not be manually edited). I have seen different assets file getting generated for the same dependencies (the lines get re-ordered) and hence it will be difficult to manage the merge conflicts. Assets file in the current form will not be able to handle floating versions. However the current idea is very similar to generating a lock file by refactoring out the relevant contents from assets file that gets generated in the obj directory. I would rather solve this issue using a lock file mechanism than re-using the assets file and then live to support the backward compatibility in the long run :)

mletterle commented 6 years ago

True, just having a packages.config as input with specific, locked versions would be enough, having the ability to generate that file on restore would go a long way. packages.config is documented and presumably will be supported for the foreseeable future.

ap0llo commented 6 years ago

I would really like to have this feature in the NuGet client. Are there any updates (or perhaps even a roadmap) you can share?

NasAmin commented 6 years ago

This would be a great feature for nuget to support CI/CD workflows. Is there any ETA on when this feature will be available?

anangaur commented 6 years ago

@forki @isaacabraham Wanted to bring the conversation here :) The current proposal is having the lock file at project level with the intention to lock the transitive dependencies per project. However, I have heard the problems it could have at runtime and I am contemplating to bring it at solution level.

Currently NuGet restore does a project level restore - may be that has to change if we want to bring the lock file to solution level? @jainaashish

isaacabraham commented 6 years ago

@anangaur hi :-)

OK. I'll try to (briefly) outline some of the reasons that come to mind why I think locking dependencies at either project or solution level will be a mistake. I'm sure @forki will have his own thoughts as well.

  1. If you lock at project level, at runtime, you'll be in the same precarious position you are now vis-a-vis dependency management i.e. you won't be able to guarantee that Project A and Project B both depend on the same version of the same dependency. At runtime, things might work, they might not.

  2. If you lock at solution level, you'll still probably run into problems. I know many people that have large code bases with multiple solutions that share projects in the same repository. 99/100 times, they'll want those solutions to have the same exact dependencies for consistency and expected behaviour. They won't want to open one solution and get one set of versions, then open another solution and get another set for the same project.

Alternatively, consider the situation where you e.g. have two solutions, one for your "main" codebase and another for integration and unit tests (which I have seen before). Do you really want to maintain a separate dependency chain for both of them? What happens when they get out of sync?

  1. Remember that some developers work outside of projects - particularly in the F# community, where exploratory scripts are a key part of their development process. Coupling dependencies to projects or solutions effectively kills working with scripts (or at least makes it much more challenging), in which you don't want (or need) a solution or project file at all. Instead, you just want to specific a set of dependencies and to bring them down onto the file system, and then reference then from a script.

In summary

Ask yourself this - how often in a repository do you explicitly want different versions of the same dependency? I suppose I would say this (because of my involvement with Paket) but decoupling yourself from projects and solutions will free you from all of these issues. Instead, consider pinning dependencies at the folder level (typically repository level). You then get consistency across all of your projects and solutions and scripts, because there's there's only one dependency graph across the whole repo.

For those cases where you need to go "outside the box" and must have different dependency graphs within a repo, consider something like Paket's Groups feature, which allows you to explicitly have distinct dependency chains (with their own section in the lock file). However, this is the exception to the rule - it doesn't happen very often.

Just my two cents - you may well feel differently, and it's entirely your choice how you proceed. Good luck! :-)

ap0llo commented 6 years ago

I agree with @isaacabraham. I know of code bases for a single application split into multiple solutions with some of these solutions even overlapping, so locking dependencies neither at project level nor at solution level works in such scenarios. Having a lock file at repository level (or a arbitrary file system subtree) sounds sensible to me

anangaur commented 6 years ago

@isaacabraham Thanks for the detailed reply. I am fully in agreement in what you mentioned above. This is something required as part of the following:

https://github.com/NuGet/Home/wiki/Manage-allowed-packages-for-a-solution-%28or-globally%29

This issue is primarily to lock down the transitive dependencies (at install time) so that restores are repeatable across space and time. I do see both of these workitems are related but I am trying to kind of segregate these out and attack one problem at a time.

Right now the proposal is to have NuGet generate the lock file at the time of install at project level as that is how NuGet restore works but I have been hearing a lot of voices to make the lock file generated at solution level. From twitter and from Paket experience, I understand that's your recommendation too.

isaacabraham commented 6 years ago

@anangaur Hi. Sorry - but no, that's not my recommendation really. My comment just above states:

decoupling yourself from projects and solutions will free you from all of these issues. Instead, consider pinning dependencies at the folder level (typically repository level). You then get consistency across all of your projects and solutions and scripts, because there's there's only one dependency graph across the whole repo.

So I hope I'm clear here - working at either projects or solutions will not, in my opinion, provide a satisfactory solution that is either simple to understand, consistent and repeatable.

Again, though, I'll repeat: That's my opinion. It's entirely up to you how you proceed.

anangaur commented 6 years ago

Sure. Thanks for the explanation. Will keep your recommendation in mind while I iterate over it. I will update the thread as we make progress. Appreciate your input here.

forki commented 6 years ago

my recommendation:

vivainio commented 6 years ago

I want to pile on "please don't introduce any functionality at solution level". Single solution typically represents a very small part of a bigger product that wants to harmonize the dll versions. Solutions should remain a superficial "ide convenience" feature instead of getting critical new responsibilities.

dasMulli commented 6 years ago

@anangaur after reading through the recent wiki updates: I would discourage using .lock as a file extension for the planned feature. a .lock carries the meaning of a semaphore file on *nix or a marker for things not to delete. That's why there's packages-lock.json and friends or .cache etc. to try to avoid using .lock as a file extension for non-"traditional" .lock meanings. (I've also seen ppl use yarn but then not add yarn.lock to their source control because of thinking it was a temporary file or existing gitignore entries)

forki commented 6 years ago

*.lock is VERY commonly for package managers. Many use that.

Martin Andreas Ullrich notifications@github.com schrieb am Sa., 14. Apr. 2018, 16:12:

@anangaur https://github.com/anangaur after reading through the recent wiki updates: I would discourage using .lock as a file extension for the planned feature. a .lock carries the meaning of a semaphore file on *nix or a marker for things not to delete. That's why there's packages-lock.json and friends or .cache etc. to try to avoid using .lock as a file extension for non-"traditional" .lock meanings. (I've also seen ppl use yarn but then not add yarn.lock to their source control because of thinking it was a temporary file or existing gitignore entries)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NuGet/Home/issues/5602#issuecomment-381331929, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNP2bC-SDypd8FQt7RekPo9eAYxaAks5togO-gaJpZM4OYukQ .

RussKie commented 6 years ago

I'll throw another angle in this discussion.

One of my clients is a financial institution and due to legislative requirements they must have repeatable builds at any given point of time. With packages.config it was achieved by committing packages into repository, however the new model employed by .NetCore that uses a shared cache has become a significant issue for those guys...

anangaur commented 6 years ago

@RussKie How is the centralized cache a problem? Why do you need to commit packages into a repository? That should be NuGet's job to bring in the same packages every time you restore irrespective of when and where you restore the packages.

RussKie commented 6 years ago

In enterprise a lot of systems are either behind proxies or do not have access to internet at all. A developer adds a Nuget package to a project on a local machine. A build server does not have a connection to internet and thus unable to restore a package. Epic fail.

Another developer with a different level of internet access clones a project to his/her workstation. Due to proxy permissions the developer is unable to connect to the source Nuget gallery (e.g. nuget.org) and restore packages. Epic fail.

mungojam commented 6 years ago

For CI builds, if we use UpdateLockFileOnRestore: deny, will that fail if some packages have been added to the repository and our projects are set to pick up latest package versions?

I'd hope not. It should only fail if it would have no way to meet the PackageReference conditions with what is in the lock file.

anangaur commented 6 years ago

Lock file will have exact versions of the packages locked and hence if it cannot find the locked version of packages it will fail to update the lock file as you chose the option ‘deny’. If you want the resolution to change then why set the option to deny?

mungojam commented 6 years ago

Sorry, probably not made the scenario clear. It's a pretty standard CI build, and I want to be sure that if the default resolution would be different to what is in the lock file it won't error or even warn.

The spec suggested that maybe if the resolution was different it would fail with deny, but it's not clear if that's just if the lock file package resolution is not possible or if the resolution has changed for a different reason like a new package version being available.

anangaur commented 6 years ago

Thanks for explaining the scenario. I guess we are on the same page then. Once you have a lock file for a project, the only check we do is if there was a change in requested versions from the time lock file was generated. If not, the lock file is just used to get the packages.

So if you had stated the following:

<PackageReference Include="Contoso.Utility.UsefulStuff" Version="*" />

And the lock file resolved to 2.0.0 (latest version at the time). On the CI machine when you trigger the build, if you didn't change the PackageReference, it restore will just resolve to the version 2.0.0 as mentioned in the lock file. Even if a newer version (say 2.1.0) is available. There won't be any warning or error raised here with UpdateLockFileOnRestore: deny or with UpdateLockFileOnRestore: warn

anangaur commented 6 years ago

Change in spec:

The option UpdateLockFileOnRestore/--update-lockfile has changed to RestoreLockedMode/--locked-mode. It is set to false by default.

NasAmin commented 6 years ago

Hi Is there any ETA on this. Wasn't this originally planned for Spring 2018? I'd appreciate if some information can be provided about the estimate completion time for this.

regards,

Nas

NasAmin commented 6 years ago

Also, what impact will this have on Semantic Versioning and GitFlow?

anangaur commented 5 years ago

Also, what impact will this have on Semantic Versioning and GitFlow?

@NasAmin Apologies for the late reply. I am not sure I understood your comment. Can you please elaborate?

anangaur commented 5 years ago

The per project lock file feature would be available with VS 15.9 preview 5. I am currently working on the docs. The PR is here: https://github.com/NuGet/docs.microsoft.com-nuget/pull/1119

OsirisTerje commented 5 years ago

This is going in the right direction. Really looking forward to the lock file per solution.

anangaur commented 5 years ago

Really looking forward to the lock file per solution.

@OsirisTerje Yepp. Remember our interaction on this one. I have a draft spec out here: https://github.com/NuGet/Home/wiki/Centrally-managing-NuGet-packages

Would love any early feedback.

rrelyea commented 5 years ago

PackageReferences with LockFile shipped in NuGet 4.9.0 Ship Vehicles:

Flavien commented 5 years ago

I have a problem with this feature. When running on my dev environment (Windows), the contentHash of some packages is different from the one on my build environment (LXSS). For example, on Windows:

      "System.Data.Common": {
        "type": "Transitive",
        "resolved": "4.3.0",
        "contentHash": "lm6E3T5u7BOuEH0u18JpbJHxBfOJPuCyl4Kg1RH10ktYLp5uEEE1xKrHW56/We4SnZpGAuCc9N0MJpSDhTHZGQ==",
        "dependencies": {
          "System.Collections": "4.3.0",
          "System.Globalization": "4.3.0",
          "System.IO": "4.3.0",
          "System.Resources.ResourceManager": "4.3.0",
          "System.Runtime": "4.3.0",
          "System.Runtime.Extensions": "4.3.0",
          "System.Text.RegularExpressions": "4.3.0",
          "System.Threading.Tasks": "4.3.0"
        }
      },

But on Ubuntu (LXSS):

      "System.Data.Common": {
        "type": "Transitive",
        "resolved": "4.3.0",
        "contentHash": "OGX4ifHI67s/2KRrXGMmEjmszCptbhWS4BieBdiBPl/8pn0A1xPMx0Gn0dKK0zKtEaelYNh5M7M5lCZL0EA4eA==",
        "dependencies": {
          "System.Collections": "4.3.0",
          "System.Globalization": "4.3.0",
          "System.IO": "4.3.0",
          "System.Resources.ResourceManager": "4.3.0",
          "System.Runtime": "4.3.0",
          "System.Runtime.Extensions": "4.3.0",
          "System.Text.RegularExpressions": "4.3.0",
          "System.Threading.Tasks": "4.3.0"
        }
      },

So of course, trying to restore on the build environment fails, and if I delete the lock file on the build environment, it gets recreated with different contentHash values for some of the packages.

Is that a bug?

forki commented 5 years ago

this is something that we noticed long long time ago. way before nuget was annoucing lock files. That's why paket doesn't have hashes in it's lock file. Hope it gets fixed server side soon. /cc @isaac_abraham

Am Di., 18. Dez. 2018 um 18:46 Uhr schrieb Flavien Charlon < notifications@github.com>:

I have a problem with this feature. When running on my dev environment (Windows), the contentHash of some packages is different from the one on my build environment (LXSS). For example, on Windows:

  "System.Data.Common": {
    "type": "Transitive",
    "resolved": "4.3.0",
    "contentHash": "lm6E3T5u7BOuEH0u18JpbJHxBfOJPuCyl4Kg1RH10ktYLp5uEEE1xKrHW56/We4SnZpGAuCc9N0MJpSDhTHZGQ==",
    "dependencies": {
      "System.Collections": "4.3.0",
      "System.Globalization": "4.3.0",
      "System.IO": "4.3.0",
      "System.Resources.ResourceManager": "4.3.0",
      "System.Runtime": "4.3.0",
      "System.Runtime.Extensions": "4.3.0",
      "System.Text.RegularExpressions": "4.3.0",
      "System.Threading.Tasks": "4.3.0"
    }
  },

But on Ubuntu (LXSS):

  "System.Data.Common": {
    "type": "Transitive",
    "resolved": "4.3.0",
    "contentHash": "OGX4ifHI67s/2KRrXGMmEjmszCptbhWS4BieBdiBPl/8pn0A1xPMx0Gn0dKK0zKtEaelYNh5M7M5lCZL0EA4eA==",
    "dependencies": {
      "System.Collections": "4.3.0",
      "System.Globalization": "4.3.0",
      "System.IO": "4.3.0",
      "System.Resources.ResourceManager": "4.3.0",
      "System.Runtime": "4.3.0",
      "System.Runtime.Extensions": "4.3.0",
      "System.Text.RegularExpressions": "4.3.0",
      "System.Threading.Tasks": "4.3.0"
    }
  },

So of course, trying to restore on the build environment fails, and if I delete the lock file on the build environment, it gets recreated with different contentHash values for some of the packages.

Is that a bug?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NuGet/Home/issues/5602#issuecomment-448308066, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNGZvpjif2i1dQvNOk2rYc-E6GyoJks5u6SnrgaJpZM4OYukQ .

anangaur commented 5 years ago

if I delete the lock file on the build environment, it gets recreated with different contentHash values for some of the packages.

@Flavien This is by design. Lock files are auto recreated if the RestoreWithLockFile property is set and if lock file is out of sync or not present. You can fail the build if lock file needs to be updated by setting the property RestoreLockedMode to true.

@forki , @Flavien Content hash being different on different OS seems weird to me. We need to investigate this. In the mean time, we may need an option to disable content hash checking. Thanks for reporting this.

anangaur commented 5 years ago

/cc: @rrelyea @jainaashish

jainaashish commented 5 years ago

Even I don't follow why contentHash is different for the same package across different OS? Is this ZipArchieve thing? or NuGet feed?

forki commented 5 years ago

for clarification: in paket we tried to verify the hash that the server sends. But that one is differing (in some case - not all the time) from what we calculate locally. So after thinking about it: our issue is probably not directly related.

Am Di., 18. Dez. 2018 um 19:13 Uhr schrieb Anand Gaurav < notifications@github.com>:

/cc: @rrelyea https://github.com/rrelyea @jainaashish https://github.com/jainaashish

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NuGet/Home/issues/5602#issuecomment-448316790, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNNi6mPk2xWQXvG_0gnqM0wa5fHxxks5u6TBZgaJpZM4OYukQ .

anangaur commented 5 years ago

@jainaashish The package in question is System.Data.Common. Something to do with Fallback folder inconsistency?

Flavien commented 5 years ago

Diffing my lock files between Windows and Ubuntu, the other packages that end up with a different hash:

jainaashish commented 5 years ago

ContentHash will only differ when there are actually multiple copies of this package across these systems which could be because of the different NuGet feeds configured across these systems or different copies in global packages folder (%userprofile%/.nuget/packages) or dotnet fallback folder (~dotnet/sdk/NuGetFallbackFolder)

anangaur commented 5 years ago

@Flavien if you delete the FallbackFolder on both Dev machine and the Build machine, restore should work fine as the contentHash would be for the package from nuget.org which should be same. Can you try once?

Flavien commented 5 years ago

@anangaur I tried and that fixed the issue.

anangaur commented 5 years ago

Thanks @Flavien for validating the hypothesis. We still need to fix this issue: #7651

springy76 commented 5 years ago

Will the content hash also be different between a W10 dev machine and a W7 build machine? I already cleared all nuget caches and removed that FallbackFolder everywhere but it still fails with hash mismatch.

Flavien commented 5 years ago

What I don't understand is how can the same package with the same version number end up with two different hashes? To me, that is the root cause of the bug.