search

NuGet / Home

Repo for NuGet Client issues
Other
1.5k stars 253 forks source link

[PackageReference] Centrally manage NuGet package versions for a solution or a repo #6764

Closed anangaur closed 2 years ago

anangaur commented 6 years ago

PackageReference enhancements are summarized as part of this Epic issue: https://github.com/NuGet/Home/issues/6763

bording commented 6 years ago

@anangaur Have you seen https://github.com/Microsoft/MSBuildSdks/tree/master/src/CentralPackageVersions? This seems like it enables the kind of control this requirement is talking about.

anangaur commented 6 years ago

@bording true that. Thanks. @jeffkl @jainaashish we should sync up on this

jainaashish commented 6 years ago

pretty neat, just wondering how does it work out with VS, if it all it does?

anangaur commented 6 years ago

Lets see how can we make this mainstream in NuGet and if satisfies all the requirements.

terrajobst commented 6 years ago

Do we have a strawman for this, design-wise? I'm not sure where that information would be when this is supported across repo boundaries (as the title suggest we do). At that point, this looks like a feature of the feed. It's almost like we'd like to have snapshotted feeds at that point.

anangaur commented 6 years ago

@terrajobst No designs as such. My current thinking right now is to be able to define it per solution level in a file. And sometimes instead of defining it in the file, it could be imported from some global file present anywhere accessible by NuGet/MSBuild.

The purpose of keeping the term repo is to help not constrain anything for a 'solution' level. There would be some cases where an app is distributed across solutions in a repo.

I agree that if this file starts being a list all versions of packages allowed across a team/product - its becomes similar to just being a curated feed/repository.

terrajobst commented 6 years ago

If we had better tooling for maintaining/populating feeds, creating snapshotted feeds is probably the most logical model. And the file that people need to check in is a nuget.config with the proper feed :-)

tebeco commented 4 years ago

Hello @anangaur @nkolev92

As i changed computer and try to find the docs for Directory.Packages.props i found both :

is the Dup' intended ?

Also i spotted this in the Wiki page : (Sorry the zoom) image

but i have no idea where to submit an issue/PR for the Wiki part

nkolev92 commented 4 years ago

As i changed computer and try to find the docs for Directory.Packages.props i found both :

It is expected. This is tracking an implementation with similar functionality as the SDK you linked.

Also i spotted this in the Wiki page :

I edited the wiki to point back to this issue because that spec is just a more elaborate version of the one linked in this issue. :) Lots of people came across the spec you linked but could never figure out the right issue back.

tebeco commented 4 years ago

not sure i understand the wiki part of your comment. Is it normal to contains twice "github.com" and so that when you click, you are redirected to the wiki home instead of the issue ?

nkolev92 commented 4 years ago

oh, I totally missed that. my bad :) Fixed now :)

npehrsson commented 4 years ago

Any ETA for this?

ds1709 commented 4 years ago

Maybe it will be usefull. I have project with two depencies, PackageA and PackageB. They in turn depends (transitive) from Newtonsoft.Json 8.0.1 and Newtonsoft.Json 9.0.1. Because of this, from build to build I have different version of Newtonsoft.Json in output directory. There's more fun when I add package reference on Newtonsoft.Json 12.0.1 directly in project. In this scenario I need maximum of package version control.

tebeco commented 4 years ago

@ds1709, As of today you can just use MsBuild variables for that, There're 2 possible ways since MsBuild 15.x (2017) :

I have created a repository to anticipate the Centralized Nuget Package here: https://github.com/dotnet-experimentations/CentrallyManagingNuGetPackageVersions

It's a bit long to explain but basically:

=> this mean you could end up with $(TargetFramework) being unset => empty string if you attempt to use them from Directory.Build.props => this is why the <project include="dependencies.props" /> is done in Directory.Build.targets

The approach with Target+Update on the other hand, needs value to be Include(d) first to be updated => The line order is important (at the end of the files)

What I mean by Include/Update + Line Order is how ItemGroup works You need to "Include" something, to be able to match it in a later Update :

<ItemGroup>
  <YourItem Include="MATCH A NAME HERE" OneProps="OneValueOnInclude" />

  <YourItem Update="MATCH A NAME HERE" OneProps="The Later Updated Value" />
</ItemGroup>
stevenbrix commented 4 years ago

Will the VS support for this work for any SDK style project? I know there is no SDK-style for C++ projects, but we are thinking of rolling our own, and would like to make sure that experiences like this will still work

MidasLamb commented 4 years ago

Anything I can do to help with this? I'm very interested in this approach and it would help me out immensely if this gets added.

tebeco commented 4 years ago

if possible that would be awesome if dotnet tool restore and dotnet restore would be able to read RestoreSources from that file too As we are building sources code from various places :

dotnet restore seems to properly loads Directory.Build.xxxx which allow us to do :

<RestoreSources Condition="''$(TF_BUILD)' == 'true'">$(RestoreSources);https://feed/from/aks/network</RestoreSources>
<RestoreSources Condition="''$(TF_BUILD)' == 'true'">$(RestoreSources);https://feed/from/lan/or/vpn</RestoreSources>

Why ?

Because nuget.config is static and url are static/hardcoded

The only alternative is --add-source but that's just the same problem again, while evrything exists in MsBuild, we now have to write a ps1 or sh script to do the if/else to use --add-source

It seems that a Directoy.Packages.props would be a the place to fully replace nuget.config

nkolev92 commented 4 years ago

@tebeco Directory.Packages.props is just another msbuild file. It'll be imported in every project and evaluated in every project individually. RestoreSources will continue to work the same way it has so far.

RestoreSources has its limitations though: https://github.com/NuGet/Home/issues/5321 & the fact that there's no tooling to edit it, so keep that in mind.

tebeco commented 4 years ago

I suppose I should open an issue on dotnet/sdk so that we could properly use dotnet tool restore with <RestoreSources> ? We can't reach nuget.org from the LAN (one CI is on the LAN) so we uses Artifactory as a "Remote/Proxy feed" We also have a CI in AKS and on this side we cannot reach the LAN (on purpose, security reason), but we can reach nuget.org

the goal is to be able to do that:

git clone ...
dotnet tool restore
dotnet restore

For that dotnet tool restore to work, well, it requires that the RestoreSources are "swapped", this works like a charm for dotnet restore and i can't add a nuget.config as it would have to manually "delete/rename" files right after a clone, with is far form ideal

Do you have idea to solves that ? I'm just hopping to know if I could replace this :

$NugetSource = "https://lan/url/to/remote/nuget
if($env:TF_BUILD -eq 'true')
{
  $NugetSource = "https://api.nuget.org/v3/index.json"
}

dotnet tool restore --source $NugetSource
nkolev92 commented 4 years ago

dotnet tool restore is restore based but is not be considered project based. Nothing in msbuild truly applies there.

The SDK repo would own that discussion though.

anangaur commented 4 years ago

We do want to think in the direction where you should be able to define sources in this directory.packages.props file. However, as of now, our focus is towards finishing this feature first before venturing on some of the next pieces.

nkolev92 commented 4 years ago

Just expanding on @anangaur's point.

Directory.Packages.Props is still MSBuild, so configuration wise everything that works today, will work with centrally managed package versions. You just will not be able to manage it through the UI. That's covered https://github.com/NuGet/Home/issues/5321.

tebeco commented 4 years ago

thx for the link, it confirms our current use of <RestoreSource> :) I'll open an issue on dotnet/sdk for dotnet tool to check why/if it can properly load Directory.Build.props

idlem1nd commented 4 years ago

Is there anything we can do to assist with this? We have a hand-rolled targets file that achieves a similar thing, but would prefer to stay with a vanilla usage of NuGet if we can. Is there any ETA for landing the work done so far?

kant2002 commented 4 years ago

@idlem1nd depends on your usage. You can use currently proposed design inside VS 16.6 All you need it to create Directory.Packages.props and add property <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>. I add that to Directory.Build.props but maybe you can do that on per-project basic to migrate step-by-step.

idlem1nd commented 4 years ago

@kant2002 thanks but I think that forces all packages to be managed centrally, correct?

nkolev92 commented 4 years ago

@idlem1nd

That's the current design of the feature.

You either manage all or none.

Refer to the spec linked in the issue body. There are additional specs linked in that spec.

tebeco commented 4 years ago

If you want to use it right now and only partially you can use something like this: https://github.com/dotnet-experimentations/CentrallyManagingNuGetPackageVersions in the WithMsBuild folder

Currently the folder WithDirectoryPackageProps is broken, I created it, waiting for this feature to be delivered so that I can toy with it before switching

reimer-atb commented 4 years ago

Hi. I have been pointed to this issue from here: https://github.com/dotnet/sdk/issues/11464

I tried following the discussion here but some things are still not clear to me:

Would what is proposed here be a solution to what I was asking for in the issue mentioned above?

And what is the status of this issue? It's been open for two years and based on some of the comments here there seems to be some kind of usable implementation but it's not clear in what version of dotnet core or nuget. Directory.packages.props is mentioned several times but I wasn't able to find any documentation about that in Microsoft's docs.

zivkan commented 4 years ago

@philipreimer the feature is still under development. Since the feature is incomplete, the parts that have already been shipped are considered preview, and therefore we haven't documented it yet.

batzen commented 4 years ago

@philipreimer You need the core-sdk 3.1.300 and visual studio 2019.6. The wiki page was enough documentation for me to get started. The only thing not mentioned there is that you currently have to set ManagePackageVersionsCentrally to true to enable everything.

reimer-atb commented 4 years ago

@batzen that would be this wiki page, correct?

I am on Linux - so no VS - but I should be fine with just the dotnet CLI I assume. If I understand the wiki page correctly. So it should also work in a CI environment?

.NET Core SDKs installed:
  3.1.103 [/usr/share/dotnet/sdk]
reimer-atb commented 4 years ago

@zivkan any ETA when this feature could be final? Is this planned for a specific upcoming version of dotnet core?

batzen commented 4 years ago

@philipreimer it should also work in CI. It works in our CI system. But you need the 3.1.300 sdk. You seem to be using 3.1.103.

tfabian commented 4 years ago

Hi, I figured out this "work around" in our huge repository: https://developercommunity.visualstudio.com/content/idea/1019345/nuget-multi-solution-package-references-1.html

In this way I can manage our nuget package references in one place.

PaulVrugt commented 4 years ago

So, what can we do to get this thing moving? We've been waiting for this over 2 years

batzen commented 4 years ago

@PaulVrugt it works since sdk version 3.1.300. Only Visual Studio has no UI support for it yet.

nkolev92 commented 4 years ago

Just a plug here to update to the latest 3.1.400 SDK/16.7. There are a few fixes in the latest version relating to CPVM.

moly commented 4 years ago

Should this feature also centralise lock files? I've set up Directory.Packages.props, but when I run dotnet restore --use-lock-file it still creates separate packages.lock.json files in each of the project folders, which is not what I'd expect.

tebeco commented 4 years ago

I think there are 2 completely different feature and then one should not affect the other so I would not expect this to more or less change anything at how lock files behave today I think the unified solution lock file is on the table since December 2018 IIRC, but not sure when it will land

moly commented 4 years ago

@tebeco thanks. One other thing, the wiki says that you should be able to add packages through the dotnet cli: https://github.com/NuGet/Home/wiki/Centrally-managing-NuGet-package-versions#dotnet-cli-experience

However when I try this it adds the package versions to the .csproj instead of the Directory.Packages.props, which causes an error on restore. I guess this isn't implemented yet either and we have to manually edit the Directory.Packages.props ourselves for now?

tebeco commented 4 years ago

Well, that's a good question (i'm not part of Nuget team / or Msft) I would guess they'll ask you which specific SDK are you using right now ? I did not test if that worked on my "POC" repo I think I broke it this weekend when trying to update it

nkolev92 commented 4 years ago

@moly

The overall feature is still being worked on. The dotnet.exe commands are not implemented yet, you can track that here: https://github.com/NuGet/Home/issues/9014.

ps-weber commented 3 years ago

When packages.lock.json came out in 2018, a central lock file was suggested: https://devblogs.microsoft.com/nuget/enable-repeatable-package-restores-using-a-lock-file/#solution-or-repo-lock-file

However, I no longer see this concept mentioned in the spec. Is this still planned?

tebeco commented 3 years ago

hello @ps-weber

This is unrelated to this issue. Your question is about lockfile while this issue is about "Centrally Managing Package" at once for a solution

Each feature can work without the other I suggest opening a new issue to avoid the confusion here

ps-weber commented 3 years ago

@tebeco Thanks for the quick answer, it seems I misunderstood the article. I think the author suggests the Directory.Packages.props itself could function as a "central lock file". If I understand the spec correctly, this is not the case.

I also came across an (outdated?) article, which links to this issue and suggests a central package.lock.json: https://github.com/NuGet/Home/wiki/Centrally-managing-NuGet-packages

Maybe it should be updated or marked as outdated?

tebeco commented 3 years ago

No worries

Regarding this thread, as @nkolev92 said few message earlier it's being reworked ;)

that's why this page shows it as "reviewing" https://github.com/NuGet/Home/wiki/Centrally-managing-NuGet-package-versions

stevenbrix commented 3 years ago

Will this be the default behavior for VS/Nuget management of packages? It would be really nice if this was done automagically for you when using VS to manage nuget packages

tebeco commented 3 years ago

I'll let Nuget team answer correctly there. But here is what you can already observe today :

roji commented 3 years ago

I gave this a try, and it's working well using dotnet SDK 5.0.0-rc2. However, when running dotnet msbuild (as opposed to dotnet build) it doesn't get picked up, and I'm getting "X does not contain an inclusive lower bound ...". The reported MSBuild version is 16.8.0-preview-20475-05+aed5e7ed0.

Am I doing something wrong, or is this an issue that needs to be raised with MSBuild or something?