PackageReference should support DevelopmentDependency metadata

[AArnott]

msbuild /t:pack fails when building a stable versioned package that depends on an unstable one, even though that unstable one is a DevelopmentDependency:

1. Create .NET Core console app
2. Add this package reference

   <PackageReference Include="Nerdbank.GitVersioning" Version="1.5.28-rc" developmentDependency="true" />`

3. Pack the project with msbuild /t:pack

Build fails with a crashed MSBuild task with this message:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Sdks\NuGet.Build.Tasks.Pack\buildCrossTargeting\NuGet.Build.Tasks.Pack.targets(73,5): error MSB4018: The "PackTask" task failed unexpectedly.\r [C:\Users\andre\OneDrive\Documents\Visual Studio 2017\Projects\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj] C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Sdks\NuGet.Build.Tasks.Pack\buildCrossTargeting\NuGet.Build.Tasks.Pack.targets(73,5): error MSB4018: System.IO.InvalidDataException: A stable release of a package should not have a prerelease dependency. Either modify the version spec of dependency "Nerdbank.GitVersionin g [1.5.61-ga5f362ccc1, )" or update the version field in the nuspec.\r [C:\Users\andre\OneDrive\Documents\Visual Studio 2017\Projects\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj]

In fact I shouldn't even have to specify DevelopmentDependency="true" in my reference because the nuspec of the referenced package itself has that property set.

[davidfowl]

You're aiming to have this package excluded from the output package right?

[AArnott]

Correct: no dependency, no embedding. And thus no restriction on using unstable packages while building a stable one.

[emgarten]

PrivateAssets is the equivalent.

<PackageReference Include="Nerdbank.GitVersioning" Version="1.5.28-rc">
   <PrivateAssets>all</PrivateAssets>
</PackageReference>

[AArnott]

Thanks @emgarten. Should that be a default then when the package's own nuspec sets DevelopmentDependency=true?

Also, I'm guessing that's not quite equivalent to what we have in project.json (in an MSBuild project). I can add a package dependency to a developmentDependency package and that propagates across project references (as I usually want), yet that doesn't limit my ability to compile those projects into stable packages that don't depend on it. Your PrivateAssets=all metadata suggests to me that it won't propagate across P2Ps.

[clairernovotny]

I need this as well for GitVersionTask for the same reason. The GitVersionTask nuspec has developmentDependency set to true.

When a project references it, it should default the <PrivateAssets>all</PrivateAssets> metadata on that package ref...or whatever is needed to ensure it doesn't get added to the generated nuspec with /t:pack

[davidfowl]

@emgarten we need 2 things:

• We need a way for a package to express what assets get exported when referenced in a project .e.g If I want all projects that reference me to have effectively <PrivateAssets>all</PrivateAssets> by default then I should be able to express that in the nuspec.
• We should infer developmentDependency set to true as implicitly saying <PrivateAssets>all</PrivateAssets> .

[AArnott]

Can we at a minimum for Dev15 RTW get developmentDependency=true in a nuspec file to be honored by having the dotnet SDK project type not propagate the dependency in nuget packages it packs?

[AArnott]

@davidfowl points out that PrivateAssets=all at the consuming side should workaround this issue. This is also consistent with the documentation. But I tried it, and the dotnet pack step still emits a package dependency on it (with the default exclude="Build,Analyzers" attribute no less).

[rohit21agrawal]

@AArnott pack should not emit a dependency when <PrivateAssets>all</PrivateAssets> is specified. Did you make sure you restored after you added the <PrivateAssets> tag to your dependency? Pack relies on reading the dependencies and their metadata from the assets file, so if the assets file doesn't reflect your changes, pack wouldn't know about them.

[AArnott]

Thanks, @rohit21agrawal. That was the trick. I was only running dotnet pack after my change rather than a restore as well.

[rohit21agrawal]

Great! Are we good to close this issue now?

[AArnott]

No. There's still the matter of developmentDependency being ignored that should have set PrivateAssets=all by default, which @davidfowl seems to agree with as well.

[kzu]

We're seeing this as well in .NETCore/.NETStandard projects. Adding a package reference to a development dependency does not add the <PrivateAssets>all<PrivateAssets metadata.

Repro:

1 - Create a .NETCore/.NETStandard project 2 - Add a package reference (via the NPM) to NuGet.Build.Packaging (which has developmentDependency=true)

Result: PackageReference does not have PrivateAssets metadata Expected: same behavior as packages.config, which adds the reference as <package id="NuGet.Build.Packaging" developmentDependency="true" ...

This is a regression IMHO.

/cc @adalon @mrward @mhutch

[alpaix]

Bug 389581:[Feedback] Installing NuGet package that is marked as a development dependency doesn't add the PrivateAssets="All" attribute to the PackageReference element

[davidfowl]

Should this really be an install time action? Shouldn't it be part of the task?

[clairernovotny]

Agree with @davidfowl. This shouldn't be an install-time action, the task should set it when it reads the package meta-data during restore. A user may choose to override this by explicitly setting the metadata on PackageRef, but not sure why anyone would really do that.

[emgarten]

Shouldn't the user be able to control if the package flows transitively or not? If this isn't an install-time action they wouldn't be able to do this.

[AArnott]

I think @davidfowl and @onovotny's point is the default behavior for a dependency on a package that sets developmentDependency=true is PrivateAssets="all" and therefore no user action and no extra project xml is required to do the commonly correct thing. @emgarten This isn't to say that the user can't override that by explicitly specifying a PrivateAssets value. So yes, the user can still control whether the package flows, if they want to.

[emgarten]

Currently there are no scenarios for restore where a package can modify how it was referenced (the incoming edge of the graph) because this can require re-walking everything.

For example if a package has a mix of versions, some with developmentDependency true and others with false. If at first a version with dd false is selected the packages flows, but then that tool package is bumped up to a higher version with dd set to true by another package in the graph the parent project needs to re-walk the graph since dependencies of the tool package could have bumped other packages also and suddenly they are no longer flowing.

The ideal solution for NerdBank.GitVersioning in my opinion is that build time packages like this would be installed under a different name similar to DotnetCliToolReference, the targets files would be referenced and the package and any dependencies would be outside of the project's dependency graph since it is only part of the build. That is a much different change however.

[AArnott]

Ok, install time it is. I like your idea about a new msbuild item name and dependency graph. I talk about that here https://github.com/Microsoft/msbuild/issues/1756

[davidfowl]

Currently there are no scenarios for restore where a package can modify how it was referenced (the incoming edge of the graph) because this can require re-walking everything.

It would happen at restore time. That package metadata should treat it like PrivateAssets="all". We should just get rid of this metadata since it means really nothing.

[mariusGundersen]

This is not limited to msbuild references (like https://github.com/Microsoft/msbuild/issues/1756 is limited to), it also applies to nugets that contain only Analyzers (like the StyleCopAnalyzers project).

[tofutim]

How does PrivateAssets get set? I need this to be in GitVersionTask and Fody/PropertyChanged which currently get brought in by 'pack' in VS2017. Can I do it from my end or do I need to get the nuspecs for GitVersionTask and Fody/PropertyChanged to be changed?

I just tried changing my csproj to add "PrivateAssets" per https://github.com/NuGet/Home/wiki/PackageReference-Specification

  <ItemGroup>
    <PackageReference Include="GitVersionTask" Version="3.6.5" >
      <PrivateAssets>contentfiles;analyzers;build</PrivateAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.AspNet.SignalR.Client" Version="2.2.2" />
    <PackageReference Include="PCLSpecialFolder" Version="1.1.0" />
    <PackageReference Include="PCLStorage" Version="1.0.2" />
    <PackageReference Include="PropertyChanged.Fody" Version="2.1.1" >
      <PrivateAssets>contentfiles;analyzers;build</PrivateAssets>
    </PackageReference>
  </ItemGroup>

but PropertyChanged.Fody and GitVersionTask still get propagated when I pack. I should add that I am not using net core. This is a netstandard13 portable library initiated in VS2017.

    <dependencies>
      <group targetFramework=".NETStandard1.3">
        <dependency id="PCLStorage" version="1.0.2" exclude="Build,Analyzers" />
        <dependency id="PCLSpecialFolder" version="1.1.0" exclude="Build,Analyzers" />
        <dependency id="PropertyChanged.Fody" version="2.1.1" exclude="Build,Analyzers" />
        <dependency id="GitVersionTask" version="3.6.5" exclude="Build,Analyzers" />
        <dependency id="NETStandard.Library" version="1.6.1" exclude="Build,Analyzers" />
        <dependency id="Microsoft.AspNet.SignalR.Client" version="2.2.2" exclude="Build,Analyzers" />
      </group>
    </dependencies>

Why? Do I need a fresher version of nuget? (using VS2017)

Update. The syntax is wrong and this spec is inaccurate - https://github.com/NuGet/Home/wiki/PackageReference-Specification

This works...

  <ItemGroup>
    <PackageReference Include="GitVersionTask" Version="3.6.5" PrivateAssets="All" />
    <PackageReference Include="Microsoft.AspNet.SignalR.Client" Version="2.2.2" />
    <PackageReference Include="PCLSpecialFolder" Version="1.1.0" />
    <PackageReference Include="PCLStorage" Version="1.0.2" />
    <PackageReference Include="PropertyChanged.Fody" Version="2.1.1" PrivateAssets="All" />
  </ItemGroup>

[mariusGundersen]

This is still a problem in the 2.0 release (released today). Neither VS15.3 nor dotnetcli 2.0.0 (with nuget cli 4.3.0.5) add PrivateAssets="all" when installing (for example) StyleCop.Analyzers v1.0.2.

[AArnott]

@mariusGundersen This issue is still active. So I don't think anyone was expecting or claiming that the problem would be fixed in this release.

[RamjotSingh]

This becomes a serious problem when you are trying to generate a package and have added StyleCop to the project. Since Stylecop is not added as a developmentDependency the resulting nuget package declares it as a required dependency which should never be the case.

[jainaashish]

I'll pick this up and first thing we'll make sure is if package has set DevelopmentDependency then respect that while installing as PackageReference and set PrivateAssets to All

[clairernovotny]

@jainaashish Are you talking about what the install gestures/UI does or how it generates the lock file? Ideally, I would like to see this value put into the lock file based on the nuspec metadata even if the PackageReference doesn't have anything. "Just" read the .nuspec from the cache directory on restore and determine if it's a development dependency on the fly?

[jainaashish]

I'm talking about the install gesture which will make it flow to lock file. Just changing lock file might not feasible since it will make it hard for users to override.

[clairernovotny]

Overriding could be still done if PrivateAssets is explicitly specified?

[bording]

I'm with @onovotny on this one. I'd much rather have DevelopmentDependency mean that the default PrivateAssets value be all instead of none for that package.

Then, if I have some reason to override the default, I can add PrivateAssets to the reference and set it to whatever value I need.

The main reason I think this is better is that if I'm adding a package reference by hand, everything would still work properly without me having to know that the package has DevelopmentDependency set, which means I also have to add PrivateAssets to get the package reference correct.

[jainaashish]

I meant you can always change PrivateAssets value on PackageReference, but I see your point as well to not make it explicit and confuse users.

[jainaashish]

I didn't see @emgarten has already commented above regarding this It might work in most cases but if it fails then it will be really hard to figure out. So our best bet would be to make this change at install time and add metadata PrivateAssets to All on PackageReference.

[AArnott]

So our best bet would be to make this change at install time and add metadata PrivateAssets to All on PackageReference.

My concern with this plan is that I personally don't use the UI for adding package references. I add them all manually. For anyone like me, that means they won't necessarily know or think to add PrivateAssets="all" to the tag. But I guess I probably know at the time I hand add that reference that I don't mean for it to propagate, so I guess it's not the end of the world if you go with that design. And besides, I guess there's some value in a visual audit of my PackageReference items being very clear which ones will propagate and which ones won't. If some do and some don't, I can see that being confusing to a lot of people.

[jainaashish]

To update the thread, while designing the proper fix for this issue, another suggestion has come up to add ExcludeAssets as Runtime to truly make this development dependency and only available at compile time. So if a nuspec sets developmentDependency to true then it will set two things to PackageReference a) PrivateAssets=All and b) ExcludeAssets=Runtime

And hopefully we'll be able to make it work for install as well as restore so either you hand edit the csproj file or install via NuGet UI, it will work.

[jainaashish]

So the final design proposal here is, as part of NuGet restore, for each package dependency, we'll evaluate developmentDependency flag and accordingly set it's transitive nature (SuppressParent to All) and exclude runtime (IncludeAssets to compile; build; native; contentfiles; analyzers) in project.assets.json file.

Now while installing through NuGet UI or PMC, or using dotnet add to install new package, these properties will be persisted on PackageReference inside .csproj file like below:

<PackageReference Include="abc" Version="1.0.0">
   <PrivateAssets>all</PrivateAssets>
   <IncludeAssets>compile; build; native; contentfiles; analyzers</IncludeAssets>
</PackageReference>

But for other cases like manually adding package dependency, these will not be persisted on PackageReference item. Rather will be used implicitly and user can still see them in project.assets.json file. Since NuGet itself didn't add this package reference so it won't be able to update it, neither it will be a good idea to update csproj file as part of just restore.

@davidfowl @AArnott @onovotny @bording Let me know for any concerns here. I've also raised a initial PR with the proposed design at https://github.com/NuGet/NuGet.Client/pull/1930

[mariusGundersen]

So if I understand correctly, when I use dotnet add, it's not nuget that is writing to the csproj file, it's something else? So for it to work the same way (adds the correct PrivateAssets and IncludeAssets), something else has to be changed? I'm guessing it would be a good idea to add this same support to that cli tool too, so it behaves the same way. Obviously, like you said, if I edit the csproj directly then nothing should happen on nuget restore.

[clairernovotny]

@jainaashish The only nit I have is that the reference won't be transitive. That means that I have to either use Directory.Build.props, which most people don't really know about, or keep all my development dependencies in sync and added to each project.

Think of analyzers - if I add it to a "common" project, should it be picked up by all projects in the graph that reference it? Do I really have to add it to each project and then ensure the versions are all up-to-date?

[bording]

The only nit I have is that the reference won't be transitive. That means that I have to either use Directory.Build.props, which most people don't really know about, or keep all my development dependencies in sync and added to each project.

@onovotny I don't think there's currently a way to allow the package reference to be both transitive and also exclude it from becoming a dependency on the generated package. PrivateAssets is used for both.

---

@jainaashish So whether or not the package reference has those specific PrivateAssets and IncludeAssets values on it, the resulting values are the same? It feels somewhat strange to me that you'd explicitly add them to the reference in some cases but not others, but the end result is the same set of values. Having them on the package reference implies to me that they need to be there. I'd personally prefer to remove them in all cases, and then they only need to be there if I need to override the default values. I can then look at documentation that explains what the defaults are if I have a scenario where I care.

[mnivet]

@onovotny I'm following to that issue to deploy analyzers settings And even in the classical dotnet environnement I was used to install the analyzers settings on all projects and ensure the versions are all up-to-date. I'm not expecting more "magic" here because it allows us to easily install different analyzers on different types of projects with ease.

For example I don't want the same analyzers on Common project and Common.Tests projects ;) So I'm glad that it's not transitive by default.

[jainaashish]

@mariusGundersen thanks for pointing that out. I got that wrong, we'll take care of dotnet add as well. Updated my comment.

Also wrote a small spec detailing it out - https://github.com/NuGet/Home/wiki/DevelopmentDependency-support-for-PackageReference

[jainaashish]

@bording We understand the concern and confusion here so we decided to not do anything in case reference is added manually. So only when you are writing new PackageReference in project, we'll make sure to consume developementDependency from nuspec and accordingly make the right decision.

[bording]

@jainaashish I'm not sure I'm following what you're saying. My point is that I don't see why you're ever adding PrivateAssets and IncludeAssets on the package references. Your spec document shows that you're handling this in two different ways.

I don't understand the benefit of doing this in two different ways based on how the package reference is added. If you can support it without requiring a specially-authored package reference in some cases, why not do it that way for all cases? Supporting both cases just makes things confusing.

I think the better approach is to never add PrivateAssets and IncludeAssets.

[jainaashish]

We need to set these metadata in order to clearly make it visible to the consumer that why some packages are transitively or runtime available vs some other packages which are not. If there is no clear indication on PackageReference, then it will become more confusing.

About two different ways, IMO its more consistent since whenever we write anything to project, we make sure we provide the correct default behavior with the option to user to overwrite. Where as adding a dependency manually is a special case, and we'd expect that user fully understand what he wants and needs to write there so we don't want to interfere.

[bording]

We need to set these metadata in order to clearly make it visible to the consumer that why some packages are transitively or runtime available vs some other packages which are not. If there is no clear indication on PackageReference, then it will become more confusing.

I don't agree. As a package consumer, I expect the author to have created the package correctly. If I reference it, then the details of the assets settings mostly are just noise in the project file. For me to even realize that runtimes are being excluded, I'd first have to have opened up the package I'm consuming to notice that it had a runtimes folder in it in the first place. (Not to mention, if the author actually marked the package as a development dependency, why is it including a runtimes folder that wouldn't be used?)

Regular package references currently have default values for the various Assets properties, but you don't write those out explicitly when you add a package reference, so I don't see why they should be added for development dependencies either.

It feels like adding the Assets properties to a package reference for a development dependencies is leaking details into the project that aren't actually relevant in the default case.

If you weren't going to support the scenario of it working properly without the Assets properties, then I can see more of an argument for them being there, but given that you have a way to make them work properly without them, then I just don't see why you want to write them in the file.

[bording]

Ultimately, it's the inconsistency that bothers me. For a regular package reference, I could remove any Assets properties it has, and that will result in a change to the project, but for another package reference (this one a development dependency) I could perform the same action of removing the Assets properties and it wouldn't result in a change at all because they've been stamped with the default values for that reference.

Anyway, I'm glad to see that the development dependency flag is being supported at all, even if I'm not 100% on board with the details.

[jainaashish]

When I say available at runtime, it does not only mean runtime folder, rather it means assets available compile time vs runtime. I'd imagine if a author is keeping developmentDependency flag to true in his package, then he meant that this package should only be consumed at compile time or to build but not at runtime.

About regular packages and default values for various assets, then this is what NuGet defines across packages which is why it's not explicitly set on PackageReference but we make sure there is proper documentation for consumers to go read, understand, and change as per their specific needs. Where as developmentDependency is not the same thing, it's defined by authors for selective packages which makes it necessary to be mentioned on reference. otherwise it'll become confusing for consumers.

[jp2masa]

I'd imagine if a author is keeping developmentDependency flag to true in his package, then he meant that this package should only be consumed at compile time or to build but not at runtime.

I think that developmentDependency (in the package nuspec) should be a special case, so that setting PrivateAssets, IncludeAssets or ExcludeAssets in the PackageReference would make no difference. Maybe a new attribute DevelopmentDependency could be set to true in the PackageReference item to make it clear that the package is a development dependency, but it wouldn't be consumed. Anyway, development dependencies are usually build tools or analyzers, and the name of the package usually contains those words, so the package consumer assumes it's a development dependency.

EDIT: Sorry, I guess that I didn't think correctly when I commented here... I agree that developmentDependency set to true should set PrivateAssets to all by default, but there are many cases where it makes sense to let the dependency flow (for example packages which pull a package for libraries and other for build tools, it's a bit common), so it totally makes sense to let the user override that. Also, IncludeAssets or ExcludeAssets may be useful for users who want to exclude some assets that cause some kind of conflicts, and it's good to have those settings.

[dasMulli]

There is a feature now in MSBuild 15.6 that may solve some(!) of the issues of development dependencies: MSBuild SDKs can be added to projects that will not be added to the project's dependency graph but will be available during build.

This allows at least for all build-only projects (e.g. versioning packages) to be added as SDK:

<Project Sdk="My.Versioning.Tasks/1.2.3;Microsoft.NET.Sdk">

This should also work for classic csproj files (non-.net core/standard projects).

This does not allow any other features like passing references to the compiler, analyzers, content files etc. but is a nice way to add build-only features (versioning, code generation etc.) without needing to set PrivateAssets="All".

[clairernovotny]

Looking forward to this!