[Feature] Allow developers to opt-in for SemVer-compatible dependency resolution

JonDouglas commented 2 months ago

NuGet Product(s) Involved

NuGet SDK

The Elevator Pitch

NuGet is one of the few modern package managers that does not allow developers to automatically resolve dependencies based on Semantic Versioning (SemVer) compatibility. Many other package managers, such as npm and cargo, automatically install the latest available version within a SemVer-compatible range for both top-level and transitive dependencies.

I propose that NuGet offer developers an option to opt-in to a SemVer-compatible resolution. This would allow:

Automatic installation of the latest versions of dependencies that are compatible with a defined SemVer range.
Consistent behavior with other package managers that already follow this approach.
More predictable updates, ensuring smoother integration of transitive dependencies as well.

Benefits:

Reduces the burden on developers to manually update packages while ensuring compatibility.
Addresses many of the common challenges related to dependency resolution in NuGet, improving developer experience.

In addition, new shorthand syntax such as ^X.Y.Z would help with consistency of other paradigms.

Related to: https://github.com/NuGet/Home/issues/5553 (But starting a new issue as there are over a hundred comments and this is a clearer definition of what to do)

Expectation:

I can opt-in via some property like <NuGetResolutionMode> or <EnableSemVerResolution>

I can use caret syntax ^ for SemVer compatible updates i.e.

<PackageReference Include="Newtonsoft.Json" Version="^12.0.0" />

I can use tilde syntax for minor version locking i.e.

<PackageReference Include="Newtonsoft.Json" Version="~12.0.0" />

I can use the existing range syntax for explicit range >=, <= i.e.

<PackageReference Include="Newtonsoft.Json" Version="[12.0.0, 13.0.0)" />

Behavior Expectation:

Top-level and transitive dependencies should be updated within the allowed SemVer compatible ranges automatically.

Any direct or transitive dependency within the defined SemVer range should be installed or updated to the latest compatible version when running restore or other nuget installation operations.

If users prefer to use the old behavior for compatibility reasons / exact versioning, they can do so by keeping the mode or disabling the property.

Example Cases:

NuGet

Exact Version: When you specify an exact version, NuGet will resolve that specific version only.
`<PackageReference Include="Newtonsoft.Json" Version="12.0.3" />`

Version Range: You can specify a range of acceptable versions using existing syntax.
`<PackageReference Include="Newtonsoft.Json" Version="[12.0.0, 13.0.0)" />`

Newly Proposed:

Caret Syntax: Automatically allows updates within the same major version.
`<PackageReference Include="Newtonsoft.Json" Version="^12.0.0" />`
- Allows updates from 12.0.0 to any version less than 13.0.0 (e.g., 12.0.4 or 12.1.0)

Tilde Syntax: Automatically allows updates within the same minor version.
`<PackageReference Include="Newtonsoft.Json" Version="~12.0.0" />`
- Allows updates from 12.0.0 to 12.0.x but not 12.1.x (e.g., 12.0.3 to 12.0.4, but not 12.1.0).

Cargo (comparison)

Exact Version: Cargo will resolve the exact version specified.
[dependencies]
serde = "1.0.104"

Caret Syntax (default): Cargo automatically interprets version specifications as allowing updates within the same major version (SemVer compatible).
[dependencies]
serde = "1.0"
- Equivalent to serde = "^1.0"
 
Tilde Syntax: Allows updates within the specified minor version.
[dependencies]
serde = "~1.0.104"

Version Range: You can specify a range of acceptable versions using existing syntax.
[dependencies]
serde = ">=1.0.104, <2.0.0"

Please 👍 or 👎 this comment to help us with the direction of this feature & leave as much feedback/questions/concerns as you'd like on this issue itself and we will get back to you shortly.

Thank You 🎉

bording commented 2 months ago

Given that PackageReferences are also used to determine the dependencies when authoring a package from a project, how would this new feature and syntax interact with producing a package? Would NuGet packages learn about the ^ and ~ syntax so that we'd see those used on the actual package dependencies?

SymbioticKilla commented 2 months ago

@JonDouglas Hi, nice! Just want to check if your description supports this scenario? SemVer resolution is enabled. All direct dependencies in my project have strict versions. All transitive deps(whole tree recursive) use latest version if maintainer of dependency uses >=

JonDouglas commented 2 months ago

Given that PackageReferences are also used to determine the dependencies when authoring a package from a project, how would this new feature and syntax interact with producing a package? Would NuGet packages learn about the ^ and ~ syntax so that we'd see those used on the actual package dependencies?

NuGet generates a .nuspec. If you use that syntax, it would imply the dependencies would need to be expressed similarly. NuGet could do this easily akin to:

^X.Y.Z would be treated as [X.Y.Z, X+1.0.0)
~X.Y.Z would be treated as [X.Y.Z, X.Y+1.0.0)

@JonDouglas Hi, nice! Just want to check if your description supports this scenario? SemVer resolution is enabled. All direct dependencies in my project have strict versions. All transitive deps(whole tree recursive) use latest version if maintainer of dependency uses >=

Yes. If you specify something like Version="X.Y.Z" it will stay locked. Transitives that have a specified version range would automatically update to the latest/newest compatible version in that range.

General comments

I'm curious of what people think about this in general and if other prior art (cargo, yarn, go, etc) do this really well and should be something we consider in this feature request.

Matheos96 commented 2 months ago

I like the suggestion but a few questions and comments :)

I may have missed it, but how would this work with existing syntax for transitive dependencies? Currently most nuget packages probably specify an exact version for their own dependencies. Would opting in to this feature allow current syntax to be interpreted as ^X.Y.Z in transitive dep resolution? That I guess would solve the issue we discuss in #13771. If package maintainers would themselves need to react and opt in for this feature and use the new syntax, I guess our issue in the other thread would remain for packages which are no longer actively maintained but are still worth using.

Additionally, I would just want to re-iterate my comment from #13771, regarding the importance of (potentially?) forcing a lock file to be used together with features like this. Much like npm and cargo already do today. I think reproducible builds is a topic which should not be overlooked.

JonDouglas commented 2 months ago

I like the suggestion but a few questions and comments :)

I may have missed it, but how would this work with existing syntax for transitive dependencies? Currently most nuget packages probably specify an exact version for their own dependencies. Would opting in to this feature allow current syntax to be interpreted as ^X.Y.Z in transitive dep resolution? That I guess would solve the issue we discuss in #13771.

Additionally, I would just want to re-iterate my comment from #13771, regarding the importance of (potentially?) forcing a lock file to be used together with this feature. Much like npm and cargo already do today. I think reproducible builds is a topic which should not be overlooked.

I don't think opt-in to this feature would change how existing packages with exact versions are handled. If desired, NuGet could provide a means of how package maintainers choose how consumers could/should interpret these.

100%. This direction would require lock files to be a key experience as they are in cargo/npm/go/etc.

0xced commented 2 months ago

Something to keep in mind when designing this new feature: not all packages are following SemVer.

Two examples coming to mind:

The GiGraph author admit to not care about backward compatibility.

So far I never cared about backward compatibility. It would cause me a lot of troubles trying to give the library the current shape. I would just remove the obsolete code ;). Especially that I believe that these pieces of code won't affect anyone.
The MongoDB C# Driver (45K downloads per day) has introduced new methods on public interfaces in patch updates, breaking implementers. And you get System.TypeLoadException at runtime after upgrading a patch version. 😱 It was caught by automated tests but it's still pretty annoying.

We are trying to minimize the breaking changes in our driver, with majority of them being additive changes. But in some cases breaking changes are required, like adding a new interface member.

For cases like this we will consider using default interface methods for newer target frameworks in the future.

malopgrics commented 2 months ago

Something to keep in mind when designing this new feature: not all packages are following SemVer.

I don't think that's really a problem as long as it is clearly stated by the package author. Packages author should take this into account when they determine version range of their dependencies, and in a case like this it should be a fixed specific version. At least this is possible with npm.

SymbioticKilla commented 2 months ago

Something to keep in mind when designing this new feature: not all packages are following SemVer.

Two examples coming to mind:

The GiGraph author admit to not care about backward compatibility.

So far I never cared about backward compatibility. It would cause me a lot of troubles trying to give the library the current shape. I would just remove the obsolete code ;). Especially that I believe that these pieces of code won't affect anyone.

The MongoDB C# Driver (45K downloads per day) has introduced new methods on public interfaces in patch updates, breaking implementers. And you get System.TypeLoadException at runtime after upgrading a patch version. 😱 It was caught by automated tests but it's still pretty annoying.

We are trying to minimize the breaking changes in our driver, with majority of them being additive changes. But in some cases breaking changes are required, like adding a new interface member. For cases like this we will consider using default interface methods for newer target frameworks in the future.

I think there should be a an option to update just one transitive dep. I can update for example just one npm transitive to update just specific dep in package lock file. Package.lock.json should stay remain and be able to be updated granularly.

JonDouglas commented 2 months ago

Just to provide transparency here based on NuGet Insights. This is how well NuGet.org conforms to SemVer:

Column	Value
TotalVersionCount	9565817
OriginalIsNormalized	9423398
HasFourthDigit	1882329
IsSemVerCompliant	7545561
IsSemVerCompliantSample	Microsoft.Extensions.Primitives 9.0.0-rc.1.24431.7
OriginalIsNotNormalizedSample	System.Text.Json 2.0.0.0
HasFourthDigitSample	System.Text.Json 2.0.0.9
OriginalIsNormalizedPct	98.51116742040957
HasFourthDigitPct	19.67766056992309
IsSemVerCompliantPct	78.88046572498722

In other words, SemVer compatibility is ~79% of total package versions at this point of time (September 2024).

You can also see that ~20% have a fourth digit. Other reasons might be: non-normalized versions, legacy versioning schemes, pre-release versions in non-standard ways, special characters, or are incomplete.

I hope this data helps guide the discussion further on this feature.

fowl2 commented 2 months ago

Given that PackageReferences are also used to determine the dependencies when authoring a package from a project, how would this new feature and syntax interact with producing a package? Would NuGet packages learn about the ^ and ~ syntax so that we'd see those used on the actual package dependencies?

NuGet generates a .nuspec. If you use that syntax, it would imply the dependencies would need to be expressed similarly. NuGet could do this easily akin to:

There's an argument that library packages should build against the lowest version, to ensure that they actually do work with that version. I guess you'd just not set this property on libraries / use a conditional in .props? This opt-in for consumers would be great to stop the library upgrade cascade treadmill, but it's also nice to have some way to indicate that a library has been tested (or at least builds) with a particular version. 1 version number/spec is not enough! The combinatorial explosion is rough and not particular well supported by the tooling.

But at least we'd be at parity with the other ecosystems.

To cover the "required feature added in a patch version across multiple version branches" thing, can we include multiple version/ranges? Not sure if anywhere else supports that or they just require the whole tree to maintain branches too. ie.

Dep 1.0 Dep 2.0 // new feature

MyProject 1.0 -> doesn't depend on new feature, works with both 1.0 and 2.0: "^1.0"

Dep 1.0.1 // old branch with patch Dep 2.0.1 // new feature with MyProject 1.0.1 -> Depends on "Dep" with patch, but not the new feature: "^1.0.1;^2.0.1"

Again a problem here is the tooling doesn't make it easy even to, ie. build against the first and then run tests against both.

(and please add another voice indicating that package.lock.json should be implied on by this feature)

glen-84 commented 2 months ago

On the subject of lock files, please consider addressing #12409 as part of this task (if not earlier).

rcollina commented 3 weeks ago

Have you considered including an attribute next to Version that'd behave like global.json rollForward?

Might be just me but I see some similarities and the shorthand syntax isn't immediately clear (my 2c of course).

NuGet / Home

[Feature] Allow developers to opt-in for SemVer-compatible dependency resolution #13779

NuGet Product(s) Involved

The Elevator Pitch