NuGet pack ignores nuspec versions [unwanted version normalization?]

[timotei]

Hello,

It seems latest NuGet pack ignores the version set in the nuspec (or via the -Version override flag) and the dependencies' versions, by normalising them, even if nobody asked for it :(

It seems this was "fixed" for this https://github.com/NuGet/Home/issues/2039, but to me this seems a super counter-intuitive change, that breaks (at least for us) the daily workflows.

For the meantime, we had to do a workaround to "fix" the versions in the generated nupkg. I'm hoping that this can be fixed either by removing this weird logic, or allowing the nuget pack command to generate nuget package which contains the same versions as in the nuspec.

I'm using NuGet Version: 3.4.4.1321.

With this nuspec:

<?xml version="1.0" encoding="UTF-8"?><package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd">
  <metadata>
    <authors>author</authors>
    <version>1.9.0.0-SNAPSHOT</version>
    <id>mylib</id>
    <dependencies>
      <dependency id="dependency" version="3.16.0.0"/>
    </dependencies>
    <description>mylib</description>
  </metadata>
</package>

executing nuget pack mylib.nuspec -Version 1.9.0.0-SNAPSHOT generates this .nuspec:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>mylib</id>
    <version>1.9.0-SNAPSHOT</version>
    <authors>author</authors>
    <owners>author</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>mylib</description>
    <dependencies>
      <dependency id="dependency" version="3.16.0" />
    </dependencies>
  </metadata>
</package>

As we can see:

• Version is 1.9.0-SNAPSHOT instead of 1.9.0.0-SNAPSHOT (even if we specifically overrode it via the command line) - same happens with a non-SNAPSHOT version
• Even worse, the dependency version is 3.16.0 instead of 3.16.0.0
• Funnily enough, the filename has the full version: mylib.1.9.0.0-SNAPSHOT.nupkg

[emgarten]

@timotei would you share more about your scenarios here? Are you expecting the version string to be an exact match as part of other scripts or tools?

[timotei]

Well, we have a ton of internal libraries that are used throughout the end products (private NuGet repo based on Artifactory). Some scripts/tools generate them directly via the nuget pack, others are using some more legacy build systems and we managed to make some bridge between them (not possible/feasible to migrate them to the newer NuGet-based process).

The issue comes from the fact that newer NuGet automatically normalizes the version, making the interop between the two types of projects hard to almost impossible. Basically, we have interop between Java and .NET libraries (via IKVM), and thus we have a specific format for the versions, which we must respect.

That's why, it's very important (for us and I think in general) that tools that generate stuff, do it in a expected way, based on the .nuspec, because that impacts the entire dependency tree. So, I'd expect the nuget pack step not to change anything that's already set in the nuspec. Or, at least, provide a way to skip that behavior.

[emgarten]

I'd like to understand how this is working, are your scripts resolving nuget dependencies without nuget? Does IKVM consume nupkgs directly? Which parts of your build process read the nupkgs without nuget?

[timotei]

Yeah, up until NuGet 3.4, we had in place a simple mechanism which would translate 1-to-1 from Gradle dependencies, to NuGet packages (which are basically IKVM-compiled assemblies - Java to .NET). We were able to create the packages.config file with the mapped dependencies + versions, then do a nuget restore, use the restored deps in a custom processing step which does the final .NET compilation.

Once NuGet 3.4+ went in, we found the issue that the versions wouldn't be mapped properly, all depending on whether the restored NuGet packages were created with NuGet 3.4+ or not. The thing is, nuget places the restored packages in folder with the full version, not the normalized one, which means, if a package was created with NuGet prior to 3.4 it will have 1.0.0.0, while with NuGet 3.4+, it will be 1.0.0. This means there's suddenly no reliable/deterministic rule for the location of the packages. This issue comes from the fact that nuget pack will normalize the generated version.

[emgarten]

The thing is, nuget places the restored packages in folder with the full version, not the normalized one

This has been fixed in 3.5.0, you can find it here: https://dist.nuget.org/win-x86-commandline/v3.5.0-beta2/NuGet.exe

there's suddenly no reliable/deterministic rule for the location of the packages

With normalization there is only one possible string, making this deterministic now. Paths can be determined upfront without opening every nupkg to check if it is the expected version.

Here is how it works:

1. Versions have three parts to match SemVer
2. If the version contains a 4th non-zero digit, then it is also added since NuGet allowed 4 part versions at the start.
3. Leading zeros are always removed

Examples: 1.0.0.1 -> 1.0.0.1 1.0.0.01 -> 1.0.0.1 1.0 -> 1.0.0 1.0.0.0 -> 1.0.0 1.0.0 -> 1.0.0

[timotei]

Hmm, yeah, but AFAICT, the 3 or 4 digits folder name will depend whether the NuGet package was generated with NuGet <3.4 or NuGet 3.4+, no?

About the normalization part, isn't possible to make nuget pack not normalize the version if explicitly not wanting? Or is the whole 3.4+ Normalized Versions Saga about normalizing the behavior (pun intended) of all NuGet behavior so it uses normalized versions everywhere? (pack + install + restore)

How I see this is, that NuGet is kinda forcing people to use normalized versions even if they don't want it, right :) ?

[emgarten]

Normalizing solves many issues within NuGet. I think your scenario is valid, but less common than the problems normalizing fixes. You can work around this problem by finding the normalized version from the package after packing and using that for your mappings instead of the -Version input. You could also modify the nuspec in the package after pack to un-normalize it.

[timotei]

Does this mean that a PR that enables the NuGet Pack to generate the NuGet package "as is" from the NuSpec (without normalizing versions) would not be accepted into the client?

IIRC, there was once a warning when doing a pack for 4 digits (that the version doesn't satisfy SemVer 2?), but that warning does not show up now, and instead it directly normalizes the version (not sure why normalize it only when ending in '0', since SemVer2 is 3 digits only nevertheless the last 4th digit)

[rrelyea]

@timotei - yes, 3.4.* does this behavior during pack. Earlier versions didn't. We'd prefer to not add more command line options to not normalize. Is it possible for you to adapt your mapping file to understand ".0" as the 4th version number will be dropped? We'd prefer to keep everybody normalizing.

[timotei]

Well, in the end we managed to get a post-NuGet pack step that denormalizes the version, to keep the previous behavior. So far it seems to work.

But this is not a very satisfactory solution :( I'd rather have NuGet not change things I specifically set in the nuspec.

[slaneyrw]

This is now causing problems downstream as tools now start upgrading their nuget versions. Octopack now uses 3.4.x which is dropping the 4th part of the version number if it's zero, and breaking templated builds in our TeamCity. Please consider command line options

[emgarten]

@slaneyrw can you explain how TeamCity uses this? Which part of the build breaks due to this change?

[slaneyrw]

Octopack generates an nupkg file which we are expecting to be in a specific format {package name}.{version number}.nupkg that we are publishing to our internal ProGet feed.

The version number is TC's system build number which is a 4 part number. The file name we expect is not present when it has been normalised. The TC configuration template is shared by many individual build projects.

All build projects run off a dedicated nuspec file in each project, this file is patched with TC's system build number before an Octopack build.

[mistachkin]

Today, I received a report from a user about NuGet mangling the version within the nuspec file that gets embedded into the nupkg file. Apparently, this "normalization" completely broke the package restore feature, with an error complaining about the package "not being found" until he manually re-added the final trailing zero.

Linked to ticket [https://system.data.sqlite.org/index.html/info/e9573e2d12387877].

[emgarten]

@mistachkin restore could only break if the user is changing the package and re-packing it using an existing id/version. That scenario is not supported since packages are cached based on the id/version, having different copies of the same id/version will cause other problems.

[mistachkin]

Sorry, but no. The user stated the package would not restore until he manually "un-normalized" the version to add the trailing zero. To be perfectly honest, this is going to be a huge headache. The packages I manage have a total of around 3.3M downloads at this point. If even a small fraction of the users end up with issues like this, that will be a major problem. Looks like the solution is to stick with NuGet version 2.x until this is fixed. Meanwhile, I'll have to look into manually fixing the current production packages. Of course, that won't work either because I cannot ever delete anything to re-upload them with the same version. I'm very frustrated now.

[emgarten]

@mistachkin would you open a new issue for what you are seeing with restore along with some repro steps?

This issue is around package creation, it sounds like you are seeing issues on the consuming side.

[mistachkin]

Respectfully, I disagree. The real issue is that what ended up in the embedded nuspec file was not what I originally authored. Subsequently, this "normalization" ended up breaking package restore (and other features?).

[emgarten]

@mistachkin NuGet supports all both normalized and non-normalized versions during restore. The question is what part of the process is treating the nuspec version as a string instead of a version.

[mistachkin]

That part of the process, whatever it is, by definition, does not "support" the version normalization. Meanwhile, some of the most popular non-Microsoft packages on NuGet.org are effectively broken.

[emgarten]

That part of the process, whatever it is, by definition, does not "support" the version normalization. Meanwhile, some of the most popular non-Microsoft packages on NuGet.org are effectively broken.

@mistachkin please open a new issue for the restore issues you are seeing. Any extra details on how this affects existing packages would be extremely helpful and I would like to understand it better to help you out.

Normalizing the version at pack time only, as this issue covers, only affects new packages from what I can see. Users installing those new packages will have the version form used in the nuspec file from the start, so there will be no changes on the consuming side.

[mistachkin]

I did not discover the issue myself and I do not know the reproduction steps. I will contact the original user and ask them to file a new issue. Done, email sent to original user.

[slaneyrw]

Do we have any direction on what NuGet will be doing in regards to normalization. Are we doing to be able to turn OFF this behaviour during pack ?

[mistachkin]

I agree with @slaneyrw, there needs to be a -verbatim option (or similar) to the NuGet pack command that retains the originally authored nuspec file exactly as authored.

[dougbw]

We are having issues as a result of this too - We use a build counter version number throughout our build, deploy, and test pipelines and having a mismatch causes failures when using the package version in a file system path.

Example scenario:

• TeamCity build is version 1.0.100.0
• Nuget pack creates version 1.0.100
• Deploy application
• Deploy the post-deploy tests to a test server, the package version is part of the target directory path Eg "c:\tests\post-deploy\1.0.100"
• The post-deploy tests fail to run as they use the build version number in the path Eg "c:\tests\post-deploy\1.0.100.0"

If a version is specified in the nuspec file then the output package should contain that exact version. We have had to roll back to a specific older version of NuGet because of this.

The behaviour of using nuget pack should remain the same as previous versions, and a -Normalize switch should be present if you want Nuget to automatically normalize the version number.

[emgarten]

I like the idea of -verbatim or possibly -verbatimversion for pack to keep nuget from trying to help out here.

Making non-normalized opt-in instead of by default makes sense for most users I think. It helps avoid duplicates having in the same folder on disk and the confusing builds that can result from that.

[chris1248]

Nuget version 4.0.0.2283 is also removing leading zeros: Converting: 27.0001.02220-alpha to this: 27.1.2220-alpha

Do I have to fork my own repo to fix this? I hope not. [Edit] This is happening on versions 3.5, 4.0 and 4.1 Version 3.4 doesn't do the name mangling, so that's what I'm going back to... Sorry Microsoft, looks like you dropped the ball on this: Forcing normalization when no one wanted it.

[emgarten]

@chris1248 I would expect the verbatim option to leave leading zeros in also.

In general I strongly suggest avoiding leading zeros, older clients such as 2.x and even 3.4.4 have difficulty finding these packages from local folder sources.

[chris1248]

After talking with some colleagues here at work we decided against against using a version with leading zeros. [Edit] This is only an issue in the first place because of version normalization. If it had been left alone in the first place... ??

Sent from my iPhone

On Apr 12, 2017, at 12:03 PM, Justin Emgarten notifications@github.com<mailto:notifications@github.com> wrote:

@chris1248https://github.com/chris1248 I would expect the verbatim option to leave leading zeros in also.

In general I strongly suggest avoiding leading zeros, older clients such as 2.x and even 3.4.4 have difficulty finding these packages from local folder sources.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/NuGet/Home/issues/3050#issuecomment-293660003, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACyJCouHwCKvdT2ik4zK6pxLHaIRdJDVks5rvRHQgaJpZM4I-_5n.

[ghost]

Is there a workaround or fix to this issue? Or the workaround is to remove the leading zero from the version number? Like instead of starting the version number at 2.0.0.0 i need to start at 2.0.0.1?

[StingyJack]

Why not allow leading zeros? Version numbers have started with 0.0.1 before, and its useful in some cases when building development packages to use low numbers like that for those of us who are small shops with one nuget server that gets dev stuff on it from time to time.

I'm using trailing zero's and I need the 4 part version number to be left alone. 6.3.2.0 should be the package version.

If you want to add another cryptic error if a chosen number is going to be a problem, that's understandable. Other than that, there is really no reason for nuget to touch the version number the author associated with a package, ever.

[MikeMugu]

This problem has also broken our builds and our version number is something that goes through much of the app. The testing effort to change the format is not terribly small and the build scripts also have to be updated. I guess there is not much choice at this point but I wish backwards compatibility was considered here.

[ghost]

While reading this discussion I got the impression that most build chains were broken while copying/deploying the generated package as they expected another filename (as was our build chain). What if the build script could “ask” nuget.exe to get the expected filename of a package? e.g.

nuget.exe resolvefilename -Version 1.0.0.0 foo.nuspec
foo-1.0.0.nupkg

or

nuget.exe resolvefilename -Version 1.0.0.0 foo
foo-1.0.0.nupkg

or

nuget.exe normalizeversion 1.0.0.0
1.0.0

This would allow build scripts to always get the right filename.

[StingyJack]

@JeanMarcHengen - Its not just build scripts. Some of us produce software that is used in regulated (pharmaceutical) environments, where changes to the software (or the means with which is produced) can require "re-validation".

Validation is tedious and costly - every action is pre-scripted, then executed, with evidence captured and signature or initials are needed for each action's result, for just about every point in the process. Most software activity isnt needing to adhere to that rigid of a process, but version numbers are visible things to be collected as evidence. Changing the number of digits can be an expensive update for that documentation and testing nightmare.

Also, its not up to a single tool to determine what is OK or not in my version numbering.

[sverrehundeide]

This is what the NuGet documentation says about version normalization (https://docs.microsoft.com/en-us/nuget/reference/package-versioning):

This normalization does not affect the version numbers in the packages themselves; it affects only how NuGet matches versions when resolving dependencies.

I think the version normalization should not be forced upon all existing users who rely on the version to be formatted in a certain way. At least there should be an option/parameter to skip version normalization when packing, as previously suggested. Else, many users will be stuck with using pre 3.4 version of NuGet.

[StingyJack]

Also, if someone normalizes a database and there is data that was present before normalization, but afterwards is not available anymore in any way, I'm betting probably everyone on this thread would agree that would constitute a problem.

[emgarten]

Just a reminder that this issue is up for grabs, I'm willing to helping someone with a PR for adding the -VerbatimVersion flag to pack.

I believe normalization happens here: https://github.com/NuGet/NuGet.Client/blob/a5abb1bdc1aecdbc1d46f36c3eced4062843c249/src/NuGet.Clients/NuGet.CommandLine/Commands/PackCommand.cs#L158

Using NuGetVersion.ToString() instead of NuGetVersion.ToFullString() would give the non-normalized form such as 1.0.

Other places where the version flows through:

https://github.com/NuGet/NuGet.Client/blob/a5abb1bdc1aecdbc1d46f36c3eced4062843c249/src/NuGet.Core/NuGet.Commands/CommandArgs/PackArgs.cs#L38

https://github.com/NuGet/NuGet.Client/blob/a5abb1bdc1aecdbc1d46f36c3eced4062843c249/src/NuGet.Core/NuGet.Packaging/PackageCreation/Authoring/ManifestMetadata.cs#L82

Pack command options: https://github.com/NuGet/NuGet.Client/blob/a5abb1bdc1aecdbc1d46f36c3eced4062843c249/src/NuGet.Clients/NuGet.CommandLine/Commands/PackCommand.cs#L32

Scenarios to check:

• nuget.exe pack -Version 1.0 -VerbatimVersion
• nuget.exe pack project.nuspec -VerbatimVersion to verify that the nuspec version is left as-is.
• All other ways that versions are populated in the nuspec.

I think it would be useful to have the -VerbatimVersion for legacy scenarios. From a correctness standpoint normalizing versions 1.0, 1.0.0, 1.0.0.0, 1.00, 1.0000000.00000 all to 1.0.0 which is the SemVer standard notation is correct, and it helps users avoid creating multiple packages that are actually an equivalent version which makes finding the latest version from a folder of nupkgs ambiguous.

[menaheme]

I think that truncating a trailing zero (revision number) from a package`s version (i.e. 1.7.1.0 -> 1.7.1) is a mistake that causes harm. The last digit is actual information, and code reading the version number must now assume the last zero, where it should not know anything about it. we downgraded the client version to avoid this.

[KevinMarquette]

This version normalization is also causing issues with PowerShellGet Modules. Modules are internally versioned. So not only are the published versions getting changed when republishing, the installs are also not valid. The package version is getting used at install time and it does not match the module version at load time. This prevents it from working.

[StingyJack]

Just a reminder that this issue is up for grabs, I'm willing to helping someone with a PR for adding the -VerbatimVersion flag to pack.

@emgarten - This reads as "We broke it, but aren't willing to fix it. One of you can." Is that really what is being said?

Whomever had the burning desire to coerce a 3 part version number - when the target is microsoft assemblies, that have been 4 part for 30+ years and still are - should just put this back the way it was ASAP without imposing upon everyone to do something extra. I say everyone because we all are actually using 4 part versions whether we prefer to or not. Not reverting the original mistake is just making a second mistake.

[KevinMarquette]

I think I found the bug with 4 part version numbers that have a trailing 0, but I would like someone that knows the project take a look and run the tests.

https://github.com/NuGet/NuGet.Client/blob/6e6f469828435d0a3c88aa153808ba380c6d0d71/src/NuGet.Core/NuGet.Versioning/NuGetVersion.cs#L180

    public virtual bool IsLegacyVersion
    {
        get { return Version.Revision > 0; }
    }

In that sample, Version is a [Version] type. A revision value of 0 is a valid value. If the revision is not in use on a version, it has a value of -1.

[version]'1.2.3'
Major  Minor  Build  Revision
-----  -----  -----  --------
1      2      3      -1

I think we can make this fix to restore the trailing 0, but it does not address the other issues in this thread.

        get { return Version.Revision >= 0; }

[InteXX]

Vote: leading zeros are needed here.

[nicswan]

+1 for at least removing "normalization" on package restore. If I specify 0.1.1.0 in my packages.config, use that - it shouldn't go looking for package 0.1.1. Then the next time I build these packages, I can enforce semantic versioning, but as it is everything breaks after Nuget 3.3.

[nschwerzler]

We are hitting the .0 version for the forth part causing an error at least once a month across our business. Request that this issues priority be increased.

[InteXX]

@emgarten

Would you comment on @StingyJack's assessment of your position:

"This reads as 'We broke it, but aren't willing to fix it. One of you can.' Is that really what is being said?"

[Peter-B-]

In out team, we use VS2015 and VS2017 in parallel for various reasons. All of them are updated to the latest version of nuget available using the visual studio installer (Nuget 3.4 on 2015 and 4.6 on 2017).

We run into issues with broken builds because of x.y.z.0 nuget packages that get lost due to different paths in the packages folder. We didn't find a schema for predicting or fixing the issue.

For example I saw today, that VS 2015 drops the .0 for a package with version 9.5.429.0, but keeps it for 9.4.326.0 of the same package. The only thing that changed in the .nuspec file is the version number.

Do you have any idea or help here?

[StingyJack]

@peter-B- one of the suggestions in this thread was to have a script or utility correct the version number on the built package, before publishing it.

This is probably going to be the only way as the nuget team is clearly not interested in fixing the defect that was introduced.

[InteXX]

@emgarten

Would you comment on @StingyJack's assessment of your position:

"This reads as 'We broke it, but aren't willing to fix it. One of you can.' Is that really what is being said?"

[emgarten]

Normalization is by design. Versions must be treated as Semantic Versions, not as strings.

External tools that read nuspec versions as strings should be fixed, or scripts should be added to work around this.

Most users in this thread are having problems due to powershell get unnecessarily extracting and re-packing packages just to copy them from one feed to another. This is an issue with powershell get. Re-packing a package will always lead to some changes.

[KevinMarquette]

@emgarten I guess the problem that I am having a hard time understanding is that we have legacy support for a string like 1.2.3.4 but not for 1.2.3.0. Why is it decided that one is to be changed, but we will preserve the other?

If the version contains a 4th non-zero digit, then it is also added since NuGet allowed 4 part versions at the start.

Why the 4th non-zero requirement? NuGet also allowed that 4th digit to be zero at the start. There is a strong request here to restore support for all 4 part version numbers. Is it worth breaking this off into its own issue to separate it from the rest of the normalization discussion?

What you could add to this conversation (that I am not seeing) is the rationale behind why the zero needs to be dropped for normalization. Someone made a conscious decision to normalize that value away and it would be healthy for this thread to know why that is. It would also be good to know what work effort would be required make that adjustment (outside of the -VerbatimVersion request)?

PowerShellGet does need to handle the republish correctly. It's silly that we solve the PowerShellGet issue by installing some 2.x version of nuget. Because as a module author, I specify my own version in my module and I expect to see that version when published to the gallery. Now that I am aware of this issue, I have appropriate workarounds but I had to dig through the source of PowerShellGet to track the issue back here.