dotnet / Nerdbank.GitVersioning

Stamp your assemblies, packages and more with a unique version generated from a single, simple version.json file and include git commit IDs for non-official builds.
https://www.nuget.org/packages/Nerdbank.GitVersioning
MIT License
1.32k stars 165 forks source link

[Feature request] Support partial / blobless clones #1045

Closed MattKotsenas closed 2 months ago

MattKotsenas commented 2 months ago

My team has some large repos that make disabling shallow clones painful. I experimented with blobless / partial clones, however it appears that the managed git implementation doesn't handle this scenario. Here's an exception from a repo cloned via GitHub Actions like this:

steps:
- uses: actions/checkout@v4
  with:
    fetch-depth: 0 # avoid shallow clone so nbgv can do its work
    filter: blob:none # <-- only eagerly download blobs for the current working branch

and here's the resulting exception (I stripped the prefix /home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/Nerdbank.GitVersioning.Inner.targets(17,5): error MSB4018: from each line to make the stacktrace a bit easier)

The "Nerdbank.GitVersioning.Tasks.GetBuildVersion" task failed unexpectedly. [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
System.InvalidOperationException: Unable to get version from commit: 79437f5edda13f9c0669b978dd7a9066dd2059f1 [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
 ---> Nerdbank.GitVersioning.GitException: An blob object with SHA 469f55d9a3003f0c51c3afaa2d221d23fd67e579 could not be found. [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.ManagedGit.GitRepository.GetObjectBySha(GitObjectId sha, String objectType) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.ManagedVersionFile.GetVersion(GitCommit commit, String repoRelativeProjectDirectory, Dictionary`2 blobVersionCache, String& actualDirectory) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.GitWalkTracker.GetVersion(GitCommit commit) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   --- End of inner exception stack trace --- [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.GitWalkTracker.GetVersion(GitCommit commit) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.CommitMatchesVersion(GitCommit commit, SemanticVersion expectedVersion, Position comparisonPrecision, GitWalkTracker tracker) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.<>c__DisplayClass3_0.<GetVersionHeight>b__0(GitCommit c) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.<GetCommitHeight>g__TryCalculateHeight|4_0(GitCommit commit, <>c__DisplayClass4_0&) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.GetCommitHeight(GitRepository repository, GitCommit startingCommit, GitWalkTracker tracker, Func`2 continueStepping) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.GetHeight(ManagedGitContext context, Func`2 continueStepping) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.GitExtensions.GetVersionHeight(ManagedGitContext context, Version baseVersion) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Managed.ManagedGitContext.CalculateVersionHeight(VersionOptions committedVersion, VersionOptions workingVersion) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.VersionOracle..ctor(GitContext context, ICloudBuild cloudBuild, Nullable`1 overrideVersionHeightOffset) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at Nerdbank.GitVersioning.Tasks.GetBuildVersion.ExecuteInner() in D:\a\1\s\src\Nerdbank.GitVersioning.Tasks\GetBuildVersion.cs:line 240 [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) [/home/runner/.nuget/packages/nerdbank.gitversioning/3.6.133/build/PrivateP2PCaching.proj]

I tried NBGV_GitEngine='LibGit2', but it appears that libgit2 doesn't support partial clones either https://www.github.com/libgit2/libgit2/issues/5564

A workaround I can use in the short term is to add the following script step to the pipeline before build to force git to hydrate just the blobs for version.json.

- name: Fetch blobs for nbgv
  run: git --no-pager blame version.json

I'm happy to do the work here / help out however I can, but I'm not sure how best to get started, nor what the design should be. If you have any questions, please let me know. Thanks!

AArnott commented 2 months ago

This is a well understood and documented limitation of git-based versioning systems. There simply is no way to make this work without history because we compute the version based on the number of commits in history for which a given version specified in version.json is set. I've never seen git --no-pager blame version.json used to get a subset of the blobs, and I guess it must get more than just the blobs, since we need trees and commits too. It's a clever workaround. But we're never going to fetch as part of computing versions so I think you'll have to keep your workaround in place if you intend to continue with your shallow clones.

Strictly speaking, we only need as many commits back as the last bump to the version specified in version.json. We don't walk any further back than that. So a 'shallow' but not too shallow clone would theoretically work, if you wanted to maintain some magic number for the git clone --depth switch that was low enough to speed up cloning but high enough to allow nb.gv to suuceed.