error: Cannot find Git revision '39272e2165ad9c49cfa9b5d749a5bb528dbadd87' in ref 'HEAD' of repository 'https://github.com/emacs-straight/org-mode'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.

[bbigras]

I have this error while trying to update to c5fc5c1b77b1348a1d96f341d160d29962a752bd

https://github.com/bbigras/nix-config/actions/runs/9657908154/job/26638090008

[marienz]

I do not understand what's going on here:

• I am passing allRefs = true
• The requested commit exists in the repository
• I was initially seeing this myself (on my local system) from nixos-rebuild, but it went away after I copied my init.el and packages.el into nix-doom-emacs-unstraightened/doomdir and ran a build from there.

I originally assumed this was some bad state specific to my system (especially since CI managed to update...) but if others are seeing this too there must be more to it.

I just tried moving ~/.cache/nix/gitv3/0wjbyw0d1v77l32ma4181iyp0wbmhlgrqqgcz9yxmv84v22cn6dq* away, and that has made the problem reappear when running nixos-rebuild, and so far I haven't managed to make it disappear again... which may be good as it may allow me to trace through what's happening.

https://github.com/NixOS/nix/issues/10431 is possibly related. But given we're both seeing this on the same commit in the same repo, and it disappeared quickly yesterday and so far isn't disappearing again today) there might be more to it...

(CI also failed but that looks like a different problem...)

[bbigras]

Note that something might be fucky with emacs-org. It might be for my non-doom emacs config. Maybe with the emacs overlay (emacsPackages.org seems fine).

building '/nix/store/njh3ds3ccwckx3kgil2ms75pfh3c5g3j-emacs-org-9.7.5.drv'...
Running phase: unpackPhase
unpacking source archive /nix/store/c4iaxkhisi8zry6iq94pf1lw7fri5h1g-org-9.7.5.tar
source root is org-9.7.5
setting SOURCE_DATE_EPOCH to timestamp 1719090513 of file org-9.7.5/README-elpa
Running phase: patchPhase
applying patch /nix/store/14mqpyqvci1kzzm6rbs25dpx5avb61rg-?id=f4cc61636947b5c2f0afc67174dd369fe3277aa8
patching file ol.el
Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file ol.el.rej
error: builder for '/nix/store/njh3ds3ccwckx3kgil2ms75pfh3c5g3j-emacs-org-9.7.5.drv' failed with exit code 1;
       last 11 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/c4iaxkhisi8zry6iq94pf1lw7fri5h1g-org-9.7.5.tar
       > source root is org-9.7.5
       > setting SOURCE_DATE_EPOCH to timestamp 1719090513 of file org-9.7.5/README-elpa
       > Running phase: patchPhase
       > applying patch /nix/store/14mqpyqvci1kzzm6rbs25dpx5avb61rg-?id=f4cc61636947b5c2f0afc67174dd369fe3277aa8
       > patching file ol.el
       > Reversed (or previously applied) patch detected!  Assume -R? [n]
       > Apply anyway? [n]
       > Skipping patch.
       > 1 out of 1 hunk ignored -- saving rejects to file ol.el.rej
       For full logs, run 'nix log /nix/store/njh3ds3ccwckx3kgil2ms75pfh3c5g3j-emacs-org-9.7.5.drv'.

https://github.com/bbigras/nix-config/actions/runs/9667909678/job/26670818795#step:6:6079

[marienz]

I've pushed a change that may help with "Cannot find Git revision... in ref 'HEAD'", at the cost of making builds very slow if you're on recent Nix (2.22.1 is slow, 2.18.2 is ok), a it removes my hacky workaround for https://github.com/NixOS/nix/issues/10773.

I'm now locally getting

       error: Cannot find Git revision 'b4dca00f89334392d770a7a67fffc935ec7354aa' in ref 'eebb2fb49d3c0a0586d1e4ead9ba618c7d003cae' of repository 'https://gitlab.com/emacs-ansible/emacs-ansible.git'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.

However, that looks like an actual problem: emacs-overlay changed the recipe from https://github.com/k1LoW/emacs-ansible to https://gitlab.com/emacs-ansible/emacs-ansible. I'll push a fix for that.

If you're still seeing some other "Cannot find Git revision..." after updating, please let me know.

That second failure is what CI is failing on here too (https://github.com/marienz/nix-doom-emacs-unstraightened/actions/runs/9676278195/job/26695512478). At first glance, that happens because of a backported security fix (https://github.com/NixOS/nixpkgs/commit/aa45c7e67ddb4fc1a8f033cbfe30bb8f1b5adfee), getting applied to a version of org that already has that fix.

I think that's https://github.com/nix-community/emacs-overlay/issues/411. Unstraightened should not hit that if you have :lang org enabled in init.el, but would if only some other module depends on Org indirectly (which I have a test for, which fails...).

[bbigras]

I seem to still have the problem with 9205f287

error: Cannot find Git revision '39272e2165ad9c49cfa9b5d749a5bb528dbadd87' in ref 'refs/heads/main' of repository 'https://github.com/emacs-straight/org-mode'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit

https://github.com/bbigras/nix-config/actions/runs/9682975326/job/26717363165#step:6:352

If you're still seeing some other "Cannot find Git revision..." after updating, please let me know.

Oh did you mean beside with org?

[bbigras]

I was wrong. If I update nixos-unstable I can update to 9205f2878ff2d173b91cf27f594d4f9a983b639f.

[bbigras]

It seems I can even go back to c5fc5c1b77b1348a1d96f341d160d29962a752bd.

[bbigras]

I still have the same problem https://github.com/bbigras/nix-config/actions/runs/9690868785/job/26741437336#step:6:362

I guess it was cached or something on my desktop. sorry about that.

[marienz]

Darn.

Once you've somehow fetched the problematic commit once, the problem does not seem to reoccur (that is: once you've fetched it once into ~/.cache/nix/gitv3/0wjbyw0d1v77l32ma4181iyp0wbmhlgrqqgcz9yxmv84v22cn6dq-shallow and not thrown that away... won't help you much with a CI build on a fresh cache each time).

I'll try to find time to dig further, but it may take a few days (longer if I'm not able to reproduce the problem locally somewhat consistently). I'll likely need to trace exactly what Nix is actually doing in terms of git invocations: I currently just do not see what's going wrong, or what might be special about the org-mode repository...

[schwanberger]

I ran into the same issue as OP after bumping my flake.lock.

I was using Nix 2.23.x, and after downgrading to 2.18.x (stable) the issue disappeared - still using the newly bumped flake.lock.

I then cleared the cache and reproduced the issue on both 2.22.x and 2.23.x. Downgrading nix once more led to being unable to reproduce the issue.

[marienz]

Thanks, that's useful information:

• It means 9205f2878ff2d173b91cf27f594d4f9a983b639f didn't fix it (and not having that slows down current (not stable) Nix enough I may reinstate the hack it removed)
• Since it only affects users already willing to use current Nix (newer than stable), I've exposed a feature that should work around the problem: set experimentalFetchTree = true to switch GitHub fetches from Git to downloading pre-generated tarballs.

That's not really a fix, but as long as we're only seeing this on one GitHub repo that should at least give you another option (besides downgrading Nix). And if I understand correctly it's "experimental" in Nix more because it's still subject to change than because it doesn't work. Just remember to turn that off again if after a future Nix upgrade you suddenly get entirely different fetch-related errors...

[marienz]

as long as we're only seeing this on one GitHub repo

...welp, apparently I tempted fate there: CI failed with

       error: Cannot find Git revision '2f0320c5f24adfee026e938ebc379ca90e3045d3' in ref 'refs/heads/master' of repository 'https://git.notmuchmail.org/git/notmuch'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.

Since that's not GitHub (and not one of the other forges Nix has a dedicated fetcher for) fetchTree's non-Git fetchers won't save us.

However, this one may be a problem between that Git server and Nix, given:

$ git init shallow
Initialized empty Git repository in /tmp/shallow/.git/
$ git remote add origin https://git.notmuchmail.org/git/notmuch
$ git fetch origin 2f0320c5f24adfee026e938ebc379ca90e3045d3
error: Server does not allow request for unadvertised object 2f0320c5f24adfee026e938ebc379ca90e3045d3

If instead I fetch master first, I can then check out the commit (it does seem to be reachable from master):

$ git fetch origin master
...
From https://git.notmuchmail.org/git/notmuch
 * branch              master     -> FETCH_HEAD
 * [new branch]        master     -> origin/master
$ git checkout 2f0320c5f24adfee026e938ebc379ca90e3045d3
Note: switching to '2f0320c5f24adfee026e938ebc379ca90e3045d3'.

What might be going on here is that Nix 2.23 is attempting to do a shallow clone (new default), but it doesn't actually work when fetching a specific rev from an upstream that does not have uploadpack.allowReachableSHA1InWant enabled (see man git-config, off by default, and I haven't learned how to properly trace Git yet but when glancing at GIT_TRACE_CURL=1 output I don't see allow-reachable-sha1-in-want in what looks like the advertised features).

That's not the full story, because fetching 39272e2165ad9c49cfa9b5d749a5bb528dbadd87 from org the same way seems to work fine. However, it just so happens that Doom sets :depth 1 in its org-mode recipe, which I translate into passing shallow = true to Nix. So "shallow fetches of specific revs don't work in newer Nix" may be part of the problem (that'd explain why several of us were seeing this specifically with org-mode).

...let me re-run CI, and if the problem recurs let me try forcing shallow = false for that repo and see if that fixes things...

[marienz]

(Another datapoint: cachix build is succeeding, but that workflow is using cachix/install-nix-action to install Nix, which is installing 2.22.1. The "check" workflow uses DeterminateSystems/nix-installer-action, which (given https://github.com/DeterminateSystems/nix-installer/releases/tag/v0.20.0) should be installing 2.23.1. Which does seem to point towards those shallow fetches...)

[marienz]

Ok, that fixed CI.

I've reported the shallow-fetches-of-specific-revs thing as https://github.com/NixOS/nix/issues/11012 (note I currently do not expect to fix the originally-reported case of org-mode from github, since github does seem to support these shallow clones).

[bbigras]

Note that I have been using lix all this time:

❯ nix --version
nix (Lix, like Nix) 2.90.0-lixpre20240527-0b91a4b