`opam clean -c` seems incompatible with `opam pin add -k git (...)`

[erikmd]

Steps to reproduce the issue

# This is not specific to dune, it is just an example package with a committed dune.opam
$ opam pin add -n -y -k git dune "https://github.com/ocaml/dune#a84b95949285966952ea6e89568362c6a5afd5e9"
$ opam install dune
$ opam clean -c
$ opam update

<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>
[coq-released] no changes from https://coq.inria.fr/opam/released
[default] synchronised from https://opam.ocaml.org

<><> Synchronising development packages <><><><><><><><><><><><><><><><><><><><>
[ERROR] Could not synchronize /home/coq/.opam/4.05.0/.opam-switch/sources/dune from
        "git+https://github.com/ocaml/dune#a84b95949285966952ea6e89568362c6a5afd5e9":
        "/usr/bin/git fetch -q" exited with code 1 "error: https://github.com/ocaml/dune did not send
        all necessary objects"
Now run 'opam upgrade' to apply any package updates.

System info

# opam config report
# opam-version      2.0.4 
# self-upgrade      no
# system            arch=x86_64 os=linux os-distribution=debian os-version=9
# solver            builtin-mccs+glpk
# install-criteria  -removed,-count[version-lag,request],-count[version-lag,changed],-changed
# upgrade-criteria  -removed,-count[version-lag,solution],-new
# jobs              1
# repositories      2 (http) (default repo at db98cef2)
# pinned            1 (git)
# current-switch    4.05.0

Additional info

This issue showed up in this coq-community CI job: https://travis-ci.com/coq-community/math-classes/jobs/192817437, that motivated that workaround patch: https://github.com/coq-community/docker-coq/commit/332ff995f9d18e276fa03c48d98e3f3fbc176a93

The issue seems related to #3485, albeit that latter issue didn't mention opam clean -c.

Being able to clean the opam download cache in this context would be useful to earn ~110 MB when building Docker images of Coq:

$ docker run --rm -it coqorg/coq:dev
  $ du -s .opam
  2670684 .opam
  $ opam clean -c
  Clearing cache of downloaded files
  $ du -s .opam
  2557616 .opam

[erikmd]

FYI here is another kind of error message that shows up in practice (excerpt from a Docker Hub build log of mathcomp/mathcomp:1.8.0-coq-dev before I removed the opam clean -c step):

+ opam update -y
[ERROR] Command "/usr/bin/git diff --no-ext-diff --quiet" failed:
"/usr/bin/git diff --no-ext-diff --quiet" exited with code 128 "error: object directory /home/coq/.opam/download-cache/git/objects does not exist; check .git/objects/info/alternates."

[AltGr]

Thanks for reporting! This seems related to the way we cache git objects (using .git/info/objects/alternates)

Interestingly enough, it doesn't reproduce if dune isn't installed:

# pin
+ /usr/bin/git "init" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
- Dépôt Git vide initialisé dans /tmp/opam/default/.opam-switch/sources/dune/.git/
+ /usr/bin/git "config" "--local" "fetch.prune" "false" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "config" "--local" "diff.noprefix" "false" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "remote" "add" "origin" "https://github.com/ocaml/dune" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "init" "--bare" (CWD=/tmp/opam/download-cache/git)
- Dépôt Git vide initialisé dans /tmp/opam/download-cache/git/
+ /usr/bin/git "remote" "set-url" "origin" "https://github.com/ocaml/dune" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "fetch" "-q" "https://github.com/ocaml/dune" "--update-shallow" "+a84b95949285966952ea6e89568362c6a5afd5e9:refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
- error: no such remote ref a84b95949285966952ea6e89568362c6a5afd5e9
+ /usr/bin/git "fetch" "-q" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "fetch" "-q" "https://github.com/ocaml/dune" "+a84b95949285966952ea6e89568362c6a5afd5e9:refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "reset" "--hard" "refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" "--" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
- HEAD est maintenant à a84b9594 Merge pull request #2057 from rgrinberg/coq-no-boot
[dune.1.9.1] synchronised (git+https://github.com/ocaml/dune#a84b95949285966952ea6e89568362c6a5afd5e9)

# update
+ /usr/bin/git "remote" "set-url" "origin" "https://github.com/ocaml/dune" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "fetch" "-q" "https://github.com/ocaml/dune" "--update-shallow" "+a84b95949285966952ea6e89568362c6a5afd5e9:refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "push" "/tmp/opam/download-cache/git" "+refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9:refs/remotes/3c4c7a7b884b7fbe8409bb93254b2392" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
- Everything up-to-date
+ /usr/bin/git "diff" "--no-ext-diff" "--quiet" "refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" "--" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
[dune.1.9.1] synchronised (no changes)

# update after clean
+ /usr/bin/git "init" "--bare" (CWD=/tmp/opam/download-cache/git)
- Dépôt Git vide initialisé dans /tmp/opam/download-cache/git/
+ /usr/bin/git "remote" "set-url" "origin" "https://github.com/ocaml/dune" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "fetch" "-q" "https://github.com/ocaml/dune" "--update-shallow" "+a84b95949285966952ea6e89568362c6a5afd5e9:refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "push" "/tmp/opam/download-cache/git" "+refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9:refs/remotes/3c4c7a7b884b7fbe8409bb93254b2392" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
- To /tmp/opam/download-cache/git
-  * [new branch]        opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9 -> 3c4c7a7b884b7fbe8409bb93254b2392
+ /usr/bin/git "diff" "--no-ext-diff" "--quiet" "refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" "--" (CWD=/tmp/opam/default/.opam-switch/sources/dune)

# update after install and clean
+ /usr/bin/git "init" "--bare" (CWD=/tmp/opam/download-cache/git)
- Dépôt Git vide initialisé dans /tmp/opam/download-cache/git/
+ /usr/bin/git "remote" "set-url" "origin" "https://github.com/ocaml/dune" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
+ /usr/bin/git "fetch" "-q" "https://github.com/ocaml/dune" "--update-shallow" "+a84b95949285966952ea6e89568362c6a5afd5e9:refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
- error: refs/heads/master does not point to a valid object!
- error: refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9 does not point to a valid object!
[...]
- error: no such remote ref a84b95949285966952ea6e89568362c6a5afd5e9
+ /usr/bin/git "fetch" "-q" (CWD=/tmp/opam/default/.opam-switch/sources/dune)
- error: refs/heads/master does not point to a valid object!
- error: refs/remotes/opam-ref-a84b95949285966952ea6e89568362c6a5afd5e9 does not point to a valid object!
[...]
- fatal: bad object 099a82d8b7766f9af21b19f068f1de8f82ce8c1a
- error: https://github.com/ocaml/dune n'a pas envoyé tous les objets nécessaires
- 
[ERROR] Could not synchronize /tmp/opam/default/.opam-switch/sources/dune from
        "git+https://github.com/ocaml/dune#a84b95949285966952ea6e89568362c6a5afd5e9":
        "/usr/bin/git fetch -q" exited with code 1 "error: https://github.com/ocaml/dune n'a pas envoyé tous les
        objets nécessaires"

Note that we are here in a case where you are pinning to an unreferred object, which is already tricky with git — you can see in the logs that the first fetch fails, and we fallback to a full fetch before recovering the commit. I am not sure the issue is related though.

[AltGr]

So indeed, removing the cache results in lost objects in the git clone, breaking it in a way that git apparently cannot recover even from the upstream on fetch (it seems it doesn't detect the missing objects before querying the server).

Here are a few options:

• disable cleanup of the git cache on opam clean -c — or maybe replace it with a git gc ?
• another way would be to cleanup the git clones of individual packages too, but that would trigger issues on updates (we'd no longer be able to detect if there were changes or not)
• find a more reliable way to use the cache, e.g. hardlinks instead of "alternates"
• disable the git cache :disappointed:

I implemented the first solution (with git gc) as a quick fix ; I don't think the git cache can easily reach a size which would make it a problem (it's 3.2MB here !) so that might be acceptable for now.