ocaml / opam

opam is a source-based package manager. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.
https://opam.ocaml.org
Other
1.23k stars 351 forks source link

"--skip-update" has no effect #4363

Open RalfJung opened 4 years ago

RalfJung commented 4 years ago

If your issue concerns a package not building, please report to https://github.com/ocaml/opam-repository/issues or to the package maintainer unless you are confident it is an issue in the opam tool itself.

I have an opam package installed whose git URL is now dead. opam uninstall coq-tlc hangs forever, trying to download the latest sources:

+ /usr/bin/git "fetch" "-q" "https://gitlab.inria.fr/charguer/tlc.git" "--update-shallow" "+9205154a:refs/remotes/opam-ref-9205154a" (CWD=/home/r/.opam/coq-8.10/.opam-switch/sources/coq-tlc)
^C[ERROR] User interruption
'opam uninstall -yvv coq-stdpp coq-tlc coq-ltac2' failed.

So I tried to add --skip-updates, but that does not help either, it still updates the package! Looks like it is impossible to uninstall or update this package, so I will have to delete the switch and start over again?

$ opam uninstall --skip-updates coq-stdpp coq-tlc coq-ltac2 -yvvv
+ /usr/bin/lsb_release "-s" "-r"
- testing
+ /usr/bin/ocamlc "-vnum"
- 4.08.1
The following actions will be performed:
  ⊘ remove    coq-spygame-builddep ~dev*                     [uses coq-tlc]
  ∗ install   ocaml-system         4.08.1                    [required by ocaml]
  ⊘ remove    coq-ltac2            0.3
  ⊘ remove    coq-tlc              dev*
  ⊘ remove    coq-iris             dev.2020-07-15.0.6b0e19f5 [uses coq-stdpp]
  ↻ recompile ocaml-config         1                         [uses ocaml-system]
  ⊘ remove    coq-stdpp            dev.2020-07-02.1.c8129a37
  ↻ recompile ocaml                4.08.1                    [uses coq-ltac2, coq-tlc]
  ↻ recompile ocamlfind            1.8.1                     [uses ocaml]
  ↻ recompile num                  1.3                       [uses ocaml]
  ↻ recompile coq                  8.10.2*                   [uses coq-stdpp, coq-ltac2, coq-tlc]
===== ∗ 1   ↻ 5   ⊘ 5 =====

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[coq.8.10.2] found in cache
[num.1.3] found in cache
+ /usr/bin/git "remote" "set-url" "origin" "https://gitlab.inria.fr/charguer/tlc.git" (CWD=/home/r/.opam/coq-8.10/.opam-switch/sources/coq-tlc)
+ /bin/tar "xfz" "/home/r/.opam/download-cache/md5/f0/f074e12325e84ebc883b37e5db10403d" "-C" "/tmp/opam-351618-1872ec"
+ /bin/cp "-PRp" "/tmp/opam-351618-1872ec/num-1.3/.gitignore" "/tmp/opam-351618-1872ec/num-1.3/Changelog" "/tmp/opam-351618-1872ec/num-1.3/LICENSE" "/tmp/opam-351618-1872ec/num-1.3/Makefile" "/tmp/opam-351618-1872ec/num-1.3/README.md" "/tmp/opam-351618-1872ec/num-1.3/dune-project" "/tmp/opam-351618-1872ec/num-1.3/num.opam" "/tmp/opam-351618-1872ec/num-1.3/src" "/tmp/opam-351618-1872ec/num-1.3/test" "/tmp/opam-351618-1872ec/num-1.3/toplevel" "/home/r/.opam/coq-8.10/.opam-switch/sources/num.1.3"
[ocamlfind.1.8.1] found in cache
+ /bin/tar "xfz" "/home/r/.opam/download-cache/md5/18/18ca650982c15536616dea0e422cbd8c" "-C" "/tmp/opam-351618-d8a4ae"
+ /bin/cp "-PRp" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/INSTALL" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/LICENSE" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/Makefile" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/Makefile.config.pattern" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/configure" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/doc" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/findlib.conf.in" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/itest" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/itest-aux" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/mini" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/site-lib-src" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/src" "/tmp/opam-351618-d8a4ae/findlib-1.8.1/tools" "/home/r/.opam/coq-8.10/.opam-switch/sources/ocamlfind.1.8.1"
+ /bin/tar "xfz" "/home/r/.opam/download-cache/sha512/80/80df91b64efc9907480388ec479362ee21067c64436da720989d6d1645ffc2f2230ae5c13069c55842da3baa7facbd143c2190d1d64d8c87935802000a02156f" "-C" "/tmp/opam-351618-9b8556"
+ /bin/cp "-PRp" "/tmp/opam-351618-9b8556/coq-8.10.2/.github" "/tmp/opam-351618-9b8556/coq-8.10.2/.gitlab-ci.yml" "/tmp/opam-351618-9b8556/coq-8.10.2/.merlin.in" "/tmp/opam-351618-9b8556/coq-8.10.2/.ocamlinit" "/tmp/opam-351618-9b8556/coq-8.10.2/CODE_OF_CONDUCT.md" "/tmp/opam-351618-9b8556/coq-8.10.2/CONTRIBUTING.md" "/tmp/opam-351618-9b8556/coq-8.10.2/CREDITS" "/tmp/opam-351618-9b8556/coq-8.10.2/INSTALL" "/tmp/opam-351618-9b8556/coq-8.10.2/LICENSE" "/tmp/opam-351618-9b8556/coq-8.10.2/META.coq.in" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.build" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.checker" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.ci" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.common" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.dev" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.doc" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.dune" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.ide" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.install" "/tmp/opam-351618-9b8556/coq-8.10.2/Makefile.vofiles" "/tmp/opam-351618-9b8556/coq-8.10.2/README.md" "/tmp/opam-351618-9b8556/coq-8.10.2/azure-pipelines.yml" "/tmp/opam-351618-9b8556/coq-8.10.2/checker" "/tmp/opam-351618-9b8556/coq-8.10.2/clib" "/tmp/opam-351618-9b8556/coq-8.10.2/config" "/tmp/opam-351618-9b8556/coq-8.10.2/configure" "/tmp/opam-351618-9b8556/coq-8.10.2/configure.ml" "/tmp/opam-351618-9b8556/coq-8.10.2/coq-refman.opam" "/tmp/opam-351618-9b8556/coq-8.10.2/coq.opam" "/tmp/opam-351618-9b8556/coq-8.10.2/coqide-server.opam" "/tmp/opam-351618-9b8556/coq-8.10.2/coqide.opam" "/tmp/opam-351618-9b8556/coq-8.10.2/coqpp" "/tmp/opam-351618-9b8556/coq-8.10.2/default.nix" "/tmp/opam-351618-9b8556/coq-8.10.2/dev" "/tmp/opam-351618-9b8556/coq-8.10.2/doc" "/tmp/opam-351618-9b8556/coq-8.10.2/dune" "/tmp/opam-351618-9b8556/coq-8.10.2/dune-project" "/tmp/opam-351618-9b8556/coq-8.10.2/engine" "/tmp/opam-351618-9b8556/coq-8.10.2/gramlib" "/tmp/opam-351618-9b8556/coq-8.10.2/ide" "/tmp/opam-351618-9b8556/coq-8.10.2/install.sh" "/tmp/opam-351618-9b8556/coq-8.10.2/interp" "/tmp/opam-351618-9b8556/coq-8.10.2/kernel" "/tmp/opam-351618-9b8556/coq-8.10.2/lib" "/tmp/opam-351618-9b8556/coq-8.10.2/library" "/tmp/opam-351618-9b8556/coq-8.10.2/man" "/tmp/opam-351618-9b8556/coq-8.10.2/parsing" "/tmp/opam-351618-9b8556/coq-8.10.2/plugins" "/tmp/opam-351618-9b8556/coq-8.10.2/pretyping" "/tmp/opam-351618-9b8556/coq-8.10.2/printing" "/tmp/opam-351618-9b8556/coq-8.10.2/proofs" "/tmp/opam-351618-9b8556/coq-8.10.2/shell.nix" "/tmp/opam-351618-9b8556/coq-8.10.2/stm" "/tmp/opam-351618-9b8556/coq-8.10.2/tactics" "/tmp/opam-351618-9b8556/coq-8.10.2/test-suite" "/tmp/opam-351618-9b8556/coq-8.10.2/theories" "/tmp/opam-351618-9b8556/coq-8.10.2/tools" "/tmp/opam-351618-9b8556/coq-8.10.2/topbin" "/tmp/opam-351618-9b8556/coq-8.10.2/toplevel" "/tmp/opam-351618-9b8556/coq-8.10.2/vernac" "/home/r/.opam/coq-8.10/.opam-switch/sources/coq"
+ /usr/bin/git "fetch" "-q" "https://gitlab.inria.fr/charguer/tlc.git" "--update-shallow" "+9205154a:refs/remotes/opam-ref-9205154a" (CWD=/home/r/.opam/coq-8.10/.opam-switch/sources/coq-tlc)
# opam config report
# opam-version      2.0.7 
# self-upgrade      no
# system            arch=x86_64 os=linux os-distribution=debian os-version=testing
# solver            builtin-mccs+glpk
# install-criteria  -removed,-count[version-lag,request],-count[version-lag,changed],-changed
# upgrade-criteria  -removed,-count[version-lag,solution],-new
# jobs              8
# repositories      4 (http), 3 (local), 2 (version-controlled) (default repo at 4fb8d40b)
# pinned            1 (git), 3 (rsync), 1 (version)
# current-switch    coq-8.10
AltGr commented 4 years ago

Hm, it's weird, for any pinned package opam is supposed to have a cache of its latest sources at $OPAM_SWITCH_PREFIX/.opam-switch/sources/$PKGNAME ; these sources are indeed updated from upstream before install/reinstall/upgrade actions when the package is listed on the command-line, unless you specified --skip-update.

For removals, however, these updates are never necessary, so indeed the option shouldn't do anything.

I only tested on 2.1.0~beta2, but here is what I get after removing a pinned package's upstream:

$ opam upgrade ocp-index

<><> Synchronising pinned packages ><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] Could not synchronize $OPAM_SWITCH_PREFIX/.opam-switch/sources/ocp-index from "git+file:///tmp/ocp-index.x#master":
        "/usr/bin/git fetch -q" exited with code 128
[ocp-index.1.2.1] fetching sources failed: git+file:///tmp/ocp-index.x#master

Already up-to-date.
Nothing to do.
$ opam upgrade ocp-index --skip
Already up-to-date.
Nothing to do.
$ opam remove ocp-index        
The following actions will be performed:
  ⊘ remove ocp-browser 1.2.1* [uses ocp-index]
  ⊘ remove ocp-index   1.2.1*
===== ⊘ 2 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⊘ removed   ocp-browser.1.2.1
⊘ removed   ocp-index.1.2.1
Done.

So there is no issue with either upgrade --skip-updates or remove ; with just upgrade, the download fails, but the command can still continue (useful for reinstall, for example). Of course, in your case it hangs rather than failing, which is more annoying...

NB: have you tried <enter> or ctrl-d when the downloading hangs ? We've had cases of downloads hanging that were actually silently waiting for input.

To conclude, at this point I don't know yet if this is fixed in 2.1 or if I just couldn't reproduce.

RalfJung commented 4 years ago

I have since worked around it by killing the hanging git-fetch-pack (or so) process. Then opam just continued, warning me that deinstallation might use outdated information because the update failed.

Note that the deinstallation did update many things because this is a system switch and the system compiler changed, so any command will rebuild all the things. (I always struggle hard to make it only rebuild the parts I intend to keep, but found it very hard to combine "rebuild" and "upgrade" and "remove stuff I do not need any more" in a single command so usually I just give up on the "upgrade" part.) Maybe that is related?

AltGr commented 4 years ago

Ah, it might be related, indeed... I'll see if I can reproduce with that extra information.

I always struggle hard to make it only rebuild the parts I intend to keep, but found it very hard to combine "rebuild" and "upgrade" and "remove stuff I do not need any more" in a single command so usually I just give up on the "upgrade" part.

Indeed, I don't think I have an easy solution at the moment, there is no compound install/remove or upgrade/remove command. We have been discussing a more generic install command where you could specify a formula and conflicts, but nothing is planned at the moment. Upgrading or not can be triggered using the solver criteria, though, maybe you could find a workaround with something like opam remove -a --criteria="-removed,-count[hidden-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new" ("upgrade criteria" as extracted from `opam config report, they will differ for 2.0.7).

Maybe the most useful solution would be to add a flag that does something similar to remove -a, but could be applied to install/reinstall/upgrade commands? opam upgrade --autoremove ?

Also note that you can do opam reinstall --forget-pending, but then it will be you responsibility to reinstall everything properly, so hardly a manageable solution.

RalfJung commented 4 years ago

Maybe the most useful solution would be to add a flag that does something similar to remove -a, but could be applied to install/reinstall/upgrade commands? opam upgrade --autoremove ?

These are not usually autoinstalled packages that I want to get rid of.

What usually happens is that I want to update some library, and then realize the system switch needs rebuilding. (All my switches are system switches. Maybe that is the mistake.^^ But it keeps them smaller.) If I now just opam upgrade it'll rebuild the old version of that library (the library in question is pinned as it is the build-dependency of what I use the switch for); if I just update the pin it'll rebuild some other library that I do not even need any more. These all take a few minutes each to build so that's quite annoying. So I want to uninstall all libraries, upgrade, and rely on the build-dependency-installation to pull them in any more. But I cannot upgrade and uninstall in one step so there are still pointless rebuilds...

AltGr commented 3 years ago

Maybe the most useful solution would be to add a flag that does something similar to remove -a, but could be applied to install/reinstall/upgrade commands? opam upgrade --autoremove ?

These are not usually autoinstalled packages that I want to get rid of.

Ah, actually the opposite could do the trick then ? opam remove PKGS --upgrade-the-rest That would be a shortcut for opam remove PKGS --criteria=<upgrade-criteria>, where <upgrade-criteria> can be extracted from opam config report. Can you check if that works for you ?

RalfJung commented 3 years ago

opam remove PKGS --upgrade-the-rest

Yeah, something like that... though then I still need two transactions if I also want to install something.

aptitude install also supports mixing in deinstallation with aptitude install package1+ package2- (installs package1 and uninstalls package2). If opam install and opam upgrade supported something similar, that would be great: opam upgrade -a package1+ package2-. I recommend reading man aptitude for some further inspiration for useful flags to be used with install.

That would be a shortcut for opam remove PKGS --criteria=, where can be extracted from opam config report. Can you check if that works for you ?

I will try to remember looking this up the next time the problem comes up (aka the next time debian upgrades its ocaml). Thanks!