opam enhancement: display downgrade reasons

edwintorok commented 9 years ago

opam (both 1.2.0 and 1.2.1-rc2) gave me the solution below for a package install. Initially I thought there is something wrong with the solver, but turns out it has to choose between somewhat equivalent situations:

downgrade ctypes and install latest sodium (1 pkg out of date (ctypes), 1 pkg downgraded)
recompile ctypes and install older sodium (0.2.0) (1 pkg out of date (sodium), 1 pkg changed)

Could opam print some hints on relevant package constraints for the downgrade?

$ opam-1.2.1-rc2-x86_64-Linux  install sodium tgls tsdl iocaml-kernel --cudf=bad --debug 2>log_bad
The following actions will be performed:
  ↘  downgrade ctypes              0.4.0 to 0.3.4  [required by tgls, iocaml-kernel, sodium, tsdl]
  ∗  install   ctypes-foreign      0.4.0           [required by tsdl]
  ∗  install   tgls                0.8.2         
  ∗  install   sodium              0.3.0         
  ∗  install   iocaml-kernel       0.4.6         
  ↻  recompile nocrypto            0.3.1           [uses ctypes]
  ∗  install   tsdl                0.8.1         
  ↻  recompile x509                0.2.1           [uses nocrypto]
  ↻  recompile otr                 0.1.0           [uses nocrypto]
  ↻  recompile erm_xmpp            0.3             [uses nocrypto]
  ↻  recompile tls                 0.3.0           [uses nocrypto]
  ↻  recompile jackline            0.1.0           [uses tls, erm_xmpp, otr, x509]
  ↻  recompile conduit             0.7.2           [uses tls]
  ↻  recompile mirage-conduit      2.0.0           [uses conduit]
  ↻  recompile cohttp              0.15.0          [uses conduit]
  ↻  recompile xen-api-client      0.9.8           [uses cohttp]
  ↻  recompile mirage-http         2.1.0           [uses cohttp, mirage-conduit]
  ↻  recompile cowabloga           0.0.9           [uses cohttp]
  ↻  recompile xe-unikernel-upload 0.4             [uses xen-api-client]
===== ∗  5   ↻  13   ↘  1 =====
Do you want to continue ? [Y/n]

$ opam info sodium.0.3.0|grep depends
             depends: base-bigarray & base-bytes & ocamlfind & ctypes >= 0.3.2 & < 0.4.0
$ opam info sodium.0.2.0|grep depends
             depends: base-bigarray & base-bytes & ocamlfind & ctypes >= 0.3 & ctypes-foreign

After it has the solution it could check where the downgrade constraints come from, and print it for example:

↘  downgrade ctypes              0.4.0 to 0.3.4  [required by tgls, iocaml-kernel, sodium.0.3.0: ctypes >= 0.3.2 & < 0.4.0, tsdl]

cudf files available here just in case: cudf downgrade log downgrade cudf recompile log recompile

AltGr commented 9 years ago

That would be useful indeed; I'll try to do something for that. Note that we can't get this out of the solver, and that the reasons printed are already computed a posteriori by a heuristic

avsm commented 9 years ago

I'm not sure if this is related or a separate bug, but if I have depext 0.3 installed and pin to a new 0.4 dev version, the downgrade reason is bogus:

$ opam pin add depext git://github.com/avsm/opam-depext#fix-detection
depext is now git-pinned to git://github.com/avsm/opam-depext#fix-detection

[depext] git://github.com/avsm/opam-depext#fix-detection updated
[depext] Installing new package description from git://github.com/avsm/opam-depext#fix-detection

depext needs to be reinstalled.
The following actions will be performed:
  - install   ocamlfind 1.5.5                           [required by cmdliner]
  - downgrade cmdliner  0.9.7 to 0.9.2                  [required by depext]
  - upgrade   depext    0.3 to 0.4*   
===== 1 to install | 1 to upgrade | 1 to downgrade =====

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[cmdliner.0.9.2] http://erratique.ch/software/cmdliner/releases/cmdliner-0.9.2.tbz downloaded
[depext] git://github.com/avsm/opam-depext#fix-detection already up-to-date
[ocamlfind.1.5.5] http://download.camlcity.org/download/findlib-1.5.5.tar.gz downloaded

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
-> removed   depext.0.3
-> removed   cmdliner.0.9.7
-> installed ocamlfind.1.5.5
-> installed cmdliner.0.9.2
-> installed depext.0.4
Done.
opam@b5caedeaff59:~/opam-repository$ opam --version
1.2.1
opam@b5caedeaff59:~/opam-repository$ opam upgrade
The following actions will be performed:
  - upgrade   cmdliner 0.9.2 to 0.9.7
  - recompile depext   0.4*                             [uses cmdliner]
===== 1 to recompile | 1 to upgrade =====

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[cmdliner.0.9.7] http://erratique.ch/software/cmdliner/releases/cmdliner-0.9.7.tbz downloaded
[depext] git://github.com/avsm/opam-depext#fix-detection already up-to-date

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
-> removed   depext.0.4
-> removed   cmdliner.0.9.2
-> installed cmdliner.0.9.7
-> installed depext.0.4
Done.

An opam pin -n followed by opam install does not do this downgrade.

AltGr commented 9 years ago

@avsm hm, weird indeed... would be interesting to see the cudf. This was with aspcud 1.9 ?

avsm commented 9 years ago

Yes this was on MacOS X. I dont have the CUDF available, but it was from a clean switch and so should be reproducible by trying a depext pin + version bump I hope.

avsm commented 9 years ago

(It showed up in the Docker builds)

edwintorok commented 9 years ago

Here is another situation where downgrade reasons would be helpful: https://gist.githubusercontent.com/edwintorok/e49f47f308d3eb842af8/raw/00e24a4c8b02f376d1c4399bf75e7fdd0b9f21c6/fixup-1.cudf

[WARNING] A conflict was detected in your installation. This can be caused by updated constraints or conflicts in your installed packages:
            - Conflicting version constraints for bin_prot
            - Conflicting version constraints for textutils
            - No package matches core.<unavailable version>.
            - No package matches core_kernel.<unavailable version>.
            - No package matches custom_printf.<unavailable version>.
            - core_kernel<111.25.00 is not available because it requires OCaml >= 4.01.0 & < 4.02.1.
            - custom_printf<111.21.00 is not available because it requires OCaml >= 4.00.0 & < 4.02.0.

The following dependencies are in cause:
  - core_bench -> bin_prot <= 112.17.00
  - core_bench -> core = 109.35.00 | core = 109.40.00 | core = 109.47.00 | core = 109.53.01 | core = 109.55.00 | core = <unavailable version> | core <= 109.31.00 | core <= 109.38.00 | core <= 109.45.00 | core <= 109.55.02 | core <= 110.01.00 -> pa_test = 109.53.00 -> core_kernel < 111.25.00
  - core_bench -> core = 109.35.00 | core = 109.40.00 | core = 109.47.00 | core = 109.53.01 | core = 109.55.00 | core = <unavailable version> | core <= 109.31.00 | core <= 109.38.00 | core <= 109.45.00 | core <= 109.55.02 | core <= 110.01.00 -> custom_printf = <unavailable version> | custom_printf < 111.21.00
  - core_bench -> textutils = 109.24.00 | textutils <= 111.28.00 | textutils <= 112.01.00 | textutils <= 112.06.00 -> core = 109.35.00 | core = 111.03.00 | core <= 109.31.00 | core <= 109.47.00 | core <= 109.55.00 | core <= 110.01.00 | core <= 111.21.00 -> core_kernel = <unavailable version>
  - core_bench -> textutils = 109.24.00 | textutils <= 111.28.00 | textutils <= 112.01.00 | textutils <= 112.06.00 -> core = 109.35.00 | core = 111.03.00 | core <= 109.31.00 | core <= 109.47.00 | core <= 109.55.00 | core <= 110.01.00 | core <= 111.21.00 -> custom_printf = <unavailable version> | custom_printf < 111.21.00
  - core_bench -> textutils = 109.24.00 | textutils <= 111.28.00 | textutils <= 112.01.00 | textutils <= 112.06.00 -> core = 109.35.00 | core = 111.03.00 | core <= 109.31.00 | core <= 109.47.00 | core <= 109.55.00 | core <= 110.01.00 | core <= 111.21.00 -> pa_test = 109.53.00 | pa_test = 110.01.00 -> core_kernel < 111.25.00

You should run "opam upgrade --fixup" to resolve the situation.
$ opam upgrade --cudf=fixup --fixup
The following actions will be performed:
  ↘  downgrade bin_prot            112.24.00 to 112.17.00  [required by core_bench, async_kernel, core, core_kernel]
  ↘  downgrade sexplib             112.24.00 to 112.17.00  [required by uri, textutils, mirage-types-lwt, etc.]
  ↘  downgrade pa_ounit            112.24.00 to 112.17.00  [required by core_kernel, core, async_kernel, etc.]
  ↘  downgrade typerep             112.24.00 to 112.17.00  [required by core_kernel]
  ↘  downgrade pa_test             112.24.00 to 111.08.01  [required by core, async_kernel, core_kernel]
  ↻  recompile uri                 1.8.0                 
  ↻  recompile ipaddr              2.6.1                   [uses sexplib]
  ↻  recompile ezjsonm             0.4.1                 
  ↻  recompile cstruct             1.5.0                   [uses sexplib]
  ↻  recompile cconv               0.3.1                 
  ↻  recompile camlhighlight       3.0                     [uses sexplib]
  ↘  downgrade custom_printf       112.24.00 to 112.17.00  [required by core]
  ↻  recompile pa_bench            112.06.00               [uses pa_ounit]
  ↻  recompile tuntap              1.0.0                   [uses ipaddr]
  ↻  recompile ocsigenserver       2.5                     [uses ipaddr]
  ↻  recompile cow                 1.2.1                 
  ↻  recompile xenstore            1.2.5                 
  ↻  recompile tar-format          0.2.1                   [uses cstruct]
  ↻  recompile nbd                 1.0.2                   [uses cstruct]
  ↻  recompile mstruct             1.3.2                   [uses cstruct]
  ↻  recompile io-page             1.2.0                   [uses cstruct]
  ↻  recompile asn1-combinators    0.1.1                   [uses cstruct]
  ↘  downgrade core_kernel         112.24.00 to 112.17.00
  ↻  recompile aws                 0.0.3                 
  ↻  recompile xenstore_transport  0.9.4                   [uses xenstore]
  ↻  recompile mirage-types        2.2.0                   [uses sexplib]
  ↻  recompile mirage-profile      0.4                     [uses cstruct]
  ↘  downgrade core                112.24.00 to 112.17.00
  ↻  recompile vhd-format          0.7.1                 
  ↻  recompile mirage-types-lwt    2.2.0                 
  ↻  recompile mirage-net-unix     2.2.0                 
  ↻  recompile mirage-clock-xen    1.0.0                   [uses mirage-types]
  ↻  recompile mirage-clock-unix   1.0.0                   [uses mirage-types]
  ↻  recompile mirage-block-unix   1.2.2                   [uses cstruct]
  ↻  recompile mirage              2.2.1                 
  ↻  recompile mbr-format          0.2                     [uses cstruct, ipaddr]
  ↻  recompile xen-gnt             2.2.0                 
  ↻  recompile shared-memory-ring  1.1.1                   [uses cstruct]
  ↘  downgrade async_kernel        112.24.00 to 112.17.00
  ↻  recompile textutils           112.17.00             
  ↻  recompile opass               1.0.2                 
  ↻  recompile mirage-fs-unix      1.1.4                 
  ↻  recompile fat-filesystem      0.10.2                  [uses cstruct]
  ↻  recompile mirage-xen          2.2.2                 
  ↻  recompile mirage-unix         2.2.2                 
  ↘  downgrade core_bench          112.24.00 to 112.17.00
  ↻  recompile vchan               2.0.2                 
  ↻  recompile mirage-net-xen      1.4.1                 
  ↻  recompile ctypes              0.4.0                   [uses mirage-xen]
  ↻  recompile mirage-console      2.1.3                 
  ↻  recompile tsdl                0.8.1                 
  ↻  recompile sodium              0.4.0*                
  ↻  recompile nocrypto            0.3.1                   [uses mirage-xen]
  ↻  recompile iocaml-kernel       0.4.6                 
  ↻  recompile tcpip               2.3.0                   [uses mirage-unix, mirage-net-unix, mirage-console, etc.]
  ↻  recompile x509                0.2.1                   [uses sexplib]
  ↻  recompile otr                 0.1.0                 
  ↻  recompile erm_xmpp            0.3                     [uses nocrypto]
  ↻  recompile dns                 0.14.0                
  ↻  recompile tls                 0.3.0                 
  ↻  recompile mirage-dns          2.0.0                   [uses dns, mirage-types-lwt]
  ↻  recompile libres3             1.0                   
  ↻  recompile conduit             0.7.2                 
  ↻  recompile mirage-conduit      2.0.0                 
  ↻  recompile cohttp              0.15.0                  [uses conduit, uri]
  ↻  recompile xen-api-client      0.9.8                   [uses uri]
  ↻  recompile mirage-http         2.1.0                 
  ↻  recompile cowabloga           0.0.9                 
  ↻  recompile xe-unikernel-upload 0.4                   
===== ↻  59   ↘  10 =====
Do you want to continue ? [Y/n]

AltGr commented 9 years ago

The following dependencies are in cause: These long disjunctions can be shortened a lot, so that it gets a bit more readable. There is code for that in OWS, I believe, I need to backport it: should yield something like

The following dependencies are in cause:
  - core_bench -> bin_prot <= 112.17.00
  - core_bench -> core <= 110.01.00 -> pa_test = 109.53.00 -> core_kernel < 111.25.00
  - core_bench -> core <= 110.01.00 -> custom_printf = <unavailable version> | custom_printf < 111.21.00
  - core_bench -> textutils <= 112.06.00 -> core <= 111.21.00 -> core_kernel = <unavailable version>
  - core_bench -> textutils <= 112.06.00 -> core <= 111.21.00 -> custom_printf = <unavailable version> | custom_printf < 111.21.00
  - core_bench -> textutils <= 112.06.00 -> core <= 111.21.00 -> pa_test = 110.01.00 -> core_kernel < 111.25.00

It seems indeed some reasons are missing in the list of actions, too, besides printing additional version constraints. It would be helpful if you still had the fixup-actions.dot file (generated along fixup-1.cudf.

Thanks!

edwintorok commented 9 years ago

Here are all the fixup* files:

git clone https://gist.github.com/c1254104fda5d235d2ad.git

AltGr commented 9 years ago

Awesome, thanks!

AltGr commented 9 years ago

For the last example, it's very difficult, because in case of a fixup, we have no way to know the roots of the action (when you do an opam install, causality is computed from the packages you specified). The root cause may actually not appear at all in the graph (e.g. some packages are removed because they conflict with foo, but foo itself is unchanged).

What is done currently is that we take the actions of input degree 0 and compute from there. I have already much enhanced the display (uncommited yet) so that it doesn't miss relationships (for some reasons, it computed from upgrades but not downgrades). But for now you'll still not get the reason for e.g. downgrade core_bench -- we don't know!

That could be added by an additional mechanism (the current one only relies on the action graph) that checks remaining unexplained nodes for compatibility with the current state and prints the offending constraints. Since that would only be useful for fixup, it may actually be simpler to just display the conflicts before the solution.

To get back to the original issue, it may be interesting, in the case of a command with specified packages, to filter the reasons for a version change to the ones that are really related to the version change: at present the "causes" only refer to package relationships, not actual actions relationships:

downgrade ctypes              0.4.0 to 0.3.4  [required by tgls]

means that ctypes is required by tgls, not that tgls requires the downgrade

AltGr commented 9 years ago

@avsm I could reproduce a similar scenario using the internal solver. I guess finding meaningful explanations to the bogus actions of the internal solver is beyond the scope of my little explanation heuristics. (I may want to start writing a Lawyer module for that)

AltGr commented 9 years ago

Also, see the end of the long post above: when it says

  - install   ocamlfind 1.5.5                           [required by cmdliner]
  - downgrade cmdliner  0.9.7 to 0.9.2                  [required by depext]
  - upgrade   depext    0.3 to 0.4*

it means that cmdliner is required by depext, and that ocamlfind is required by cmdliner (0.9.2). Which is right. But doesn't explain a lot :)

edwintorok commented 9 years ago

On 03/27/2015 10:55 AM, Louis Gesbert wrote:

For the last example, it's very difficult, because in case of a |fixup|, we have no way to know the roots of the action (when you do an |opam install|, causality is computed from the packages you specified). The root cause may actually not appear at all in the graph (e.g. some packages are removed because they conflict with |foo|, but |foo| itself is unchanged).

What is done currently is that we take the actions of input degree 0 and compute from there. I have already much enhanced the display (uncommited yet) so that it doesn't miss relationships (for some reasons, it computed from upgrades but not downgrades). But for now you'll still not get the reason for e.g. |downgrade core_bench| -- we don't know!

That could be added by an additional mechanism (the current one only relies on the action graph) that checks remaining unexplained nodes for compatibility with the current state and prints the offending constraints. Since that would only be useful for |fixup|, it may actually be simpler to just display the conflicts before the solution.

To get back to the original issue, it may be interesting, in the case of a command with specified packages, to filter the reasons for a version change to the ones that are really related to the version change: at present the "causes" only refer to /package/ relationships, not actual /actions/ relationships:

downgrade ctypes 0.4.0 to 0.3.4 [required by tgls]

means that ctypes is required by tgls, not that tgls requires the downgrade

Yep, the 'required by' is great when the action is install, although it'd probably be confusing to change its semantics now.

I thought that you could say 'conflicting with ..., keeping due to ...', or 'due to constraints in ..., needed by ...' but it might get too verbose.

Perhaps just using the unicode pictograms would work, and using 'requested by' for the action, and 'required by' for listing reverse dependencies:

install: [requested by , ∗ newly-installed-pkg, ↗ upgraded-pkg, ↘ downgraded-pkg; required by ? depopt-pkgs, other-pkgs]
recompile: [requested by ∗ new-dep(opt)-pkg, ⊘ removed-dep-pkg,↻ recompiled-dep,↗ upgraded-dep,↘ downgraded-dep; required by , ?depopt-reverse-dep, other-reverse-dep]

Here the two needs to be separated as the 'required by' are dependencies, and the keep is reverse dependencies

downgrade/removal: [requested by , ∗ new-pkg, ↻ upstream-changes-reverse-dep, ↗ upgraded-reverse-dep, ↘ downgraded-reverse-dep; required by: , ?depopt-reverse-dep, other-reverse-dep]

A package is listed in the 'required by' only if that change actually caused the install/downgrade/removal action like you said, i.e. just because a reverse-dep gets upgraded/downgraded it doesn't mean the arrow should be displayed in required by, it should be displayed only if the previously installed version didn't require this action to be taken, and the newly installed one does. The '*' should be easier as it should always be traceable back to an install root requested on the cmdline. The '?' would be displayed if the reverse-dep is just a depopt and wouldn't cause the install on its own.

Not sure whether packages should be listed twice (both in requested by and required by), at least for the install action that'd look odd, so perhaps install should be kept closer to current situation (unicode pictogram only shown if that action triggered the install): install: [required by , ∗ newly-installed-pkg, ↗ upgraded-pkg, ↘ downgraded-pkg, ? depopt-pkgs, other-pkgs]

Not sure how to represent that the action is (or was) a direct cmdline action (opam install/opam remove), I marked this with (direct cmdline action now), and (install root due to old cmdline action). It needs a better word and a unicode pictogram :)

edwintorok commented 9 years ago

On 03/27/2015 10:55 AM, Louis Gesbert wrote:

What is done currently is that we take the actions of input degree 0 and compute from there. I have already much enhanced the display (uncommited yet) so that it doesn't miss relationships (for some reasons, it computed from upgrades but not downgrades). But for now you'll still not get the reason for e.g. |downgrade core_bench| -- we don't know!

I think the reason is "upstream changes": core_bench.112.24.00 doesn't exist (it was removed from the opam repository), which brings along all the core downgrades and recompilations. I think that in opam's output is a hint to that (or to opam's confusion), except its core_bench itself that is unavailable not the other ones. So the downgrade reason could be something like: ↘ downgrade core_bench 112.24.00 to 112.17.00 [package version 112.24.00 removed upstream]

AltGr commented 9 years ago

I think that in opam's output is a hint to that (or to opam's confusion), except its core_bench itself that is unavailable not the other ones.

This one should no longer appear since my latest patches, I now pass along the version maps, allowing the original referred to version to be printed.

I am beginning to see an algorithm that could improve, but that could get quite complicated -- actually, it already is, trying to cover a graph in a custom order, extracting likely reasons depending on node relationships -- so it should probably be done as a separate pass, either before or after (can't decide yet). For upgrade, downgrade or removal actions, we would take the version that used to be installed and check:

its availability (if unavailable, it's "upstream changes")
its conflicts that hold in the new package universe (if any, they forced the change: "Conflicts with bar.new-version"). Note that conflicts are symmetric, so we should include conflicts from other packages (in the new universe) referring to this old version.
its dependencies that aren't satisfied in the new package universe (if any, these triggered the change: "Uses bar" -- or e.g "Used bar < 2" when bar is being upgraded to 2 ?)
its reverse dependencies, i.e. all dependencies of new packages on foo that aren't satisfied by the old version: ">= 1.0 Required by bar.2"

The current algorithm processes only on the action graph, so doesn't have (direct) access to the package universes, availability information, dependencies or conflicts.

Also note that we may still be interested in relations that are not of real causality, in the case of an opam upgrade without arguments for example: foo may be upgraded just because of the optimisation criteria, but is it still worthwhile to display that it depends on bar even if the upgrade of bar didn't really force the upgrade of foo ?

ocaml / opam

opam enhancement: display downgrade reasons #2058