opam upgrade misreports installed packages

jonsterling commented 1 year ago

Observed behavior

When I run opam upgrade, I receive the following message:

Everything as up-to-date as possible (run with --verbose to show unavailable upgrades).

The following packages are not being upgraded because the new versions conflict with other installed packages:
  - ocaml-base-compiler.5.1.0~alpha1
    ∗ ocaml.5.2.0 is installed and requires ocaml-base-compiler = 5.2.0
However, you may "opam upgrade" these packages explicitly, which will ask permission to downgrade or uninstall the
conflicting packages.
Nothing to do.
# Run eval $(opam env) to update the current shell environment

Why the observed behavior is a malfunction

Note that it says ocaml.5.2.0 is installed; this is not correct, and is very confusing. Indeed, here is the output of opam info ocaml.5.2.0 verifying that only ocaml.5.0.0 is installed:

<><> ocaml: information on all versions <><><><><><><><><><><><><><><><><><>  🐫
name                   ocaml
all-installed-versions 5.0.0 [default /Users/jon/Source/ocaml-forester]

Expected behavior

I am not sure what the expected behavior should be because I am not an expert in opam; it is OK that opam reports that ocaml-base-compiler.5.1.0~alpha1 cannot be installed, but it is not OK that it makes up an untrue reason.

Output of `opam config report`


# opam config report
# opam-version         2.1.0
# self-upgrade         no
# system               arch=arm64 os=macos os-distribution=homebrew os-version=13.3.1
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 7
# repositories         1 (http) (default repo at f38d24d2)
# pinned               1 (git)
# current-switch       /Users/jon/Source/ocaml-forester
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /Users/jon/Source/ocaml-forester/_opam/lib/ocaml/stublibs:/Users/jon/Source/ocaml-forester/_opam/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       5.0.0```

ejgallego commented 1 year ago

Hi @jonsterling , what does opam list show? Can you provide a bit more information on how to reproduce your problem?

jonsterling commented 1 year ago

@ejgallego Here is the output of opam list:

# Packages matching: installed
# Name                # Installed # Synopsis
alcotest              1.7.0       Alcotest is a lightweight and colourful test framework
algaeff               0.2.1       Reusable Effects-Based Components
astring               0.8.5       Alternative String module for OCaml
base-bigarray         base
base-bytes            base        Bytes library distributed with the OCaml compiler
base-domains          base
base-nnp              base        Naked pointers prohibited in the OCaml heap
base-threads          base
base-unix             base
bwd                   2.1.0       Backward lists
chrome-trace          3.7.1       Chrome trace event generation library
cmdliner              1.2.0       Declarative definition of command line interfaces for OCaml
cppo                  1.6.9       Code preprocessor like cpp for OCaml
csexp                 1.5.2       Parsing and printing of S-expressions in Canonical form
domain_shims          0.1.0       A non-parallel implementation of Domains compatible with OCaml 4
domainslib            0.5.0       Nested-parallel programming library
dune                  3.7.1       Fast, portable, and opinionated build system
dune-build-info       3.7.1       Embed build information inside executable
dune-configurator     3.7.1       Helper library for gathering system configuration
dune-rpc              3.6.2       Communicate with dune using rpc
dyn                   3.6.2       Dynamic type
fiber                 3.6.2       Structured concurrency library
fmt                   0.9.0       OCaml Format pretty-printer combinators
forester              2.0         pinned to version 2.0 at git+file:///Users/jon/Source/ocaml-forester#main
lambda-term           3.3.1       Terminal manipulation library for OCaml
lockfree              0.3.1       Lock-free data structures for multicore OCaml
logs                  0.7.0       Logging infrastructure for OCaml
lwt                   5.6.1       Promises and event-driven I/O
lwt_react             1.2.0       Helpers for using React with Lwt
menhir                20230415    An LR(1) parser generator
menhirLib             20230415    Runtime support library for parsers generated by Menhir
menhirSdk             20230415    Compile-time library for auxiliary tools related to Menhir
mew                   0.1.0       Modal editing witch
mew_vi                0.5.0       Modal editing witch, VI interpreter
ocaml                 5.0.0       The OCaml compiler (virtual package)
ocaml-base-compiler   5.0.0       Official release 5.0.0
ocaml-compiler-libs   v0.12.4     OCaml compiler libraries repackaged
ocaml-config          3           OCaml Switch Configuration
ocaml-lsp-server      1.15.1-5.0  LSP Server for OCaml
ocaml-options-vanilla 1           Ensure that OCaml is compiled with no special options enabled
ocaml-syntax-shims    1.0.0       Backport new syntax to older OCaml versions
ocamlbuild            0.14.2      OCamlbuild is a build system with builtin rules to easily build most OCaml projects
ocamlc-loc            3.6.2       Parse ocaml compiler output into structured form
ocamlfind             1.9.6       A library manager for OCaml
ocamlformat-rpc-lib   0.25.1      Auto-formatter for OCaml code (RPC mode)
ocamlgraph            2.0.0       A generic graph library for OCaml
ocp-indent            1.8.1       A simple tool to indent OCaml programs
ocplib-endian         1.2         Optimised functions to read and write int16/32/64 from strings and bigarrays
octavius              1.2.2       Ocamldoc comment syntax parser
omd                   1.3.2       A Markdown frontend in pure OCaml
ordering              3.6.2       Element ordering
orec                  1.0.1       dynamic open records
pp                    1.1.2       Pretty-printing library
ppx_derivers          1.2.1       Shared [@@deriving] plugin registry
ppx_deriving          5.2.1       Type-driven code generation for OCaml
ppx_yojson_conv_lib   v0.15.0     Runtime lib for ppx_yojson_conv
ppxlib                0.29.1      Standard library for ppx rewriters
re                    1.10.4      RE is a regular expression library for OCaml
react                 1.2.2       Declarative events and signals for OCaml
result                1.5         Compatibility Result module
seq                   base        Compatibility package for OCaml's standard iterator type starting from 4.07.
sexplib0              v0.15.1     Library containing the definition of S-expressions and some base converters
spawn                 v0.15.1     Spawning sub-processes
stdlib-shims          0.3.0       Backport some of the new stdlib features to older compiler
stdune                3.6.2       Dune's unstable standard library
talaria-bibtex        0.5         A parser for bibtex files
topkg                 1.0.7       The transitory OCaml software packager
trie                  1.0.0       Strict impure trie tree
uchar                 0.0.2       Compatibility library for OCaml's Uchar module
utop                  2.12.1      Universal toplevel for OCaml
uucp                  15.0.0      Unicode character properties for OCaml
uuseg                 15.0.0      Unicode text segmentation for OCaml
uutf                  1.0.3       Non-blocking streaming Unicode codec for OCaml
xdg                   3.7.1       XDG Base Directory Specification
xmlm                  1.4.0       Streaming XML codec for OCaml
yojson                2.1.0       Yojson is an optimized parsing and printing library for the JSON format
yuujinchou            4.0.0       A domain-specific language for manipulating hierarchical names
zed                   3.2.1       Abstract engine for text edition in OCaml

As you can see, contrary to the comment on Twitter, it is not the case that I have ocaml 5.2.0 installed.

ejgallego commented 1 year ago

Thanks for the info. So IIUC, on that switch, you run opam upgrade and you get that error, correct?

What does opam repos output?

jonsterling commented 1 year ago

@ejgallego Yes, that is correct (and thanks for your help trying to diagnose this!). On this switch, here is the output of a fresh call to opam upgrade:

Everything as up-to-date as possible (run with --verbose to show unavailable upgrades).

The following packages are not being upgraded because the new versions conflict with other installed packages:
  - dune-rpc.3.7.1
  - dyn.3.7.1
    ∗ fiber.3.6.2 is installed and requires dyn = 3.6.2
  - ocaml-base-compiler.5.1.0~alpha1
    ∗ ocaml.5.2.0 is installed and requires ocaml-base-compiler = 5.2.0
  - ocamlc-loc.3.7.1
    ∗ ocaml-lsp-server.1.15.1-5.0 is installed and requires ocamlc-loc (>= 3.5.0 & < 3.7.0)
  - omd.2.0.0~alpha3
    ∗ ocaml-lsp-server.1.15.1-5.0 is installed and requires omd (>= 1.3.2 & < 2.0.0~alpha1)
  - ordering.3.7.1
  - stdune.3.7.1
    ∗ fiber.3.6.2 is installed and requires stdune = 3.6.2
However, you may "opam upgrade" these packages explicitly, which will ask permission to downgrade or uninstall the conflicting packages.
Nothing to do.

Here is the output of a call to opam repos on the same switch:

[NOTE] These are the repositories in use by the current switch. Use '--all' to see all configured repositories.

<><> Repository configuration for switch default ><><><><><><><><><><><><><>  🐫
 1 default https://opam.ocaml.org

jonsterling commented 1 year ago

To reproduce this, I expect you should be able to simply start a completely fresh switch on your own machine. I have done nothing strange, and this actually started on a totally fresh installation of ocaml and opam.

ejgallego commented 1 year ago

Thanks, that output certainly looks weird...

Maybe that's some opam 2.1.0 issue?

Could you test if 2.1.4 does the same?

Looking from here, it seems to me that opam should never consider for upgrade the ocaml packages as these should be marked "avoid-version", but that's new in 2.1 so likely you could be hitting a bug.

ejgallego commented 1 year ago

I can't reproduce the first error, but with opam 2.1.4 I get this which is indeed horribly confusing:

The following packages are not being upgraded because the new versions conflict with other installed packages:
  - dune-rpc.3.7.1
  - dyn.3.7.1
    ∗ fiber.3.6.2 is installed and requires dyn = 3.6.2
  - ocaml.5.2.0
    ∗ dune.3.7.1 is installed and requires ocaml (>= 4.02 & < 4.08~~)
    ∗ ocaml-lsp-server.1.15.1-5.0 is installed and requires ocaml (>= 5.0 & < 5.1)
    ∗ ppxlib.0.29.1 is installed and requires ocaml (>= 4.04.1 & < 5.1.0)
  - ocamlc-loc.3.7.1
    ∗ ocaml-lsp-server.1.15.1-5.0 is installed and requires ocamlc-loc (>= 3.5.0 & < 3.7.0)
  - omd.2.0.0~alpha3
    ∗ ocaml-lsp-server.1.15.1-5.0 is installed and requires omd (>= 1.3.2 & < 2.0.0~alpha1)
  - ordering.3.7.1
  - stdune.3.7.1
    ∗ fiber.3.6.2 is installed and requires stdune = 3.6.2

Part of the mess is that ocaml-lsp-server is dragging an older fiber version which in turn blocks stdune from being upgraded. This is more a mess of the packagers than of opam itself IMHO, the kind of coupling seen here is gonna produce problems.

If get rid of ocaml-lsp-server I can then upgrade normally and then we get:

The following packages are not being upgraded because the new versions conflict with other installed packages:
  - ocaml.5.2.0
    ∗ dune.3.7.1 is installed and requires ocaml (>= 4.02 & < 4.08~~)
    ∗ ppxlib.0.29.1 is installed and requires ocaml (>= 4.04.1 & < 5.1.0)

My hypothesis was that the update blockage was confusing the solver, but that doesn't seem to be the case. If I uninsall ppxlib we get:

The following packages are not being upgraded because the new versions conflict with other installed packages:
  - ocaml.5.2.0
    ∗ dune.3.7.1 is installed and requires ocaml (>= 4.02 & < 4.08~~)

which is total nonsense !

Requesting an explicit install we can see:

$ opam install ocaml.5.2.0
[ERROR] Package conflict!
  * Incompatible packages:
    - (invariant) → ocaml-base-compiler = 5.0.0
    - ocaml >= 5.2.0 → ocaml-variants < 5.2.1~
    You can temporarily relax the switch invariant with `--update-invariant'
  * Missing dependency:
    - ocaml >= 5.2.0 → ocaml-variants < 5.2.1~ → ocaml-beta
    unmet availability conditions: 'enable-ocaml-beta-repository'

No solution found, exiting

which is not much better...

ejgallego commented 1 year ago

I'll stop for now, I can conclude that:

ocaml-lsp-server packaging has an upper bound in ocamlc-loc which is causing lots of pain here (as most upper bounds do in general)
I couldn't reproduce the first error message, which still seems pretty weird
The new avoid-version mechanism in 2.1 need some usability improvements, in particular should not spam users on opam upgrade.

jonsterling commented 1 year ago

@ejgallego I have tried this on a completly different machine with a completely fresh configuration of opam 2.1.4 without any packages of my own installed, and I can easily reproduce exactly the wrong behavior that I originally reported. So I am not sure what is going on in your investigations... Let me give you the details below.

`opam config report`

# opam config report
# opam-version         2.1.4
# self-upgrade         no
# system               arch=arm64 os=macos os-distribution=homebrew os-version=13.3.1
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 7
# repositories         1 (http) (default repo at e14a98f4)
# pinned               0
# current-switch       default
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /Users/au598479/.opam/default/lib/ocaml/stublibs:/Users/au598479/.opam/default/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       5.0.0

`opam repos list`

[NOTE] These are the repositories in use by the current switch. Use '--all' to see all configured repositories.

<><> Repository configuration for switch default ><><><><><><><><><><><><><><><>
 1 default https://opam.ocaml.org

`opam list`

# Packages matching: installed
# Name                # Installed # Synopsis
base-bigarray         base
base-domains          base
base-nnp              base        Naked pointers prohibited in the OCaml heap
base-threads          base
base-unix             base
ocaml                 5.0.0       The OCaml compiler (virtual package)
ocaml-base-compiler   5.0.0       Official release 5.0.0
ocaml-config          3           OCaml Switch Configuration
ocaml-options-vanilla 1           Ensure that OCaml is compiled with no special options enabled

`opam upgrade`

Everything as up-to-date as possible (run with --verbose to show unavailable upgrades).

The following packages are not being upgraded because the new versions conflict with other installed packages:
  - ocaml-base-compiler.5.1.0~alpha1
    -- ocaml.5.2.0 is installed and requires ocaml-base-compiler = 5.2.0
However, you may "opam upgrade" these packages explicitly, which will ask permission to downgrade or uninstall the
conflicting packages.
Nothing to do.

remarks

Observe that with a completely fresh installation on the latest version of opam, it still reports the lie that ocaml.5.2.0 is installed. It therefore seems like the other problems you noticed (e.g. with ocaml-lsp-server's upper bounds, and the behavior of the older version of opam with respect to the ocaml packages) might be orthogonal to the bug being observed here. Please let me know if there are any other commands I can run that would help you to reproduce this behavior.

ejgallego commented 1 year ago

Hi @jonsterling , great, thanks a lot for taking the time to provide careful reproduction instructions, that's very helpful!

I will try and report back.

rjbou commented 1 year ago

I was able to reproduce with 2.1.4 & master. It seems that it is a bug from #3354.

jonsterling commented 1 year ago

@rjbou Wonderful, that is great progress!! Thanks to you and @ejgallego for engaging.

ocaml / opam