opam install should show a hint when installing post dependencies

[liyishuai]

My understanding is that an uninstall command should only remove packages, but observed: https://github.com/ocaml-community/cppo/actions/runs/10004261944/job/27652618559?pr=88

Run opam uninstall .
Ok, cppo_ocamlbuild is no longer pinned to git+file:///Users/runner/work/cppo/cppo#HEAD (version dev)
Ok, cppo is no longer pinned to git+file:///Users/runner/work/cppo/cppo#HEAD (version dev)
The following actions will be performed:
  ⊘ remove    fftw3                0.8.5   [uses cppo]
  ⊘ remove    integration1d        0.5.1   [uses cppo]
  ⊘ remove    notty                0.2.3   [uses cppo]
  ⊘ remove    ocp-browser          1.3.6   [uses cppo]
  ⊘ remove    clarity-lang         0.1.0   [uses cppo]
  ⊘ remove    bytearray            1.0.3   [uses cppo]
  ⊘ remove    goblint-cil          2.0.3   [uses cppo]
  ⊘ remove    arg-complete         0.1.0   [uses cppo]
  ⊘ remove    js_of_ocaml-compiler 3.7.1   [uses yojson]
  ⊘ remove    mindstorm-lwt        0.8     [uses cppo]
  ⊘ remove    dryunit              0.5.0   [uses cppo]
  ⊘ remove    asak                 0.3     [uses cppo]
  ⊘ remove    dose3                7.0.0   [uses extlib]
  ⊘ remove    odoc                 2.4.2   [uses cppo]
  ∗ install   ctypes               0.20.2
  ⊘ remove    mdx                  2.4.1   [uses cppo]
  ⊘ remove    lsp                  1.6.0   [uses yojson]
  ⊘ remove    config-file          1.2.1   [uses cppo]
  ⊘ remove    cppo_ocamlbuild      dev
  ⊘ remove    malfunction          0.6     [uses cppo]
  ⊘ remove    ocolor               1.3.1   [uses cppo]
  ⊘ remove    General              0.7.0   [uses cppo]
  ⊘ remove    curve-sampling       0.2.1   [uses cppo]
  ⊘ remove    monomorphic          2.1.0   [uses cppo]
  ⊘ remove    ocaml-top            1.2.0   [uses cppo]
  ⊘ remove    graphql_ppx          1.0.1   [uses cppo]
  ⊘ remove    gettext              0.4.2   [uses cppo]
  ⊘ remove    lambda-term          3.3.2   [uses lwt]
  ⊘ remove    ppx_deriving_yojson  3.5.3   [uses cppo]
  ⊘ remove    mindstorm            0.8     [uses cppo]
  ⊘ remove    cudf                 0.10    [uses extlib]
  ⊘ remove    ppx_yojson_conv_lib  v0.15.0 [uses yojson]
  ⊘ remove    ocp-index            1.3.6   [uses cppo]
  ⊘ remove    reason               3.8.2   [uses merlin-extend]
  ⊘ remove    lwt_react            1.2.0   [uses cppo]
  ⊘ remove    ppx_deriving         4.5-1   [uses cppo]
  ⊘ remove    extlib               1.7.9   [uses cppo]
  ⊘ remove    yojson               1.7.0   [uses cppo]
  ⊘ remove    merlin-extend        0.6.1   [uses cppo]
  ⊘ remove    lwt                  5.7.0   [uses cppo]
  ⊘ remove    ppx_tools            6.6     [uses cppo]
  ↻ recompile cconv                0.5     [uses yojson]
  ↻ recompile logs                 0.7.0   [uses lwt]
  ⊘ remove    ocplib-endian        1.2     [uses cppo]
  ⊘ remove    cppo                 dev
===== ∗ 1   ↻ 2   ⊘ [42](https://github.com/ocaml-community/cppo/actions/runs/10004261944/job/27652618559?pr=88#step:12:43) =====

...

<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
┌─ The following actions failed
│ λ build ctypes 0.20.2
└─ 

[kit-ty-kate]

something weird happens in your logs before this command is ran:

2024-07-19T09:28:17.7282750Z [WARNING] Opam package conf-libffi.2.0.0 depends on the following system package that can no longer be found: libffi
2024-07-19T09:28:49.9008540Z opam: PACKAGES... arguments: Invalid character in package name
2024-07-19T09:28:49.9011660Z       "\027[01mconf-libffi\027[0m\027[m"
2024-07-19T09:28:49.9013340Z Usage: opam install [OPTION]... [PACKAGES]...
2024-07-19T09:28:49.9016690Z Try `opam install --help' or `opam --help' for more information.
2024-07-19T09:28:49.9027820Z ##[endgroup]
2024-07-19T09:28:49.9030590Z Dependencies broken
2024-07-19T09:28:49.9166720Z ##[group]Build and test yojson
2024-07-19T09:28:50.9178940Z # Detecting depexts using vars: arch=x86_64, os=macos, os-distribution=homebrew, os-family=homebrew
2024-07-19T09:28:56.0089110Z [WARNING] Opam package conf-libffi.2.0.0 depends on the following system package that can no longer be found: libffi
2024-07-19T09:29:04.7414220Z # No extra OS packages requirements found.
2024-07-19T09:29:08.4599100Z [WARNING] Opam package conf-libffi.2.0.0 depends on the following system package that can no longer be found: libffi
2024-07-19T09:29:37.6535820Z opam: PACKAGES... arguments: Invalid character in package name
2024-07-19T09:29:37.6537480Z       "\027[01mconf-libffi\027[0m\027[m"
2024-07-19T09:29:37.6538160Z Usage: opam install [OPTION]... [PACKAGES]...
2024-07-19T09:29:37.6539470Z Try `opam install --help' or `opam --help' for more information.

I bet this is the reason why. Do you know where does that come from?

[liyishuai]

Here's the script:

  PACKAGES=`opam list -s --color=never --installable --depends-on cppo,cppo_ocamlbuild`
  echo "Dependants:" $PACKAGES
  for PACKAGE in $PACKAGES
  do
     echo $SKIP_BUILD | tr ' ' '\n' | grep ^$PACKAGE$ > /dev/null &&
       echo Skip $PACKAGE && continue
     OPAMWITHTEST=true
     echo $SKIP_TEST | tr ' ' '\n' | grep ^$PACKAGE$ > /dev/null &&
         OPAMWITHTEST=false
     ([ $OPAMWITHTEST == false ] &&
      echo ::group::Build          $PACKAGE) ||
      echo ::group::Build and test $PACKAGE
     DEPS_FAILED=false
     (opam depext $PACKAGE &&
      opam install --deps-only -t $PACKAGE) || DEPS_FAILED=true
     [ $DEPS_FAILED == false ] && opam install $PACKAGE
     echo ::endgroup::
     [ $DEPS_FAILED == false ] || echo Dependencies broken
  done

What's happening here:

1. Search for packages that depend on cppo or cppo_ocamlbuild, store them in PACKAGES, and print them out:

   Dependants: arg-complete asak bytearray camlimages camomile caqti cconv cconv-ppx clarity-lang config-file curve-sampling dose3 dryunit extlib fftw3 freetds General genprint gettext goblint-cil graphql_ppx hacl-star hdf5 integration1d js_of_ocaml-compiler lbfgs lsp lwt lwt_react malfunction mdx merlin-extend mindstorm mindstorm-lwt monomorphic notty ocaml-top ocolor ocp-browser ocp-index ocp-index-top ocplib-endian odoc oml opam-core override phantom-algebra pla plato ppx_cstubs ppx_deriving ppx_deriving_argparse ppx_deriving_cmdliner ppx_deriving_protobuf ppx_deriving_yojson ppx_getenv ppx_tools reanalyze reed-solomon-erasure rotor sek setr shcaml slap stdcompat stdlib-random touist utop uwt visitors yojson

2. (In the logs you mentioned:) Install the dependencies of yojson, which failed, setting DEPS_FAILED to true.
3. Print "Dependencies broken" and skip the installation of yojson.

Seems like conf-libffi is a transitive dependency of yojson, which I haven't figured out (is there a tool to show the entire dependency tree?). Not sure why OPAM got such a weird package name.

[kit-ty-kate]

I found the reason why opam is installing ctypes:

  <><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
  ┌─ The following actions failed
  │ λ build conf-gfortran 0
  │ λ build ctypes        0.17.1
  └─ 
  ┌─ The following changes have been performed (the rest was aborted)
  │ ⊘ remove    ctypes         0.20.2
  │ ⊘ remove    hacl-star      0.7.1
  │ ⊘ remove    hacl-star-raw  0.7.1
  │ ↘ downgrade lacaml         11.0.10 to 9.2.2
  │ ∗ install   conf-libffi    2.0.0
  │ ∗ install   conf-libffi    2.0.0
  │ ∗ install   ctypes-foreign 0.18.0
  └─ 
  # Run eval $(opam env) to update the current shell environment

  The former state can be restored with:
      /Users/runner/hostedtoolcache/opam/2.1.6/x86_64/opam switch import "/Users/runner/work/cppo/cppo/_opam/.opam-switch/backup/state-20240719103143.export"
  Or you can retry to install your package selection with:
      /Users/runner/hostedtoolcache/opam/2.1.6/x86_64/opam install --restore
Dependencies broken
Build and test opam-core
  # Detecting depexts using vars: arch=x86_64, os=macos, os-distribution=homebrew, os-family=homebrew
  [WARNING] Opam package conf-libffi.2.0.0 depends on the following system package that can no longer be found: libffi
  # No extra OS packages requirements found.
  [WARNING] Opam package conf-libffi.2.0.0 depends on the following system package that can no longer be found: libffi
  opam: PACKAGES... arguments: Invalid character in package name
        "\027[01mconf-libffi\027[0m\027[m"
  Usage: opam install [OPTION]... [PACKAGES]...
  Try `opam install --help' or `opam --help' for more information.

While building oml, ctypes failed to build but ctypes-foreign got installed. However those two packages in these specific versions (newer versions don't have this problem), use post-dependencies and thus since ctypes-foreign is installed and ctypes is not, opam will try to fix its current state by installing ctypes.

Now that said i think we could add [required by ctypes-foreign] in the UI as a hint. I'll change the title and label the issue, we'll look at it when we have some time. Thanks for the report!

PS: as for your conf-libffi issue I believe that it comes from the use of opam-depext (a tool that have been deprecated 3 years ago) together with ocaml/setup-ocaml@v3. Maybe it could be something to report to https://github.com/ocaml/setup-ocaml/issues (i believe v3 should never install opam-depext). The reason is that opam-depext uses the opam binary (opam list --resolve=<pkg>) which can fail to solve, and in which cases returns a conflict message which contains some colours if the tool used ignore errors. I'm not sure how opam-depext would skip the non-zero exit code but it's a deprecated tool meant to be used with opam 2.0 (released 6 years ago) so i don't think it's worth the investigation

[liyishuai]

The problem occured in two settings:

1. macos-13 (x86), setup-ocaml@v2 with OPAM 2.1.6 and depext 1.2.1;
2. macos-latest (ARM), setup-ocaml@v3 with OPAM 2.2.0 and no opam-depext installed (See "List installed packages").

Should I skip the CI for now, and wait for a fix of OPAM?

[kit-ty-kate]

• macos-latest (ARM), setup-ocaml@v3 with OPAM 2.2.0 and no opam-depext installed (See "List installed packages").

the link is wrong (it was a link to setup-ocaml@v2). https://github.com/ocaml-community/cppo/actions/runs/10004261944/job/27652628377?pr=88 doesn't show the error

[liyishuai]

• macos-latest (ARM), setup-ocaml@v3 with OPAM 2.2.0 and no opam-depext installed (See "List installed packages").

the link is wrong (it was a link to setup-ocaml@v2). ocaml-community/cppo/actions/runs/10004261944/job/27652628377?pr=88 doesn't show the error

It's another instance of failing opam uninstall that installs packages. Not sure if it's the same issue or a new one.

[kit-ty-kate]

as i explained above, the "opam uninstall that installs packages" is expected, the UI is not ideal though and we'll fix that in the future.

The problem you have is different. You're installing some packages that do not depend on cppo/cppo_ocamlbuild (because opam list --depends-on only returns a list of package name, not the versions of those packages that actually depend on cppo, for that you need to add --all-versions or wait for https://github.com/ocaml/opam/pull/5375) and then only removing only cppo/cppo_ocamlbuild. You should probably instead of opam uninstall . do opam uninstall -a $PACKAGES. This way the problem should disappear.