search

ocaml / opam

opam is a source-based package manager. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.
https://opam.ocaml.org
Other
1.25k stars 359 forks source link

`opam upgrade -y` fails saying "Upgrade is not possible because of conflicts or packages that are no longer available" #3586

Closed RalfJung closed 3 years ago

RalfJung commented 6 years ago

We are using opam on our CI, and we are getting intermittent failures that look as follows:

The following actions will be performed:
  - recompile coq-igps-builddep dev*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
-> removed   coq-igps-builddep.dev
-> installed coq-igps-builddep.dev
Done.
[WARNING] Upgrade is not possible because of conflicts or packages that are no longer available:
The following dependencies are the cause:
  - ocaml -> ocaml-base-compiler = 4.02.3
      conflict with the base packages of this switch
  - ocaml -> ocaml-system = 4.02.3
      unmet availability conditions: sys-ocaml-version = "4.02.3"
  - ocaml -> ocaml-variants >= 4.02.3
      conflict with the base packages of this switch

This is when running opam upgrade -y --fixup && opam upgrade -y (the --fixup is needed because even with opam 2 I have seen it refuse to perform upgrades in some circumstances otherwise).

Notice the

unmet availability conditions: sys-ocaml-version = "4.02.3"

which is plain wrong, that's exactly the ocaml version we have installed on the system. I will enable verbose logging and see if I can reproduce the issue. The only thing I have found so far that helps reliably is to clear the opam root (which we usually keep as a cache across build jobs).

This is with opam 2.0.0.

More generally, it'd be really nice if there was some documentation on how to use opam on a CI. I have found it to be very hard to write shell scripts interacting with opam in a reliable way. I keep adding hacks to work-around failure modes that opam only shows under some hard to reproduce circumstances, like the above (this is by far not the first issue like this). So far, my general veridct is that opam is not suited for non-interactive use.

RalfJung commented 6 years ago

The -v was not very helpful, the message just changed to:

The following actions will be performed:
  - recompile coq-igps-builddep dev*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
-> removed   coq-igps-builddep.dev
-> installed coq-igps-builddep.dev
Done.
[WARNING] Upgrade is not possible because of conflicts or packages that are no longer available:
The following dependencies are the cause:
  - ocaml -> ocaml-base-compiler = 4.02.3
      conflict with the base packages of this switch
  - ocaml -> ocaml-system = 4.02.3
      unmet availability conditions: sys-ocaml-version = "4.02.3"
  - ocaml -> ocaml-variants >= 4.02.3
      conflict with the base packages of this switch

You may run "opam upgrade --fixup" to let opam fix the current state.
'opam upgrade -yv' failed.

But at least now we know it is the second command that fails. opam upgrade tells me to use opam upgrade --fixup, but in fact that command has just been run and it did not see any problems.

RalfJung commented 6 years ago

Here's the -vv output, in case it means anything to you (it does not to me):

he following actions will be performed:
  - recompile coq-igps-builddep dev*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
-> removed   coq-igps-builddep.dev
-> installed coq-igps-builddep.dev
Done.
+ /usr/bin/ocamlc "-vnum"
- 4.02.3
- 4.02.3
[WARNING] Upgrade is not possible because of conflicts or packages that are no longer available:
The following dependencies are the cause:
  - ocaml -> ocaml-base-compiler = 4.02.3
      conflict with the base packages of this switch
  - ocaml -> ocaml-system = 4.02.3
      unmet availability conditions: sys-ocaml-version = "4.02.3"
  - ocaml -> ocaml-variants >= 4.02.3
      conflict with the base packages of this switch

You may run "opam upgrade --fixup" to let opam fix the current state.
'opam upgrade -yvv' failed.
AltGr commented 6 years ago

This seems weird inded. You shouldn't have to run the --fixup operation, in any case. What I would guess is that you are using a system compiler, which happens not to be detected correctly in the second command, for some reason (PATH issue ?). --debug would be more helpful than --verbose, in this case. If you coud try opam config list, that might give some hints, too.

Thanks for reporting!

RalfJung commented 6 years ago

This is now happening for pretty much all our CI jobs. Our CI has kidn of grind to a halt, I have to figure out something quickly or we cannot even work...

Here's what --debug says

00:00.003  GSTATE                          LOAD-GLOBAL-STATE @ /builds/FP/iris-coq/opamroot
00:00.004  RSTATE                          LOAD-REPOSITORY-STATE @ /builds/FP/iris-coq/opamroot
00:00.110  RSTATE                          Loaded /builds/FP/iris-coq/opamroot/repo/state.cache in 0.106s
00:00.152  RSTATE                          Cache found
00:00.152  STATE                           LOAD-SWITCH-STATE @ default
00:00.201  STATE                           Detected changed packages (marked for reinstall): {}
00:00.205  STATE                           Switch state loaded in 0.053s
00:00.205  FILE(switch-state)              Wrote /builds/FP/iris-coq/opamroot/default/.opam-switch/backup/state-20181010154935.export in 0.000s
+ /usr/bin/ocamlc "-vnum"
- 4.02.3
- 4.02.3
00:00.251  SYSTEM                          [log-217-5a61c5] (in 0.022s) ocamlc -vnum
00:00.333  UPDATE                          update-dev-packages
00:00.333  PARALLEL                        Iterate over 1 task(s) with 3 process(es)
00:00.333  PARALLEL                        Starting job 0 (worker 1/3): 0
00:00.333  UPDATE                          update-dev-package coq-iris-builddep.dev
00:00.333  UPDATE                          update-pinned-package coq-iris-builddep

<><> Synchronising pinned packages ><><><><><><><><><><><><><><><><><><><><><><>
00:00.335  UPDATE                          updating file:///builds/FP/iris-coq/build-dep
00:00.336  RSYNC                           rsync: src=/builds/FP/iris-coq/build-dep/ dst=/builds/FP/iris-coq/opamroot/default/.opam-switch/sources/coq-iris-builddep
00:00.340  PARALLEL                        Next task in job 0: /usr/bin/rsync -rLptgoDrvc --exclude .git --exclude _darcs --exclude .hg --exclude .#* --exclude _opam* --delete --delete-excluded /builds/FP/iris-coq/build-dep/ /builds/FP/iris-coq/opamroot/default/.opam-switch/sources/coq-iris-builddep
Processing  1/1: [coq-iris-builddep.dev: rsync]
+ /usr/bin/rsync "-rLptgoDrvc" "--exclude" ".git" "--exclude" "_darcs" "--exclude" ".hg" "--exclude" ".#*" "--exclude" "_opam*" "--delete" "--delete-excluded" "/builds/FP/iris-coq/build-dep/" "/builds/FP/iris-coq/opamroot/default/.opam-switch/sources/coq-iris-builddep"
- sending incremental file list
- 
- sent 83 bytes  received 12 bytes  190.00 bytes/sec
- total size is 498  speedup is 5.24
[coq-iris-builddep.dev] no changes from file:///builds/FP/iris-coq/build-dep
00:00.347  PARALLEL                        Collected task for job 0 (ret:0)
00:00.348  PARALLEL                        Job 0 finished
00:00.348  SWACT                           add-to-reinstall unpinned_only:false packages:{}
00:00.348  CLIENT                          UPGRADE coq-iris-builddep

00:00.373  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.373  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.422  STATE                           ERR: undefined filter variable in dependencies of hts_shrink.1.0.0: test
00:00.424  STATE                           ERR: undefined filter variable in dependencies of influxdb-async.0.1.0: test
00:00.424  STATE                           ERR: undefined filter variable in dependencies of influxdb-lwt.0.1.0: test
00:00.624  CLIENT                          Orphans: (changes: , transitive: true) -> full {}, versions { ocaml-system.4.02.3 }
00:00.633  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.633  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.680  STATE                           ERR: undefined filter variable in dependencies of hts_shrink.1.0.0: test
00:00.682  STATE                           ERR: undefined filter variable in dependencies of influxdb-async.0.1.0: test
00:00.682  STATE                           ERR: undefined filter variable in dependencies of influxdb-lwt.0.1.0: test
00:00.849  SOLVER                          resolve request=install:() remove:() upgrade:(ocamlfind & ocaml-system & ocaml & num & coq-unicoq & coq-stdpp & coq-mtac & coq-mathcomp-ssreflect & coq-iris-builddep & coq & conf-m4 & camlp5 & base-unix & base-threads & base-num & base-bigarray)
00:01.539  SOLVER                          Load cudf universe (depopts:false, build:true, post:true)
00:01.583  FILTER                          ERR: value_bool: "/builds/FP/iris-coq/opamroot/default/doc"
00:01.583  FILTER                          ERR: value_bool: "/builds/FP/iris-coq/opamroot/default/doc"
00:01.583  FILTER                          ERR: value_bool: "/builds/FP/iris-coq/opamroot/default/doc"
00:01.583  FILTER                          ERR: value_bool: "/builds/FP/iris-coq/opamroot/default/doc"
00:01.587  FILTER                          ERR: value_bool: "/builds/FP/iris-coq/opamroot/default/doc"
00:01.620  CUDF                            resolve request=install:() remove:() upgrade:(base-bigarray (>= 1) & base-num (>= 1) & base-threads (>= 1) & base-unix (>= 1) & camlp5 (>= 18) & conf-m4 (>= 1) & coq (>= 36) & coq-iris-builddep (>= 1) & coq-mathcomp-ssreflect (>= 5) & coq-mtac (>= 1) & coq-stdpp (>= 2) & coq-unicoq (>= 4) & num (>= 1) & ocaml (>= 39) & ocaml-system & ocamlfind (>= 25))
00:01.620  SOLVER                          Calling solver builtin-mccs+glpk with criteria -removed,-count[version-lag,solution],-new
00:02.430  CUDF                            Solver call done in 0.810
00:02.430  CLIENT                          conflict!
The following dependencies couldn't be met:
  - ocaml -> ocaml-base-compiler = 4.02.3
      conflict with the base packages of this switch
  - ocaml -> ocaml-system = 4.02.3
      unmet availability conditions: sys-ocaml-version = "4.02.3"
  - ocaml -> ocaml-variants >= 4.02.3
      conflict with the base packages of this switch

00:02.495  SYSTEM                          rm /builds/FP/iris-coq/opamroot/default/.opam-switch/backup/state-20181010154935.export
'opam upgrade -yvv -a coq-iris-builddep --debug' failed.
ERROR: Job failed: exit code 1
RalfJung commented 6 years ago

What strikes me as odd is

+ /usr/bin/ocamlc "-vnum"
- 4.02.3
- 4.02.3

Usually this is just

+ /usr/bin/ocamlc "-vnum"
- 4.02.3
RalfJung commented 6 years ago

Hm, actually it does not seem to affect all our CI jobs. Just those that have some package pinned to a git repo. Not that that makes any sense...^^

rjbou commented 6 years ago

It seems from the logs that opam tries to upgrade the ocaml package also (cudf request). It is possible, if the switch have an unlocked base. We can't reproduce, as it is a CI problem, can you share the scripts or at least the process that leads to that error?

RalfJung commented 6 years ago

The script is at https://gitlab.mpi-sws.org/FP/iris-ci/blob/master/prepare-opam.sh. We don't want an "unlocked base" but maybe we got it accidentally?

I am trying to get a opam config list, but of course now that I want it to happen I cannot reproduce. I bet tomorrow I will have emails again for all our nightlies job having failed.

RalfJung commented 6 years ago

I also got opam config list now (I still have no idea why, of these two CI jobs I am looking at, one has the problem and one does not...)

<><> Global opam variables ><><><><><><><><><><><><><><><><><><><><><><><><><><>
arch              x86_64
          # Inferred from system
jobs              39
          # The number of parallel jobs set up in opam configuration
make              make
          # The 'make' command to use
opam-version      2.0.0
          # The currently running opam version
os                linux
          # Inferred from system
os-distribution   debian
          # Inferred from system
os-family         debian
          # Inferred from system
os-version        9
          # Inferred from system
root              /builds/FP/LambdaRust-coq/opamroot
          # The current opam root directory
switch            ocaml-system
          # The identifier of the current switch
sys-ocaml-version 4.02.3
                  4.02.3
                  4.02.3
                  4.02.3
                  4.02.3
                  4.02.3
                  4.02.3
          # OCaml version present on your system independently of opam, if any

<><> Configuration variables from the current switch ><><><><><><><><><><><><><>
prefix   /builds/FP/LambdaRust-coq/opamroot/ocaml-system
lib      /builds/FP/LambdaRust-coq/opamroot/ocaml-system/lib
bin      /builds/FP/LambdaRust-coq/opamroot/ocaml-system/bin
sbin     /builds/FP/LambdaRust-coq/opamroot/ocaml-system/sbin
share    /builds/FP/LambdaRust-coq/opamroot/ocaml-system/share
doc      /builds/FP/LambdaRust-coq/opamroot/ocaml-system/doc
etc      /builds/FP/LambdaRust-coq/opamroot/ocaml-system/etc
man      /builds/FP/LambdaRust-coq/opamroot/ocaml-system/man
toplevel /builds/FP/LambdaRust-coq/opamroot/ocaml-system/lib/toplevel
stublibs /builds/FP/LambdaRust-coq/opamroot/ocaml-system/lib/stublibs
user     ci
group    ci

<><> Package variables ('opam config list PKG' to show) <><><><><><><><><><><><>
PKG:name       # Name of the package
PKG:version    # Version of the package
PKG:depends    # Resolved direct dependencies of the package
PKG:installed  # Whether the package is installed
PKG:enable     # Takes the value "enable" or "disable" depending on whether the package is installed
PKG:pinned     # Whether the package is pinned
PKG:bin        # Binary directory for this package
PKG:sbin       # System binary directory for this package
PKG:lib        # Library directory for this package
PKG:man        # Man directory for this package
PKG:doc        # Doc directory for this package
PKG:share      # Share directory for this package
PKG:etc        # Etc directory for this package
PKG:build      # Directory where the package was built
PKG:hash       # Hash of the package archive
PKG:dev        # True if this is a development package
PKG:build-id   # A hash identifying the precise package version with all its dependencies

Again it seems rather odd that it reports "4.02.3" seven times, does it not?

EDIT: Ah, I found why CI kept being green. It had the failure but didn't propagate it properly. My mistake.

RalfJung commented 6 years ago

Now, that is strange... if I just make CI try the same thing again, it works! (I reproduced this on a second project.)

I also identified that we have one CI runner where the failure always occurred, and one where it never did (for this particular project). The runners are using the same Docker image, but have their own separate cached opam roots. I downloaded the caches and will see if I can find any difference. I can also provide them to you, if you want.

RalfJung commented 6 years ago

Here's the list of files that differ between the two:

Only in cache-2/opamroot/download-cache/git/objects/pack: pack-42fded6b41e8b029a2401f32430b7de63f2bf96f.idx
Only in cache-2/opamroot/download-cache/git/objects/pack: pack-42fded6b41e8b029a2401f32430b7de63f2bf96f.pack
Only in cache-1/opamroot/download-cache/git/objects/pack: pack-a42369d609a527eb27d355deaf57796c3503337f.idx
Only in cache-1/opamroot/download-cache/git/objects/pack: pack-a42369d609a527eb27d355deaf57796c3503337f.pack
Only in cache-2/opamroot/download-cache/md5: 7e
Only in cache-1/opamroot/log: camlp5-466-126656.env
Only in cache-1/opamroot/log: camlp5-466-126656.info
Only in cache-1/opamroot/log: camlp5-466-126656.out
Only in cache-1/opamroot/log: camlp5-466-562576.env
Only in cache-1/opamroot/log: camlp5-466-562576.info
Only in cache-1/opamroot/log: camlp5-466-562576.out
Only in cache-1/opamroot/log: camlp5-466-736596.env
Only in cache-1/opamroot/log: camlp5-466-736596.info
Only in cache-1/opamroot/log: camlp5-466-736596.out
Only in cache-1/opamroot/log: conf-m4-466-8c0dd3.env
Only in cache-1/opamroot/log: conf-m4-466-8c0dd3.info
Only in cache-1/opamroot/log: conf-m4-466-8c0dd3.out
Only in cache-1/opamroot/log: coq-466-384e1f.env
Only in cache-1/opamroot/log: coq-466-384e1f.info
Only in cache-1/opamroot/log: coq-466-384e1f.out
Only in cache-1/opamroot/log: coq-466-427793.env
Only in cache-1/opamroot/log: coq-466-427793.info
Only in cache-1/opamroot/log: coq-466-427793.out
Only in cache-1/opamroot/log: coq-466-60c5c2.env
Only in cache-1/opamroot/log: coq-466-60c5c2.info
Only in cache-1/opamroot/log: coq-466-60c5c2.out
Only in cache-1/opamroot/log: coq-466-a8fa88.env
Only in cache-1/opamroot/log: coq-466-a8fa88.info
Only in cache-1/opamroot/log: coq-466-a8fa88.out
Only in cache-1/opamroot/log: coq-466-f32074.env
Only in cache-1/opamroot/log: coq-466-f32074.info
Only in cache-1/opamroot/log: coq-466-f32074.out
Only in cache-1/opamroot/log: coq-iris-265-2dea01.env
Only in cache-1/opamroot/log: coq-iris-265-2dea01.info
Only in cache-1/opamroot/log: coq-iris-265-2dea01.out
Only in cache-1/opamroot/log: coq-iris-265-ad6566.env
Only in cache-1/opamroot/log: coq-iris-265-ad6566.info
Only in cache-1/opamroot/log: coq-iris-265-ad6566.out
Only in cache-1/opamroot/log: coq-iris-265-e7966f.env
Only in cache-1/opamroot/log: coq-iris-265-e7966f.info
Only in cache-1/opamroot/log: coq-iris-265-e7966f.out
Only in cache-2/opamroot/log: coq-iris-267-2dea01.env
Only in cache-2/opamroot/log: coq-iris-267-2dea01.info
Only in cache-2/opamroot/log: coq-iris-267-2dea01.out
Only in cache-2/opamroot/log: coq-iris-267-ad6566.env
Only in cache-2/opamroot/log: coq-iris-267-ad6566.info
Only in cache-2/opamroot/log: coq-iris-267-ad6566.out
Only in cache-2/opamroot/log: coq-iris-267-e7966f.env
Only in cache-2/opamroot/log: coq-iris-267-e7966f.info
Only in cache-2/opamroot/log: coq-iris-267-e7966f.out
Only in cache-1/opamroot/log: coq-iris-466-268fea.env
Only in cache-1/opamroot/log: coq-iris-466-268fea.info
Only in cache-1/opamroot/log: coq-iris-466-268fea.out
Only in cache-1/opamroot/log: coq-iris-466-f53971.env
Only in cache-1/opamroot/log: coq-iris-466-f53971.info
Only in cache-1/opamroot/log: coq-iris-466-f53971.out
Only in cache-1/opamroot/log: coq-stdpp-265-05296d.env
Only in cache-1/opamroot/log: coq-stdpp-265-05296d.info
Only in cache-1/opamroot/log: coq-stdpp-265-05296d.out
Only in cache-1/opamroot/log: coq-stdpp-265-6cc03f.env
Only in cache-1/opamroot/log: coq-stdpp-265-6cc03f.info
Only in cache-1/opamroot/log: coq-stdpp-265-6cc03f.out
Only in cache-1/opamroot/log: coq-stdpp-265-d8a4ae.env
Only in cache-1/opamroot/log: coq-stdpp-265-d8a4ae.info
Only in cache-1/opamroot/log: coq-stdpp-265-d8a4ae.out
Only in cache-2/opamroot/log: coq-stdpp-267-05296d.env
Only in cache-2/opamroot/log: coq-stdpp-267-05296d.info
Only in cache-2/opamroot/log: coq-stdpp-267-05296d.out
Only in cache-2/opamroot/log: coq-stdpp-267-6cc03f.env
Only in cache-2/opamroot/log: coq-stdpp-267-6cc03f.info
Only in cache-2/opamroot/log: coq-stdpp-267-6cc03f.out
Only in cache-2/opamroot/log: coq-stdpp-267-d8a4ae.env
Only in cache-2/opamroot/log: coq-stdpp-267-d8a4ae.info
Only in cache-2/opamroot/log: coq-stdpp-267-d8a4ae.out
Only in cache-1/opamroot/log: coq-stdpp-466-58a313.env
Only in cache-1/opamroot/log: coq-stdpp-466-58a313.info
Only in cache-1/opamroot/log: coq-stdpp-466-58a313.out
Only in cache-1/opamroot/log: coq-stdpp-466-74316f.env
Only in cache-1/opamroot/log: coq-stdpp-466-74316f.info
Only in cache-1/opamroot/log: coq-stdpp-466-74316f.out
Only in cache-2/opamroot/log: log-185-5a61c5.stdout
Only in cache-1/opamroot/log: log-193-5a61c5.stdout
Only in cache-2/opamroot/log: log-245-5a61c5.stdout
Only in cache-2/opamroot/log: log-245-d2d111.stdout
Only in cache-2/opamroot/log: log-251-5a61c5.env
Only in cache-2/opamroot/log: log-251-5a61c5.info
Only in cache-2/opamroot/log: log-251-5a61c5.out
Only in cache-2/opamroot/log: log-251-5a61c5.stdout
Only in cache-2/opamroot/log: log-251-dd682e.env
Only in cache-2/opamroot/log: log-251-dd682e.info
Only in cache-2/opamroot/log: log-251-dd682e.out
Only in cache-2/opamroot/log: log-263-ccabea.stdout
Only in cache-1/opamroot/log: log-265-58c514.env
Only in cache-1/opamroot/log: log-265-58c514.info
Only in cache-1/opamroot/log: log-265-58c514.out
Only in cache-1/opamroot/log: log-265-5a61c5.env
Only in cache-1/opamroot/log: log-265-5a61c5.info
Only in cache-1/opamroot/log: log-265-5a61c5.out
Only in cache-1/opamroot/log: log-265-5a61c5.stdout
Only in cache-1/opamroot/log: log-265-7afd23.env
Only in cache-1/opamroot/log: log-265-7afd23.info
Only in cache-1/opamroot/log: log-265-7afd23.out
Only in cache-1/opamroot/log: log-265-8b8a2d.env
Only in cache-1/opamroot/log: log-265-8b8a2d.info
Only in cache-1/opamroot/log: log-265-8b8a2d.out
Only in cache-1/opamroot/log: log-265-a00279.env
Only in cache-1/opamroot/log: log-265-a00279.info
Only in cache-1/opamroot/log: log-265-a00279.out
Only in cache-1/opamroot/log: log-265-b76d46.env
Only in cache-1/opamroot/log: log-265-b76d46.info
Only in cache-1/opamroot/log: log-265-b76d46.out
Only in cache-1/opamroot/log: log-265-c743ac.env
Only in cache-1/opamroot/log: log-265-c743ac.info
Only in cache-1/opamroot/log: log-265-c743ac.out
Only in cache-1/opamroot/log: log-265-d2c37b.env
Only in cache-1/opamroot/log: log-265-d2c37b.info
Only in cache-1/opamroot/log: log-265-d2c37b.out
Only in cache-1/opamroot/log: log-265-d2d111.env
Only in cache-1/opamroot/log: log-265-d2d111.info
Only in cache-1/opamroot/log: log-265-d2d111.out
Only in cache-1/opamroot/log: log-265-dd682e.env
Only in cache-1/opamroot/log: log-265-dd682e.info
Only in cache-1/opamroot/log: log-265-dd682e.out
Only in cache-1/opamroot/log: log-265-e4db2f.env
Only in cache-1/opamroot/log: log-265-e4db2f.info
Only in cache-1/opamroot/log: log-265-e4db2f.out
Only in cache-2/opamroot/log: log-267-58c514.env
Only in cache-2/opamroot/log: log-267-58c514.info
Only in cache-2/opamroot/log: log-267-58c514.out
Only in cache-2/opamroot/log: log-267-5a61c5.env
Only in cache-2/opamroot/log: log-267-5a61c5.info
Only in cache-2/opamroot/log: log-267-5a61c5.out
Only in cache-2/opamroot/log: log-267-5a61c5.stdout
Only in cache-2/opamroot/log: log-267-7afd23.env
Only in cache-2/opamroot/log: log-267-7afd23.info
Only in cache-2/opamroot/log: log-267-7afd23.out
Only in cache-2/opamroot/log: log-267-8b8a2d.env
Only in cache-2/opamroot/log: log-267-8b8a2d.info
Only in cache-2/opamroot/log: log-267-8b8a2d.out
Only in cache-2/opamroot/log: log-267-a00279.env
Only in cache-2/opamroot/log: log-267-a00279.info
Only in cache-2/opamroot/log: log-267-a00279.out
Only in cache-2/opamroot/log: log-267-b76d46.env
Only in cache-2/opamroot/log: log-267-b76d46.info
Only in cache-2/opamroot/log: log-267-b76d46.out
Only in cache-2/opamroot/log: log-267-c743ac.env
Only in cache-2/opamroot/log: log-267-c743ac.info
Only in cache-2/opamroot/log: log-267-c743ac.out
Only in cache-2/opamroot/log: log-267-d2c37b.env
Only in cache-2/opamroot/log: log-267-d2c37b.info
Only in cache-2/opamroot/log: log-267-d2c37b.out
Only in cache-2/opamroot/log: log-267-d2d111.env
Only in cache-2/opamroot/log: log-267-d2d111.info
Only in cache-2/opamroot/log: log-267-d2d111.out
Only in cache-2/opamroot/log: log-267-dd682e.env
Only in cache-2/opamroot/log: log-267-dd682e.info
Only in cache-2/opamroot/log: log-267-dd682e.out
Only in cache-2/opamroot/log: log-267-e4db2f.env
Only in cache-2/opamroot/log: log-267-e4db2f.info
Only in cache-2/opamroot/log: log-267-e4db2f.out
Only in cache-2/opamroot/log: log-306-5a61c5.stdout
Only in cache-1/opamroot/log: log-314-5a61c5.stdout
Only in cache-2/opamroot/log: log-440-5a61c5.stdout
Only in cache-1/opamroot/log: log-448-5a61c5.stdout
Only in cache-2/opamroot/log: log-458-5a61c5.stdout
Only in cache-1/opamroot/log: log-466-03b1a4.env
Only in cache-1/opamroot/log: log-466-03b1a4.info
Only in cache-1/opamroot/log: log-466-03b1a4.out
Only in cache-1/opamroot/log: log-466-06fa5f.env
Only in cache-1/opamroot/log: log-466-06fa5f.info
Only in cache-1/opamroot/log: log-466-06fa5f.out
Only in cache-1/opamroot/log: log-466-0ee25e.env
Only in cache-1/opamroot/log: log-466-0ee25e.info
Only in cache-1/opamroot/log: log-466-0ee25e.out
Only in cache-1/opamroot/log: log-466-0f1b61.env
Only in cache-1/opamroot/log: log-466-0f1b61.info
Only in cache-1/opamroot/log: log-466-0f1b61.out
Only in cache-1/opamroot/log: log-466-1872ec.env
Only in cache-1/opamroot/log: log-466-1872ec.info
Only in cache-1/opamroot/log: log-466-1872ec.out
Only in cache-1/opamroot/log: log-466-201c9f.env
Only in cache-1/opamroot/log: log-466-201c9f.info
Only in cache-1/opamroot/log: log-466-201c9f.out
Only in cache-1/opamroot/log: log-466-2c871f.env
Only in cache-1/opamroot/log: log-466-2c871f.info
Only in cache-1/opamroot/log: log-466-2c871f.out
Only in cache-1/opamroot/log: log-466-320155.env
Only in cache-1/opamroot/log: log-466-320155.info
Only in cache-1/opamroot/log: log-466-320155.out
Only in cache-1/opamroot/log: log-466-4197d2.env
Only in cache-1/opamroot/log: log-466-4197d2.info
Only in cache-1/opamroot/log: log-466-4197d2.out
Only in cache-1/opamroot/log: log-466-42737e.env
Only in cache-1/opamroot/log: log-466-42737e.info
Only in cache-1/opamroot/log: log-466-42737e.out
Only in cache-1/opamroot/log: log-466-42ef32.env
Only in cache-1/opamroot/log: log-466-42ef32.info
Only in cache-1/opamroot/log: log-466-42ef32.out
Only in cache-1/opamroot/log: log-466-4519e5.env
Only in cache-1/opamroot/log: log-466-4519e5.info
Only in cache-1/opamroot/log: log-466-4519e5.out
Only in cache-1/opamroot/log: log-466-487c34.env
Only in cache-1/opamroot/log: log-466-487c34.info
Only in cache-1/opamroot/log: log-466-487c34.out
Only in cache-1/opamroot/log: log-466-489228.env
Only in cache-1/opamroot/log: log-466-489228.info
Only in cache-1/opamroot/log: log-466-489228.out
Only in cache-1/opamroot/log: log-466-4d7566.env
Only in cache-1/opamroot/log: log-466-4d7566.info
Only in cache-1/opamroot/log: log-466-4d7566.out
Only in cache-1/opamroot/log: log-466-4eee66.env
Only in cache-1/opamroot/log: log-466-4eee66.info
Only in cache-1/opamroot/log: log-466-4eee66.out
Only in cache-1/opamroot/log: log-466-4f1270.env
Only in cache-1/opamroot/log: log-466-4f1270.info
Only in cache-1/opamroot/log: log-466-4f1270.out
Only in cache-1/opamroot/log: log-466-520fac.env
Only in cache-1/opamroot/log: log-466-520fac.info
Only in cache-1/opamroot/log: log-466-520fac.out
Only in cache-1/opamroot/log: log-466-52d546.env
Only in cache-1/opamroot/log: log-466-52d546.info
Only in cache-1/opamroot/log: log-466-52d546.out
Only in cache-1/opamroot/log: log-466-548b09.env
Only in cache-1/opamroot/log: log-466-548b09.info
Only in cache-1/opamroot/log: log-466-548b09.out
Only in cache-1/opamroot/log: log-466-559acc.env
Only in cache-1/opamroot/log: log-466-559acc.info
Only in cache-1/opamroot/log: log-466-559acc.out
Only in cache-1/opamroot/log: log-466-5a61c5.env
Only in cache-1/opamroot/log: log-466-5a61c5.info
Only in cache-1/opamroot/log: log-466-5a61c5.out
Only in cache-1/opamroot/log: log-466-5a61c5.stdout
Only in cache-1/opamroot/log: log-466-5dd0bd.env
Only in cache-1/opamroot/log: log-466-5dd0bd.info
Only in cache-1/opamroot/log: log-466-5dd0bd.out
Only in cache-1/opamroot/log: log-466-5f1023.env
Only in cache-1/opamroot/log: log-466-5f1023.info
Only in cache-1/opamroot/log: log-466-5f1023.out
Only in cache-1/opamroot/log: log-466-60459a.env
Only in cache-1/opamroot/log: log-466-60459a.info
Only in cache-1/opamroot/log: log-466-60459a.out
Only in cache-1/opamroot/log: log-466-629056.env
Only in cache-1/opamroot/log: log-466-629056.info
Only in cache-1/opamroot/log: log-466-629056.out
Only in cache-1/opamroot/log: log-466-66af47.env
Only in cache-1/opamroot/log: log-466-66af47.info
Only in cache-1/opamroot/log: log-466-66af47.out
Only in cache-1/opamroot/log: log-466-6aa130.env
Only in cache-1/opamroot/log: log-466-6aa130.info
Only in cache-1/opamroot/log: log-466-6aa130.out
Only in cache-1/opamroot/log: log-466-6cc03f.env
Only in cache-1/opamroot/log: log-466-6cc03f.info
Only in cache-1/opamroot/log: log-466-6cc03f.out
Only in cache-1/opamroot/log: log-466-6d07ae.env
Only in cache-1/opamroot/log: log-466-6d07ae.info
Only in cache-1/opamroot/log: log-466-6d07ae.out
Only in cache-1/opamroot/log: log-466-6ef1df.env
Only in cache-1/opamroot/log: log-466-6ef1df.info
Only in cache-1/opamroot/log: log-466-6ef1df.out
Only in cache-1/opamroot/log: log-466-74f651.env
Only in cache-1/opamroot/log: log-466-74f651.info
Only in cache-1/opamroot/log: log-466-74f651.out
Only in cache-1/opamroot/log: log-466-81a3ce.env
Only in cache-1/opamroot/log: log-466-81a3ce.info
Only in cache-1/opamroot/log: log-466-81a3ce.out
Only in cache-1/opamroot/log: log-466-8bc1db.env
Only in cache-1/opamroot/log: log-466-8bc1db.info
Only in cache-1/opamroot/log: log-466-8bc1db.out
Only in cache-1/opamroot/log: log-466-910341.env
Only in cache-1/opamroot/log: log-466-910341.info
Only in cache-1/opamroot/log: log-466-910341.out
Only in cache-1/opamroot/log: log-466-9de511.env
Only in cache-1/opamroot/log: log-466-9de511.info
Only in cache-1/opamroot/log: log-466-9de511.out
Only in cache-1/opamroot/log: log-466-9e78df.env
Only in cache-1/opamroot/log: log-466-9e78df.info
Only in cache-1/opamroot/log: log-466-9e78df.out
Only in cache-1/opamroot/log: log-466-a00279.env
Only in cache-1/opamroot/log: log-466-a00279.info
Only in cache-1/opamroot/log: log-466-a00279.out
Only in cache-1/opamroot/log: log-466-a3393d.env
Only in cache-1/opamroot/log: log-466-a3393d.info
Only in cache-1/opamroot/log: log-466-a3393d.out
Only in cache-1/opamroot/log: log-466-a5b00a.env
Only in cache-1/opamroot/log: log-466-a5b00a.info
Only in cache-1/opamroot/log: log-466-a5b00a.out
Only in cache-1/opamroot/log: log-466-a7c577.env
Only in cache-1/opamroot/log: log-466-a7c577.info
Only in cache-1/opamroot/log: log-466-a7c577.out
Only in cache-1/opamroot/log: log-466-aa4a43.env
Only in cache-1/opamroot/log: log-466-aa4a43.info
Only in cache-1/opamroot/log: log-466-aa4a43.out
Only in cache-1/opamroot/log: log-466-ab22a1.env
Only in cache-1/opamroot/log: log-466-ab22a1.info
Only in cache-1/opamroot/log: log-466-ab22a1.out
Only in cache-1/opamroot/log: log-466-af57d6.env
Only in cache-1/opamroot/log: log-466-af57d6.info
Only in cache-1/opamroot/log: log-466-af57d6.out
Only in cache-1/opamroot/log: log-466-b1a82b.env
Only in cache-1/opamroot/log: log-466-b1a82b.info
Only in cache-1/opamroot/log: log-466-b1a82b.out
Only in cache-1/opamroot/log: log-466-b36dbe.env
Only in cache-1/opamroot/log: log-466-b36dbe.info
Only in cache-1/opamroot/log: log-466-b36dbe.out
Only in cache-1/opamroot/log: log-466-bb845d.env
Only in cache-1/opamroot/log: log-466-bb845d.info
Only in cache-1/opamroot/log: log-466-bb845d.out
Only in cache-1/opamroot/log: log-466-c61e7b.env
Only in cache-1/opamroot/log: log-466-c61e7b.info
Only in cache-1/opamroot/log: log-466-c61e7b.out
Only in cache-1/opamroot/log: log-466-c743ac.env
Only in cache-1/opamroot/log: log-466-c743ac.info
Only in cache-1/opamroot/log: log-466-c743ac.out
Only in cache-1/opamroot/log: log-466-d18c27.env
Only in cache-1/opamroot/log: log-466-d18c27.info
Only in cache-1/opamroot/log: log-466-d18c27.out
Only in cache-1/opamroot/log: log-466-d474c2.env
Only in cache-1/opamroot/log: log-466-d474c2.info
Only in cache-1/opamroot/log: log-466-d474c2.out
Only in cache-1/opamroot/log: log-466-d8a4ae.env
Only in cache-1/opamroot/log: log-466-d8a4ae.info
Only in cache-1/opamroot/log: log-466-d8a4ae.out
Only in cache-1/opamroot/log: log-466-dd682e.env
Only in cache-1/opamroot/log: log-466-dd682e.info
Only in cache-1/opamroot/log: log-466-dd682e.out
Only in cache-1/opamroot/log: log-466-de8fad.env
Only in cache-1/opamroot/log: log-466-de8fad.info
Only in cache-1/opamroot/log: log-466-de8fad.out
Only in cache-1/opamroot/log: log-466-de96f2.env
Only in cache-1/opamroot/log: log-466-de96f2.info
Only in cache-1/opamroot/log: log-466-de96f2.out
Only in cache-1/opamroot/log: log-466-e1320e.env
Only in cache-1/opamroot/log: log-466-e1320e.info
Only in cache-1/opamroot/log: log-466-e1320e.out
Only in cache-1/opamroot/log: log-466-e4db2f.env
Only in cache-1/opamroot/log: log-466-e4db2f.info
Only in cache-1/opamroot/log: log-466-e4db2f.out
Only in cache-1/opamroot/log: log-466-e4db2f.stdout
Only in cache-1/opamroot/log: log-466-ebe0e0.env
Only in cache-1/opamroot/log: log-466-ebe0e0.info
Only in cache-1/opamroot/log: log-466-ebe0e0.out
Only in cache-1/opamroot/log: log-466-ece7a3.env
Only in cache-1/opamroot/log: log-466-ece7a3.info
Only in cache-1/opamroot/log: log-466-ece7a3.out
Only in cache-1/opamroot/log: log-466-eec089.env
Only in cache-1/opamroot/log: log-466-eec089.info
Only in cache-1/opamroot/log: log-466-eec089.out
Only in cache-1/opamroot/log: log-466-f936b0.env
Only in cache-1/opamroot/log: log-466-f936b0.info
Only in cache-1/opamroot/log: log-466-f936b0.out
Only in cache-1/opamroot/log: log-466-ffb3fd.env
Only in cache-1/opamroot/log: log-466-ffb3fd.info
Only in cache-1/opamroot/log: log-466-ffb3fd.out
Only in cache-1/opamroot/log: ocamlfind-466-000417.env
Only in cache-1/opamroot/log: ocamlfind-466-000417.info
Only in cache-1/opamroot/log: ocamlfind-466-000417.out
Only in cache-1/opamroot/log: ocamlfind-466-2913ab.env
Only in cache-1/opamroot/log: ocamlfind-466-2913ab.info
Only in cache-1/opamroot/log: ocamlfind-466-2913ab.out
Only in cache-1/opamroot/log: ocamlfind-466-543ebd.env
Only in cache-1/opamroot/log: ocamlfind-466-543ebd.info
Only in cache-1/opamroot/log: ocamlfind-466-543ebd.out
Only in cache-1/opamroot/log: ocamlfind-466-8b9749.env
Only in cache-1/opamroot/log: ocamlfind-466-8b9749.info
Only in cache-1/opamroot/log: ocamlfind-466-8b9749.out
Only in cache-1/opamroot/log: ocamlfind-466-fbcac7.env
Only in cache-1/opamroot/log: ocamlfind-466-fbcac7.info
Only in cache-1/opamroot/log: ocamlfind-466-fbcac7.out
Files cache-1/opamroot/ocaml-system/bin/coqc and cache-2/opamroot/ocaml-system/bin/coqc differ
Files cache-1/opamroot/ocaml-system/bin/coqchk and cache-2/opamroot/ocaml-system/bin/coqchk differ
Files cache-1/opamroot/ocaml-system/bin/coqdep and cache-2/opamroot/ocaml-system/bin/coqdep differ
Files cache-1/opamroot/ocaml-system/bin/coqdoc and cache-2/opamroot/ocaml-system/bin/coqdoc differ
Files cache-1/opamroot/ocaml-system/bin/coq_makefile and cache-2/opamroot/ocaml-system/bin/coq_makefile differ
Files cache-1/opamroot/ocaml-system/bin/coqmktop and cache-2/opamroot/ocaml-system/bin/coqmktop differ
Files cache-1/opamroot/ocaml-system/bin/coqtop and cache-2/opamroot/ocaml-system/bin/coqtop differ
Files cache-1/opamroot/ocaml-system/bin/coqtop.byte and cache-2/opamroot/ocaml-system/bin/coqtop.byte differ
Files cache-1/opamroot/ocaml-system/bin/coqworkmgr and cache-2/opamroot/ocaml-system/bin/coqworkmgr differ
Files cache-1/opamroot/ocaml-system/lib/coq/lib/clib.a and cache-2/opamroot/ocaml-system/lib/coq/lib/clib.a differ
Files cache-1/opamroot/ocaml-system/lib/coq/lib/clib.cma and cache-2/opamroot/ocaml-system/lib/coq/lib/clib.cma differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqc and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqc differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqchk and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqchk differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqchk.byte and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqchk.byte differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqdep and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqdep differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqdoc and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqdoc differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coq_makefile and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coq_makefile differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqmktop and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqmktop differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqtop and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqtop differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqtop.byte and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqtop.byte differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqworkmgr and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/coqworkmgr differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/fake_ide and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/bin/fake_ide differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/checker/check.a and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/checker/check.a differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/checker/check.cma and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/checker/check.cma differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/config/coq_config.cmo and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/config/coq_config.cmo differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/config/coq_config.ml and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/config/coq_config.ml differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/config/coq_config.o and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/config/coq_config.o differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/lib/clib.a and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/lib/clib.a differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/lib/clib.cma and cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/lib/clib.cma differ
diff: cache-1/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/test-suite/bugs/closed/4722/tata: No such file or directory
diff: cache-2/opamroot/ocaml-system/.opam-switch/build/coq.8.7.2/test-suite/bugs/closed/4722/tata: No such file or directory
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/FETCH_HEAD and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/FETCH_HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/index and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/index differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/HEAD and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/refs/heads/master and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/refs/heads/master differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/refs/remotes/opam-ref and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/refs/remotes/opam-ref differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/refs/remotes/opam-ref-master and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/logs/refs/remotes/opam-ref-master differ
Only in cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/objects/pack: pack-6526a4e8fd3ffaa9f8d5fd296415267da02bb9b5.idx
Only in cache-1/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/objects/pack: pack-6526a4e8fd3ffaa9f8d5fd296415267da02bb9b5.pack
Only in cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/objects/pack: pack-d787cbdad2f26f7e3128a63fa6252c3d0e4254ad.idx
Only in cache-2/opamroot/ocaml-system/.opam-switch/build/coq-iris.dev/.git/objects/pack: pack-d787cbdad2f26f7e3128a63fa6252c3d0e4254ad.pack
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/FETCH_HEAD and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/FETCH_HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/index and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/index differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/HEAD and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/refs/heads/master and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/refs/heads/master differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/refs/remotes/opam-ref and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/refs/remotes/opam-ref differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/refs/remotes/opam-ref-master and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.git/logs/refs/remotes/opam-ref-master differ
Files cache-1/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.lia.cache and cache-2/opamroot/ocaml-system/.opam-switch/build/coq-stdpp.dev/.lia.cache differ
Files cache-1/opamroot/ocaml-system/.opam-switch/install/camlp5.changes and cache-2/opamroot/ocaml-system/.opam-switch/install/camlp5.changes differ
Files cache-1/opamroot/ocaml-system/.opam-switch/install/coq.changes and cache-2/opamroot/ocaml-system/.opam-switch/install/coq.changes differ
Files cache-1/opamroot/ocaml-system/.opam-switch/install/coq-iris.changes and cache-2/opamroot/ocaml-system/.opam-switch/install/coq-iris.changes differ
Files cache-1/opamroot/ocaml-system/.opam-switch/install/coq-stdpp.changes and cache-2/opamroot/ocaml-system/.opam-switch/install/coq-stdpp.changes differ
Files cache-1/opamroot/ocaml-system/.opam-switch/install/ocamlfind.changes and cache-2/opamroot/ocaml-system/.opam-switch/install/ocamlfind.changes differ
Files cache-1/opamroot/ocaml-system/.opam-switch/overlay/coq-iris/opam and cache-2/opamroot/ocaml-system/.opam-switch/overlay/coq-iris/opam differ
Files cache-1/opamroot/ocaml-system/.opam-switch/overlay/coq-stdpp/opam and cache-2/opamroot/ocaml-system/.opam-switch/overlay/coq-stdpp/opam differ
Files cache-1/opamroot/ocaml-system/.opam-switch/packages/coq-iris.dev/opam and cache-2/opamroot/ocaml-system/.opam-switch/packages/coq-iris.dev/opam differ
Files cache-1/opamroot/ocaml-system/.opam-switch/packages/coq-stdpp.dev/opam and cache-2/opamroot/ocaml-system/.opam-switch/packages/coq-stdpp.dev/opam differ
Files cache-1/opamroot/ocaml-system/.opam-switch/reinstall and cache-2/opamroot/ocaml-system/.opam-switch/reinstall differ
Only in cache-2/opamroot/ocaml-system/.opam-switch: remove
diff: cache-1/opamroot/ocaml-system/.opam-switch/sources/coq/test-suite/bugs/closed/4722/tata: No such file or directory
diff: cache-2/opamroot/ocaml-system/.opam-switch/sources/coq/test-suite/bugs/closed/4722/tata: No such file or directory
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/FETCH_HEAD and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/FETCH_HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/index and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/index differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/HEAD and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/refs/heads/master and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/refs/heads/master differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/refs/remotes/opam-ref and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/refs/remotes/opam-ref differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/refs/remotes/opam-ref-master and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/logs/refs/remotes/opam-ref-master differ
Only in cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/objects/pack: pack-6526a4e8fd3ffaa9f8d5fd296415267da02bb9b5.idx
Only in cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/objects/pack: pack-6526a4e8fd3ffaa9f8d5fd296415267da02bb9b5.pack
Only in cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/objects/pack: pack-d787cbdad2f26f7e3128a63fa6252c3d0e4254ad.idx
Only in cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-iris/.git/objects/pack: pack-d787cbdad2f26f7e3128a63fa6252c3d0e4254ad.pack
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/FETCH_HEAD and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/FETCH_HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/index and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/index differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/HEAD and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/HEAD differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/refs/heads/master and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/refs/heads/master differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/refs/remotes/opam-ref and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/refs/remotes/opam-ref differ
Files cache-1/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/refs/remotes/opam-ref-master and cache-2/opamroot/ocaml-system/.opam-switch/sources/coq-stdpp/.git/logs/refs/remotes/opam-ref-master differ
Files cache-1/opamroot/repo/iris-dev/.git/index and cache-2/opamroot/repo/iris-dev/.git/index differ
Files cache-1/opamroot/repo/iris-dev/.git/logs/HEAD and cache-2/opamroot/repo/iris-dev/.git/logs/HEAD differ
Files cache-1/opamroot/repo/iris-dev/.git/logs/refs/heads/master and cache-2/opamroot/repo/iris-dev/.git/logs/refs/heads/master differ
Files cache-1/opamroot/repo/iris-dev/.git/logs/refs/remotes/opam-ref and cache-2/opamroot/repo/iris-dev/.git/logs/refs/remotes/opam-ref differ

This excludes some files; it was generated using diff -ur cache-1 cache-2 -x "*.aux" -q -x "*.cmx*".

rjbou commented 6 years ago

Are CIs of these caches launching the same job (same commit to test) or different ones? Is the folder cache-2/opamroot/ocaml-system/.opam-switch/remove empty?

RalfJung commented 6 years ago

Are CIs of these caches launch the same job (same commit to test) or different ones?

The only difference is in how much debugging they show during CI execution. I do not think that is significant. I can't run the same job of the same commit twice at the same time, so there has to be some distance for me to be able to force it to use a certain CI runner.

Is the folder cache-2/opamroot/ocaml-system/.opam-switch/remove empty?

Yes.

RalfJung commented 6 years ago

Note that I do not know which of the two is the "good" cache. :/ I extracted them from the CI runner, but they are stored by some ID that I cannot correlate with anything else...

rjbou commented 6 years ago

If they test same packages, etc, they shouldn't be differences in these files:

Maybe the last oe ca be more problematic.

Have you tried to delete the cache of failing CI and relaunch?

RalfJung commented 6 years ago

I have so far fixed the issue by deleting caches, yes. It always came back eventually, I cannot say how or why or when. I can of course do that again... but then I also lose the one case I have right now where I can reproduce this. (Well, maybe can. Now that I fixed it, it is possible the cache is now in a good state and the issue will not come back.)

This job uses the latest git versions of coq, coq-iris and coq-stdpp. So it is expected that not all caches have the same version. Thinking about it, the difference might just be in the fact that one cache is already up-to-date and so the opam upgrade does not have to do anything, whereas the other cache is outdated and so some packages need rebuilding.

The install differences mostly look like timestamps: https://gist.github.com/RalfJung/d2db29d277da7454a5a6e3df56c4bd26

overlay diff:

$ diff -ur ./cache-1/opamroot/ocaml-system/.opam-switch/overlay/ ./cache-2/opamroot/ocaml-system/.opam-switch/overlay/
diff --color -ur ./cache-1/opamroot/ocaml-system/.opam-switch/overlay/coq-iris/opam ./cache-2/opamroot/ocaml-system/.opam-switch/overlay/coq-iris/opam
--- ./cache-1/opamroot/ocaml-system/.opam-switch/overlay/coq-iris/opam  2018-10-11 14:23:34.000000000 +0200
+++ ./cache-2/opamroot/ocaml-system/.opam-switch/overlay/coq-iris/opam  2018-10-11 10:16:31.000000000 +0200
@@ -18,5 +18,5 @@
 synopsis: "This is the Coq development of the Iris Project."
 flags: light-uninstall
 url {
-  src: "git+https://gitlab.mpi-sws.org/FP/iris-coq.git#master"
+  src: "git+https://gitlab.mpi-sws.org/FP/iris-coq.git"
 }
diff --color -ur ./cache-1/opamroot/ocaml-system/.opam-switch/overlay/coq-stdpp/opam ./cache-2/opamroot/ocaml-system/.opam-switch/overlay/coq-stdpp/opam
--- ./cache-1/opamroot/ocaml-system/.opam-switch/overlay/coq-stdpp/opam 2018-10-11 14:23:33.000000000 +0200
+++ ./cache-2/opamroot/ocaml-system/.opam-switch/overlay/coq-stdpp/opam 2018-10-11 10:16:30.000000000 +0200
@@ -37,5 +37,5 @@
 - It is entirely dependency- and axiom-free."""
 flags: light-uninstall
 url {
-  src: "git+https://gitlab.mpi-sws.org/robbertkrebbers/coq-stdpp/#master"
+  src: "git+https://gitlab.mpi-sws.org/robbertkrebbers/coq-stdpp/"
 }

packages diff:

$ diff -ur ./cache-1/opamroot/ocaml-system/.opam-switch/packages/ ./cache-2/opamroot/ocaml-system/.opam-switch/packages/
diff --color -ur ./cache-1/opamroot/ocaml-system/.opam-switch/packages/coq-iris.dev/opam ./cache-2/opamroot/ocaml-system/.opam-switch/packages/coq-iris.dev/opam
--- ./cache-1/opamroot/ocaml-system/.opam-switch/packages/coq-iris.dev/opam 2018-10-11 14:27:16.000000000 +0200
+++ ./cache-2/opamroot/ocaml-system/.opam-switch/packages/coq-iris.dev/opam 2018-10-10 18:47:18.000000000 +0200
@@ -16,5 +16,5 @@
 remove: ["rm" "-rf" "%{lib}%/coq/user-contrib/iris"]
 dev-repo: "git+https://gitlab.mpi-sws.org/FP/iris-coq.git"
 url {
-  src: "git+https://gitlab.mpi-sws.org/FP/iris-coq.git#master"
+  src: "git+https://gitlab.mpi-sws.org/FP/iris-coq.git"
 }
diff --color -ur ./cache-1/opamroot/ocaml-system/.opam-switch/packages/coq-stdpp.dev/opam ./cache-2/opamroot/ocaml-system/.opam-switch/packages/coq-stdpp.dev/opam
--- ./cache-1/opamroot/ocaml-system/.opam-switch/packages/coq-stdpp.dev/opam    2018-10-11 14:24:29.000000000 +0200
+++ ./cache-2/opamroot/ocaml-system/.opam-switch/packages/coq-stdpp.dev/opam    2018-10-10 18:44:35.000000000 +0200
@@ -35,5 +35,5 @@
 remove: ["rm" "-rf" "%{lib}%/coq/user-contrib/stdpp"]
 dev-repo: "git+https://gitlab.mpi-sws.org/robbertkrebbers/coq-stdpp.git"
 url {
-  src: "git+https://gitlab.mpi-sws.org/robbertkrebbers/coq-stdpp/#master"
+  src: "git+https://gitlab.mpi-sws.org/robbertkrebbers/coq-stdpp/"
 }

reinstall diff

$ diff -ur ./cache-1/opamroot/ocaml-system/.opam-switch/reinstall ./cache-2/opamroot/ocaml-system/.opam-switch/reinstall 
--- ./cache-1/opamroot/ocaml-system/.opam-switch/reinstall  2018-10-11 14:24:28.000000000 +0200
+++ ./cache-2/opamroot/ocaml-system/.opam-switch/reinstall  2018-10-10 20:58:22.000000000 +0200
@@ -0,0 +1 @@
+coq-lambda-rust-builddep   dev
rjbou commented 6 years ago

Caches

yes, seems all of them are just timestamps.

Re reading the issue, sees some questions were left unanswered.

Again it seems rather odd that it reports "4.02.3" seven times, does it not?

Yes, it's not normal. it should be the output of ocaml -vnum. In another comment, there is also a double version number as this command output. You can check with a -vv in the config list if it is the command that is run several times or is it is ocaml that displays it several times. But I don't really know what is the impact.

We don't want an "unlocked base" but maybe we got it accidentally?

From the script, no. You can check that with the command opam list --base. Usually it originates from an empty switch creation, or an install unlocking base, that is not the case here.

Finally, reproduces it! But it is really weird that you ended in this state... In fact, when you do the upgrade, the package ocaml-system is marked as orphan 

00:00.624  CLIENT                          Orphans: (changes: , transitive: true) -> full {}, versions { ocaml-system.4.02.3 }

As explained here, it is considered no more available on the repository. I removed this package from my local repo and updated, got the same error. Now still need to find why this package is considered as removed by this opam root. You can check with the repository field opam show ocaml-system -f repository. or with find $OPAMROOT -name ocaml-system.4.02.3.

It is present in the default opam repository, and don't seems present on yours, so there is no reason to be considered as removed.

RalfJung commented 6 years ago

Well that didn't last long. The failures are back, even in the re-try call, without me changing anything.

I added some more debugging calls based on your recommendations, so if this happens again we should know more.

RalfJung commented 6 years ago

Here you go:

++ opam config list -vv
<><> Global opam variables ><><><><><><><><><><><><><><><><><><><><><><><><><><>
+ /usr/bin/ocamlc "-vnum"
- 4.02.3
- 4.02.3
arch              x86_64
          # Inferred from system
jobs              39
          # The number of parallel jobs set up in opam configuration
make              make
          # The 'make' command to use
opam-version      2.0.0
          # The currently running opam version
os                linux
          # Inferred from system
os-distribution   debian
          # Inferred from system
os-family         debian
          # Inferred from system
os-version        9
          # Inferred from system
root              /builds/FP/iris-atomic/opamroot
          # The current opam root directory
switch            ocaml-system
          # The identifier of the current switch
sys-ocaml-version 4.02.3
                  4.02.3
          # OCaml version present on your system independently of opam, if any

<><> Configuration variables from the current switch ><><><><><><><><><><><><><>
prefix   /builds/FP/iris-atomic/opamroot/ocaml-system
lib      /builds/FP/iris-atomic/opamroot/ocaml-system/lib
bin      /builds/FP/iris-atomic/opamroot/ocaml-system/bin
sbin     /builds/FP/iris-atomic/opamroot/ocaml-system/sbin
share    /builds/FP/iris-atomic/opamroot/ocaml-system/share
doc      /builds/FP/iris-atomic/opamroot/ocaml-system/doc
etc      /builds/FP/iris-atomic/opamroot/ocaml-system/etc
man      /builds/FP/iris-atomic/opamroot/ocaml-system/man
toplevel /builds/FP/iris-atomic/opamroot/ocaml-system/lib/toplevel
stublibs /builds/FP/iris-atomic/opamroot/ocaml-system/lib/stublibs
user     ci
group    ci

<><> Package variables ('opam config list PKG' to show) <><><><><><><><><><><><>
PKG:name       # Name of the package
PKG:version    # Version of the package
PKG:depends    # Resolved direct dependencies of the package
PKG:installed  # Whether the package is installed
PKG:enable     # Takes the value "enable" or "disable" depending on whether the package is installed
PKG:pinned     # Whether the package is pinned
PKG:bin        # Binary directory for this package
PKG:sbin       # System binary directory for this package
PKG:lib        # Library directory for this package
PKG:man        # Man directory for this package
PKG:doc        # Doc directory for this package
PKG:share      # Share directory for this package
PKG:etc        # Etc directory for this package
PKG:build      # Directory where the package was built
PKG:hash       # Hash of the package archive
PKG:dev        # True if this is a development package
PKG:build-id   # A hash identifying the precise package version with all its dependencies

Looks like it got two lines in a single call. However, I am also doing echo -n "OCaml version " && ocaml -vnum earlier in that job, and that just prints

OCaml version 4.02.3

Next:

++ opam show ocaml-system -f repository
default

And finally:

++ opam upgrade -yvv -a coq-iris-atomic-builddep --debug
00:00.004  GSTATE                          LOAD-GLOBAL-STATE @ /builds/FP/iris-atomic/opamroot
00:00.004  RSTATE                          LOAD-REPOSITORY-STATE @ /builds/FP/iris-atomic/opamroot
00:00.120  RSTATE                          Loaded /builds/FP/iris-atomic/opamroot/repo/state.cache in 0.115s
00:00.164  RSTATE                          Cache found
00:00.164  STATE                           LOAD-SWITCH-STATE @ ocaml-system
00:00.214  STATE                           Detected changed packages (marked for reinstall): {}
00:00.218  STATE                           Switch state loaded in 0.054s
00:00.218  FILE(switch-state)              Wrote /builds/FP/iris-atomic/opamroot/ocaml-system/.opam-switch/backup/state-20181015072928.export in 0.000s
+ /usr/bin/ocamlc "-vnum"
- 4.02.3
- 4.02.3
00:00.262  SYSTEM                          [log-277-5a61c5] (in 0.024s) ocamlc -vnum

<><> Synchronising pinned packages ><><><><><><><><><><><><><><><><><><><><><><>
00:00.360  UPDATE                          update-dev-packages
00:00.360  PARALLEL                        Iterate over 1 task(s) with 3 process(es)
00:00.360  PARALLEL                        Starting job 0 (worker 1/3): 0
00:00.360  UPDATE                          update-dev-package coq-iris-atomic-builddep.dev
00:00.360  UPDATE                          update-pinned-package coq-iris-atomic-builddep
00:00.363  UPDATE                          updating file:///builds/FP/iris-atomic/build-dep
00:00.363  RSYNC                           rsync: src=/builds/FP/iris-atomic/build-dep/ dst=/builds/FP/iris-atomic/opamroot/ocaml-system/.opam-switch/sources/coq-iris-atomic-builddep
00:00.397  PARALLEL                        Next task in job 0: /usr/bin/rsync -rLptgoDrvc --exclude .git --exclude _darcs --exclude .hg --exclude .#* --exclude _opam* --delete --delete-excluded /builds/FP/iris-atomic/build-dep/ /builds/FP/iris-atomic/opamroot/ocaml-system/.opam-switch/sources/coq-iris-atomic-builddep
Processing  1/1: [coq-iris-atomic-builddep.dev: rsync]
+ /usr/bin/rsync "-rLptgoDrvc" "--exclude" ".git" "--exclude" "_darcs" "--exclude" ".hg" "--exclude" ".#*" "--exclude" "_opam*" "--delete" "--delete-excluded" "/builds/FP/iris-atomic/build-dep/" "/builds/FP/iris-atomic/opamroot/ocaml-system/.opam-switch/sources/coq-iris-atomic-builddep"
- sending incremental file list
- 
- sent 83 bytes  received 12 bytes  190.00 bytes/sec
- total size is 381  speedup is 4.01
- sending incremental file list
- 
- sent 83 bytes  received 12 bytes  190.00 bytes/sec
- total size is 381  speedup is 4.01
[coq-iris-atomic-builddep.dev] synchronised from file:///builds/FP/iris-atomic/build-dep
00:00.583  PARALLEL                        Collected task for job 0 (ret:0)
00:00.584  PARALLEL                        Job 0 finished
00:00.584  SWACT                           add-to-reinstall unpinned_only:false packages:{ coq-iris-atomic-builddep.dev }

00:00.585  FILE(package-version-list)      Wrote /builds/FP/iris-atomic/opamroot/ocaml-system/.opam-switch/reinstall in 0.000s
00:00.585  CLIENT                          UPGRADE coq-iris-atomic-builddep
00:00.635  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.635  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.701  STATE                           ERR: undefined filter variable in dependencies of hts_shrink.1.0.0: test
00:00.703  STATE                           ERR: undefined filter variable in dependencies of influxdb-async.0.1.0: test
00:00.703  STATE                           ERR: undefined filter variable in dependencies of influxdb-lwt.0.1.0: test
00:00.889  CLIENT                          Orphans: (changes: , transitive: true) -> full {}, versions { ocaml-system.4.02.3 }
00:00.897  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.897  STATE                           ERR: undefined filter variable in dependencies of batteries.2.9.0: test
00:00.937  STATE                           ERR: undefined filter variable in dependencies of hts_shrink.1.0.0: test
00:00.939  STATE                           ERR: undefined filter variable in dependencies of influxdb-async.0.1.0: test
00:00.939  STATE                           ERR: undefined filter variable in dependencies of influxdb-lwt.0.1.0: test
00:01.076  SOLVER                          resolve request=install:() remove:() upgrade:(ocamlfind & ocaml-system & ocaml & num & coq-stdpp & coq-iris-atomic-builddep & coq-iris & coq & conf-m4 & camlp5 & base-unix & base-threads & base-num & base-bigarray)
00:01.670  SOLVER                          Load cudf universe (depopts:false, build:true, post:true)
00:01.706  FILTER                          ERR: value_bool: "/builds/FP/iris-atomic/opamroot/ocaml-system/doc"
00:01.706  FILTER                          ERR: value_bool: "/builds/FP/iris-atomic/opamroot/ocaml-system/doc"
00:01.706  FILTER                          ERR: value_bool: "/builds/FP/iris-atomic/opamroot/ocaml-system/doc"
00:01.706  FILTER                          ERR: value_bool: "/builds/FP/iris-atomic/opamroot/ocaml-system/doc"
00:01.711  FILTER                          ERR: value_bool: "/builds/FP/iris-atomic/opamroot/ocaml-system/doc"
00:01.738  CUDF                            resolve request=install:() remove:() upgrade:(base-bigarray (>= 1) & base-num (>= 1) & base-threads (>= 1) & base-unix (>= 1) & camlp5 (>= 18) & conf-m4 (>= 1) & coq (>= 36) & coq-iris (>= 34) & coq-iris-atomic-builddep (>= 1) & coq-stdpp (>= 2) & num (>= 1) & ocaml (>= 39) & ocaml-system & ocamlfind (>= 25))
00:01.738  SOLVER                          Calling solver builtin-mccs+glpk with criteria -removed,-count[version-lag,solution],-new
00:02.253  CUDF                            Solver call done in 0.515
00:02.253  CLIENT                          conflict!
The following dependencies couldn't be met:
  - ocaml -> ocaml-base-compiler = 4.02.3
      conflict with the base packages of this switch
  - ocaml -> ocaml-system = 4.02.3
      unmet availability conditions: sys-ocaml-version = "4.02.3"
  - ocaml -> ocaml-variants >= 4.02.3
      conflict with the base packages of this switch

00:02.305  SYSTEM                          rm /builds/FP/iris-atomic/opamroot/ocaml-system/.opam-switch/backup/state-20181015072928.export
'opam upgrade -yvv -a coq-iris-atomic-builddep --debug' failed.
rjbou commented 6 years ago

Can you apply this patch and share the link of the next failing CI job (no need to quote those verbose output :) ).

Thinking about it, the difference might just be in the fact that one cache is already up-to-date and so the opam upgrade does not have to do anything, whereas the other cache is outdated and so some packages need rebuilding

Not really as the solver is called and conclude that everything is up to date. You can see it in this log: first it fails, but when relaunching exactly the same command (expect verbose), it calls the solver (but not having ocaml-system as an orphan), it succeed.

Another thing surprises me in this log: on update and at first upgrade, no changes are detected on the pinned package coq-iris-atomic-builddep.dev, but when relaunching upgrade, it detects a change. The thing is that there is no modification on the package directory, no related command is launched and opam doesn't modify source. I don't think it is related directly to the issue, but another CI mystery...

RalfJung commented 6 years ago

Can you apply this patch and share the link of the next failing CI job (no need to quote those verbose output :) ).

Patch applied, I expect some more failures this night.

You can see it in this log: first it fails, but when relaunching exactly the same command (expect verbose), it calls the solver (but not having ocaml-system as an orphan), it succeed.

Good find. There goes the only pattern I thought I had seen.^^

Another thing surprises me in this log: on update and at first upgrade, no changes are detected on the pinned package coq-iris-atomic-builddep.dev, but when relaunching upgrade, it detects a change. The thing is that there is no modification on the package directory, no related command is launched and opam doesn't modify source. I don't think it is related directly to the issue, but another CI mystery...

:man_shrugging:

RalfJung commented 6 years ago

Only one failure this night, but it got the logs you asked for: https://gitlab.mpi-sws.org/FP/iris-atomic/-/jobs/18987

rjbou commented 6 years ago

New patch.

Another thing common to successful jobs is that ocaml version has the good format (as you noticed in a previous comment). All failed jobs have more that one line of version. In the test you added, you checked ocaml -vnum while opam launches /usr/bin/ocamlc -vnum.

As this command result is the content of sys-ocaml-version variable, package ocaml-system becomes unavailable in this context because of the condition available: sys-ocaml-version = "4.02.3".

RalfJung commented 6 years ago

Patch applied. (I left some of that extra debugging in, though.)

All failed jobs have more that one line of version. In the test you added, you checked ocaml -vnum while opam launches /usr/bin/ocamlc -vnum.

Ah, good catch! I still don't get why the output should depend on the switch though... and why sometimes, re-running the opam upgrade fixes it. Very mysterious.

In any case, thanks a lot for your patience with this problem. :)

RalfJung commented 6 years ago

No CI failure this night, but https://gitlab.mpi-sws.org/FP/iris-atomic/-/jobs/19005 did go into the "update failed the first time" branch. It didn't fail again on the second try, though.

rjbou commented 6 years ago

I still don't get why the output should depend on the switch though...

The output don't just depends on the switch or cache, e.g. in this log, ocamlc -vnum have different output (three versions once, and two versions twice).

go into the "update failed the first time" branch. It didn't fail again on the second try, though.

maybe you should keep the -vv on the two upgrade, and other commands. --debug is no more useful at this stage. Verbose mode (that prints called commands and their output) will help to confirm that each failing jobs have several version on failed jobs.

But yes, it is a mystery that the output has several lines... opam just launches the command and retrieve the output. It is possible to add more debugging check: running the exact command, running the command using the end function Unix.create_process_env, running the command using opam library OpamSystem.read_command_output, run on CI a modified opam to get more information, etc.

RalfJung commented 6 years ago

It was calm for some nights, but now it is happening again: https://gitlab.mpi-sws.org/FP/iris-atomic/-/jobs/19112 https://gitlab.mpi-sws.org/amintimany/iris-ora/-/jobs/19113

RalfJung commented 6 years ago

We are now seeing this in regular CI runs, not just nightly automatic ones: https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19139 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19140 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19153 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19154

rjbou commented 6 years ago

All have in common the multiple line ocaml version. As mentioned here, we can add more targeted checks, or even pin a debug version of opam to be able to retrieve the command output.

RalfJung commented 6 years ago

How would we go about this?

The opam in there is just the binary 2.0.0 release you are providing.

RalfJung commented 6 years ago

Unsurprisingly, opam 2.0.1 does not help. :( See https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19309.

Also, I added some shell tests that queries ocamlc -vnum 100 times and tests that it always just produces one line of output. It does. I think the error is on the opam side which somehow fails to capture the output properly. Maybe something about file descriptor buffering?

RalfJung commented 6 years ago

@rjbou was so kind to provide a patch for opam that would help debugging, and I have now rolled that out for our CI. Here's are some logs with this failure that happened with the patch included:

https://gitlab.mpi-sws.org/robbertkrebbers/coq-stdpp/-/jobs/19519 https://gitlab.mpi-sws.org/robbertkrebbers/coq-stdpp/-/jobs/19521

RalfJung commented 6 years ago

@rjbou we got some straces:

https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19642 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19639 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19638

RalfJung commented 6 years ago

Some more:

https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19691 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19692 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19694 https://gitlab.mpi-sws.org/FP/iris-coq/-/jobs/19695

rjbou commented 5 years ago

What is the status of this issue? The problem seems fixed (from the new ci logs). Did you find why ocamlc output was duplicated?

RalfJung commented 5 years ago

The problem seems fixed (from the new ci logs)

We got frustrated and stopped using system OCaml. This costs extra CI time for building OCaml, but that's better than builds failing all the time. The problem never occured with OCaml compiled by opam. We never figured out what was going on, and the bug is probably still there.

rjbou commented 5 years ago

Ok, i'll keep issue open in the case it happens to someone else. To recap Upgrade fails with unmet availability conditions because of ocamlc -vnum output is on more than one line (randomly). stracing the command shows that version is written only once.

dra27 commented 3 years ago

Closing this one, as it doesn't look like we're a position to investigate it further. If this does come up again, please feel free either to reference this issue from a new one or re-open this one.

The log doubling (seeing the output of ocamlc -vnum twice) has been seen elsewhere and is probably benign (although probably a display bug somewhere in opam).