opam rebuilds dependency for "upstream changes", but there are no changes

[RalfJung]

Steps to reproduce

Run the following script twice:

#!/bin/bash
set -e
# This script installes the build dependencies for CI builds

THIS_PACKAGE="coq-iris"

# Prepare OPAM configuration
export OPAMROOT="$(pwd)/opamroot"
export OPAM_EDITOR="$(which false)"

# Make sure we got a good OPAM
test -d "$OPAMROOT" || (mkdir "$OPAMROOT" && opam init -n)
test -d "$OPAMROOT/repo/coq-released" || opam repo add coq-released https://coq.inria.fr/opam/released -p 10
opam update
opam install ocamlfind -y

# Pick fixed versions of some dependencies
echo
opam pin add coq 8.5.3 -y -n

# Install/upgrade build-dependencies
echo
opam pin add "$THIS_PACKAGE" "https://gitlab.mpi-sws.org/FP/iris-coq#5c3ad2f9bbc8471d27bc4e4fe72b4fb1f3f15391" -n -y -k git
opam upgrade -y
opam install "$THIS_PACKAGE" --deps-only -y

Expected behavior

On the second run, nothing is compiled because everything is already in the correct version

Actual behavior

coq (and its reverse dependencies) are recompiled on the second run:

The following actions will be performed:
  - recompile coq                    8.5.3*             [upstream changes]
  - recompile coq-mathcomp-ssreflect 1.6                [uses coq]
===== 2 to recompile =====

On the third and following runs, no more recompilation happens.

Context

I am using this script to set up the build dependencies of a project for a CI build. The opamroot folder is cached between CI runs so that build dependencies are not unnecessarily re-compiled. However, due to this bug, when I clear the CI cache, it takes two CI runs until the following runs are fast and properly re-use build dependencies.

opam config report

# OPAM config report
# opam-version    1.2.2 
# self-upgrade    no
# os              linux
# external-solver aspcud $in $out $criteria
# criteria        -count(removed),-notuptodate(request),-sum(request,version-lag),-count(down),-notuptodate(changed),-count(changed),-notuptodate(solution),-sum(solution,version-lag)
# jobs            4
# repositories    4* (http), 1 (version-controlled)
# pinned          1 (version), 2 (version control)
# current-switch  coq-8.5*
# last-update     2016-11-25 15:57

[AltGr]

It's indeed a bug, which I could reproduce — thanks for the detailed report. The way opam 1.2.2 uses to detect package changes was not very reliable, and here it seems it gets confused by the pinning; this has already been fixed in the development branch for 2.0, where we keep track of the actual opam files used for installation, and compare them with their upstream to detect changes — I can confirm that the bug is no longer there.

As a workaround, you could remove the $(opam config var prefix)/reinstall file after update to make opam forget about pending reinstallations.

By the way, caching a preliminary opam switch is a good approach, that I use for opam itself

[RalfJung]

Great to here the bug got fixed!

As a workaround, you could remove the $(opam config var prefix)/reinstall file after update to make opam forget about pending reinstallations.

I assume that would mean that if there is an actual upstream update, it would be missed?

[AltGr]

I assume that would mean that if there is an actual upstream update, it would be missed?

Unfortunately, yes.