failure to build build-tool-depends with allow-newer

Describe the bug

A build-tool-depends fails to compile when built with cabal build, seemingly ignoring the freeze file because of an allow-newer.

$ cabal --version
cabal-install version 3.2.0.0
compiled using version 3.2.0.0 of the Cabal library

This bug was originally reported in the context of the tool that fails to build, but this looks more like a cabal bug: https://github.com/tweag/ormolu/issues/534

To Reproduce

Rather than creating a fresh repo, we can reproduce this on any project that is compatible with a recent LTS. Taking one of my own very simple free software projects as the demonstration:

git clone https://gitlab.com/fommil/ffunctor.git
cd ffunctor
curl https://www.stackage.org/lts-15.5/cabal.config > cabal.project.freeze

then add build-tool-depends: ormolu:ormolu ==0.0.3.1 to the common stanza of ffunctor.cabal (line 35)

cabal configure -w ghc-8.8.3 --allow-newer
cabal build :pkg:ormolu:exe:ormolu

Giving

cabal build :pkg:ormolu:exe:ormolu
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - ormolu-0.0.3.1 (lib) (requires build)
 - ormolu-0.0.3.1 (exe:ormolu) (requires build)
Starting     ormolu-0.0.3.1 (lib)
Building     ormolu-0.0.3.1 (lib)
...
[ 1 of 41] Compiling GHC              ( src/GHC.hs, dist/build/GHC.o )

src/GHC.hs:9:1: error:
    Could not load module ‘HsBinds’
    It is a member of the hidden package ‘ghc-lib-parser-8.8.3.20200224’.
    Perhaps you need to add ‘ghc-lib-parser’ to the build-depends in your .cabal file.
    It is a member of the hidden package ‘ghc-lib-parser-8.8.3.20200224’.
    Perhaps you need to add ‘ghc-lib-parser’ to the build-depends in your .cabal file.
    It is a member of the hidden package ‘ghc-8.8.3’.
    Perhaps you need to add ‘ghc’ to the build-depends in your .cabal file.
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
9 | import HsBinds as X
  | ^^^^^^^^^^^^^^^^^^^

The problem seems to be that the freeze file's version of ghc-lib-parser ==8.8.3.20200224 is ignored when allow-newer is used, with the constraint solver picking ghc-lib-parser ==8.10.1.20200324 (among other differences).

Expected behavior

I expected the freeze file to be observed and for the allow-newer to only be used when necessary. The solver seems to do the opposite and ignore the freeze file, picking what it thinks will work.

Note that performing a cabal freeze will actually change the values of some of the already frozen libraries, which seems like a definite bug.

constraints: abstract-deque ==0.3

(as in cabal.config stackage produces) don't constraint all dependencies, especially not setup nor build-tool-depends.

With %s/^ */&any./ regex magic and manual editing one gets something like

constraints: any.abstract-deque ==0.3,
             any.abstract-par ==0.3.3,
             any.AC-Angle ==1.0,
             any.accuerr ==0.2.0.2,
             any.ace ==0.6,
             any.action-permutations ==0.0.0.1,
             any.active ==0.2.0.14,
             any.ad ==4.4,
             any.adjunctions ==4.4,
             any.adler32 ==0.1.2.0,
...

with that change, ormolu builds fine. The constraints and allow-newer work together: allow-newer relaxes build-depends constraints, but constraints adds new (other) ones. I don't completely understand what's the idea of using --allow-newer and Stackage all-freezing-snapshot, but it does "work".

haskell / cabal

failure to build build-tool-depends with allow-newer #6687