Cabal-3.12.0.0 passes include dirs to GHC in a new order

[andreasabel]

A full reproducer with analysis is at https://github.com/andreasabel/bug-cabal-3.12-setup .

I noticed this issue when using ghc-9.10.1 for the first time to build Agda using cabal v1-install. The Agda parser was malfunctioning. Turned out I had a stale Lexer.hs next to my Lexer.x, and this would be used when building with ghc-9.10 but not when building with ghc-9.8. After some hours I traced the problem down to the version of the Cabal library shipped with these GHCs.

In short

Cabal-3.12 passes the include directories to GHC in a different order than Cabal-3.10. Cabal-3.10 puts dist/build before the hs-source-dirs whereas Cabal-3.12 does it the other way round. This leads GHC to pick up a different .hs file:

• In case of Cabal-3.10 the one generated by the build-tool.
• In case of Cabal-3.12 the one sitting in the source tree.

The problem is reproduced in a GitHub workflow run here: https://github.com/andreasabel/bug-cabal-3.12-setup/actions/runs/9291884301 . Passes with GHC 9.8, fails with GHC 9.10, independent of the version of cabal-install.

Full description

It follows a longer description copied from https://github.com/andreasabel/bug-cabal-3.12-setup/blob/61cdad9444a3b13e2b833e1dde36019c4a31f782/README.md

Cabal-3.12 passes include dirs in wrong order to GHC

Conditions:

• GHC 9.10
• cabal v1-install
• custom setup (can be default Setup.hs), so that Cabal-3.12 is used
• building a library (not just an executable)
• using a build-tool: we use happy here to shadow a .hs file by a .y file
• hs-source-dirs is not . but e.g. src

Files:

• fred.cabal with
  • build-type: Custom
  • library with hs-source-dirs: src (in particular not .)
• Setup.hs (standard)
• src/Fred.y: processed by happy
• src/Fred.hs: stale code that should be shadowed by Fred.y always

In this setting cabal v1-install malfunctions if ghc is ghc-9.10.1. (Works with older GHCs.) It picks up src/Fred.hs instead of dist/build/Fred.hs that is created by happy from src/Fred.y.

Looking at the verbose output cabal v1-install -v3, we notice a difference in the call to ghc when the GHC is 9.10.1.

• GHC 9.10 / Cabal-3.12

  /usr/local/bin/ghc --make -fbuilding-cabal-package \
    ...
    -isrc -idist/build \
    ... \
    Fred
  
  [1 of 2] Compiling Main ( Fred.hs, dist/build/Main.o, ...

• GHC 9.8 / Cabal-3.10

  /usr/local/bin/ghc --make -fbuilding-cabal-package \
    ... \
    -idist/build -isrc \
    ... \
    Fred
  
  [1 of 1] Compiling Parser ( dist/build/Parser.hs, ... )

When building with GHC 9.8, library Cabal-3.10 is used which places the path dist/build with the generated files correctly before the path . of the source files; but with GHC 9.10, library Cabal-3.12 is used which does it the other way.

Full calls:

• GHC 9.10 / Cabal-3.12

  /usr/local/bin/ghc --make -fbuilding-cabal-package -O -static -dynamic-too -dynosuf dyn_o -dynhisuf dyn_hi \
    -outputdir dist/build -odir dist/build -hidir dist/build \
    -hiedir dist/build/extra-compilation-artifacts/hie \
    -stubdir dist/build -i \
    -i. -idist/build \
    -idist/build/autogen -idist/build/global-autogen \
    -Idist/build/autogen -Idist/build/global-autogen -Idist/build \
    -I/usr/local/opt/icu4c/include -I/usr/local/opt/libxml2/include/libxml2 \
    -optP-include -optPdist/build/autogen/cabal_macros.h \
    -this-unit-id stale-0-Ly26ql3fYpuLnJ9xynYGbf \
    -hide-all-packages -Wmissing-home-modules -package-db dist/package.conf.inplace \
    -package-id array-0.5.7.0-2002 -package-id base-4.20.0.0-8a80 \
    -XHaskell2010 \
    Fred
  
  [1 of 2] Compiling Main ( Fred.hs, dist/build/Main.o, ...

• GHC 9.8 / Cabal-3.10

  /usr/local/bin/ghc --make -fbuilding-cabal-package -O -static -dynamic-too -dynosuf dyn_o -dynhisuf dyn_hi \
    -outputdir dist/build -odir dist/build -hidir dist/build \
    -stubdir dist/build -i \
    -idist/build -i. \
    -idist/build/autogen -idist/build/global-autogen \
    -Idist/build/autogen -Idist/build/global-autogen -Idist/build \
    -I/usr/local/opt/icu4c/include -I/usr/local/opt/libxml2/include/libxml2 \
    -optP-include -optPdist/build/autogen/cabal_macros.h \
    -this-unit-id stale-0-F795SeGwWezHPbPW8onWYj \
    -hide-all-packages -Wmissing-home-modules -package-db dist/package.conf.inplace \
    -package-id array-0.5.6.0-28ee -package-id base-4.19.1.0-654f \
    -XHaskell2010 \
    Fred
  
  [1 of 1] Compiling Parser ( dist/build/Parser.hs, ... )

[ulysses4ever]

Make sure Haskell files in explicit source directories take precedence over autogenerated Haskell files https://github.com/haskell/cabal/issues/8689 https://github.com/haskell/cabal/pull/8690

• Changed order or directories in GHC invocation so that source directories explicitly specified in cabal file will be considered before Cabal’s internal build directory.

https://github.com/haskell/cabal/blob/master/release-notes/WIP-cabal-install-3.12.x.0.md

(the fact that it's in release not yet-unreleased cabal-install rather than Cabal is a bug in itself)

These various orderings are such a mess: I think we've had several troubles like this one recently. I wonder if the ultimate solution is to check for duplicate entries in the set of paths and halt if such entries are found.

[andreasabel]

So this is an intended behavior change, not a regression. This means that one has to be more careful not to have outdated build-tool products left behind from manual invocations.

I wonder if the ultimate solution is to check for duplicate entries in the set of paths and halt if such entries are found.

Yes, it would not hurt if Cabal was more aware what it is doing. I have seen Stack emitting a warning in situations where there are duplicate sources for a module.

Also, since Cabal is a make-like tool, it could take timestamps into consideration. E.g. in my situation, Cabal-3.12 would "prefer" an existing Lexer.hs from 2023 over regenerating it from the Lexer.x of 2024. (Of course, it does not actively prefer something if it simply gives include paths to GHC in a certain order.)

Timestamps of course get lost during sdist, which creates this archive:

fred-0/Setup.hs
fred-0/fred.cabal
fred-0/src/
fred-0/src/Fred.hs
fred-0/src/Fred.y

Technically, the present issue is a change in Cabal-3.12 already and not just in cabal-install-3.12, so it would deserve some mentioning also in the Cabal-3.12 release notes at https://github.com/haskell/cabal/blob/master/release-notes/Cabal-3.12.0.0.md , with examples explaining the new behavior. (But maybe one was not aware of extent of the change introduced by #8690.)

[mpickering]

FWIW, hadrian has this behaviour (prefers .hs files over .x) files, which is by design so that we can ship source distributions which don't need alex/happy to build.

[sergv]

It looks like the trick is to know which module would be picked up and which the user actually intended to pick up.

It does seem cabal has facility to state that modules should be only generated from a non-.hs file via autogen-modules but OTOH it needs to do something (reasonable?) when that field is not used in the cabal file and module is just listed in a list of modules in which case cabal will magically do the right thing. Perhaps all the confusion is due to too much magic.

Maybe cabal should be sticter and plainly refuse to do anything if the user asked for, say, module Foo not listed under autogen-modules but cabal can only find Foo.x. And vice versa don't proceed if module is listed under autogen-modules but a .hs file was found.

[andreasabel]

@sergv wrote:

Maybe cabal should be sticter and plainly refuse to do anything if the user asked for, say, module Foo not listed under autogen-modules but cabal can only find Foo.x.

Yes, the problem is that in ambiguous situations, cabal autonomously decides what to do, and different versions decide differently. If one goes to a more stricter handling, one has to find a way not to break the thousands of of packages on Hackage (out of which at least 3000 are worth keeping, those that are tracked by Stackage).
One could bind the new behavior to the cabal-version. (However, folks are in general unwilling to bump the cabal-version because of fearing to lose buildability with older versions of cabal / older LTSs.)

[sergv]

@andreasabel It seems there's no good way other than introduce stricter handling with newest cabal-version, breaking lots of existing packages is not an option.

I guess people don't bump cabal version because it's not clear what it would improve other than reduce buildability. Personally I lean to newer cabal versions but even then I'm not sure which one is the newest and why it's better than 3.0. Potentially, and with suitable advertisement, ala "this release has less magic and brings more control to the programmer" people could start trying out e.g. cabal-version: 3.14 or something.