search

commercialhaskell / stack

The Haskell Tool Stack
http://haskellstack.org
BSD 3-Clause "New" or "Revised" License
4k stars 843 forks source link

`-fplugin` could not find module when using GHC 9.6 #6251

Open bristermitten opened 1 year ago

bristermitten commented 1 year ago

When trying to build a project using polysemy-plugin with GHC 9.6, the build fails with a strange error.

As far as I can tell this is a Stack issue, with the build command not including the package-db and package flags. GHC 9.4.7 works as intended

Steps to reproduce

  1. Clone this repository which contains a minimal reproducible example: https://github.com/knightzmc/psymy
  2. Run stack build
  3. It should throw an error
  4. Change the resolver to lts-21.12
  5. Build again
  6. It should work

Expected

Expected to build without errors or warnings.

Actual

The final stage of building throws an error

Installing library in /private/tmp/psymy/.stack-work/install/aarch64-osx/d648be467e8db3bbfce8aea3e8a40e40c2a4aa997e10a82332008572e1290566/9.6.2/lib/aarch64-osx-ghc-9.6.2/problematic-0.0.0.0-7Tqbt47XADvFA7jcxUQnZt
Error: Cabal-simple_6HauvNHV_3.10.1.0_ghc-9.6.2:
'/Users/alex/.ghcup/ghc/9.6.2/bin/ghc' exited with an error:
<command line>: Could not find module ‘Polysemy.Plugin’
Use -v (or `:set -v` in ghci) to see a list of the files searched for.

Completed 14 action(s).    

Error: [S-7282]
       Stack failed to execute the build plan.

       While executing the build plan, Stack encountered the error:

       [S-7011]
       While building package problematic-0.0.0.0 (scroll up to its section to see the error) using:
       /Users/alex/.stack/setup-exe-cache/aarch64-osx/Cabal-simple_6HauvNHV_3.10.1.0_ghc-9.6.2 --verbose=1 --builddir=.stack-work/dist/aarch64-osx/Cabal-3.10.1.0 register
       Process exited with code: ExitFailure 1

Stack version

stack --version
Version 2.11.1, Git revision d06506a16289c6b90e6ffc0dcc8be7f80487ac07 (dirty) aarch64 hpack-0.35.2

Method of installation

Platform

MacOS Sonoma aarch64, also reproducible on Ubuntu 22.04.2 LTS x86_64

mpilgrem commented 1 year ago

@knightzmc, thanks for reporting. With stack --verbose build --cabal-verbosity verbose I get (extract, reformatted):

GHC 9.4.7, with Cabal 3.8.1.0 (works):

[debug - Stack] Run process within C:\Users\mikep\Documents\Code\Haskell\psymy\: 
  C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_9p6GVs8J_3.8.1.0_ghc-9.4.7.exe 
--verbose=2 
--builddir=.stack-work\dist\74a2d300 
register

[info - Cabal] "C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.4.7\bin\ghc-9.4.7.exe" 
"--abi-hash" 
"-fbuilding-cabal-package" 
"-O" 
"-outputdir" ".stack-work\dist\74a2d300\build" 
"-odir" ".stack-work\dist\74a2d300\build" 
"-hidir" ".stack-work\dist\74a2d300\build" 
"-stubdir" ".stack-work\dist\74a2d300\build" 
"-i" 
"-i.stack-work\dist\74a2d300\build" 
"-isrc" 
"-i.stack-work\dist\74a2d300\build\autogen" 
"-i.stack-work\dist\74a2d300\build\global-autogen" 
"-I.stack-work\dist\74a2d300\build\autogen" 
"-I.stack-work\dist\74a2d300\build\global-autogen" 
"-I.stack-work\dist\74a2d300\build" 
"-IC:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20230526\mingw64\include"
"-optP-include" 
"-optP.stack-work\dist\74a2d300\build\autogen\cabal_macros.h" 
"-this-unit-id" "problematic-0.0.0.0-Av9fhBRW0672BVqZNF20AQ" 
"-hide-all-packages" 
"-Wmissing-home-modules" 
"-no-user-package-db" 
"-XGHC2021" 
"-XDataKinds" 
"A" 
"-fplugin=Polysemy.Plugin" 
"-fhide-source-paths"

GHC 9.6.2, with Cabal 3.10.1.0 (fails): 

[debug - Stack] Run process within C:\Users\mikep\Documents\Code\Haskell\psymy\: 
  C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_9p6GVs8J_3.10.1.0_ghc-9.6.2.exe 
--verbose=2 
--builddir=.stack-work\dist\18095c9f 
register

[info - Cabal] Running: "C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.6.2\bin\ghc-9.6.2.exe" 
"--abi-hash" 
"-fbuilding-cabal-package" 
"-O" 
"-outputdir" ".stack-work\dist\18095c9f\build" 
"-odir" ".stack-work\dist\18095c9f\build" 
"-hidir" ".stack-work\dist\18095c9f\build" 
"-stubdir" ".stack-work\dist\18095c9f\build" 
"-i" 
"-i.stack-work\dist\18095c9f\build" 
"-isrc" 
"-i.stack-work\dist\18095c9f\build\autogen" 
"-i.stack-work\dist\18095c9f\build\global-autogen" 
"-I.stack-work\dist\18095c9f\build\autogen" 
"-I.stack-work\dist\18095c9f\build\global-autogen" 
"-I.stack-work\dist\18095c9f\build" 
"-IC:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20230526\mingw64\include"
"-optP-include" 
"-optP.stack-work\dist\18095c9f\build\autogen\cabal_macros.h" 
"-this-unit-id" "problematic-0.0.0.0-7Tqbt47XADvFA7jcxUQnZt" 
"-hide-all-packages" 
"-Wmissing-home-modules" 
"-no-user-package-db" 
"-XGHC2021" 
"-XDataKinds" 
"A" 
"-fplugin=Polysemy.Plugin" 
"-fhide-source-paths"

[warn] CallStack (from HasCallStack):
[warn]   withMetadata, called at libraries\Cabal\Cabal\src\Distribution\Simple\Utils.hs:368:14 in Cabal-3.10.1.0:Distribution.Simple.Utils
[warn] Error: Cabal-simple_9p6GVs8J_3.10.1.0_ghc-9.6.2.exe:
[warn] 'C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.6.2\bin\ghc-9.6.2.exe'
[warn] exited with an error:
[warn] <command line>: Could not find module ‘Polysemy.Plugin’
[warn] Use -v (or `:set -v` in ghci) to see a list of the files searched for.
[warn]

It looks to me that Stack is giving identical commands to Cabal (the tool) and it, in turn, is giving the same command to different versions of GHC. The next step is to turn on GHC's verbosity ...

bristermitten commented 1 year ago

Ahh sorry, I totally forgot the verbose log! I was getting a totally unrelated error when using Cabal (the tool) which I assumed was just an issue with my machine, so perhaps this isn't a Stack issue at all. However when using Cabal (as in, cabal build) 3.10.1.0 and GHC 9.6.2 on x86_64 Linux it works as intended, whereas stack build fails.

The logs of both can be found here, https://gist.github.com/knightzmc/77e219e582242362b2d7d1370b529d28 There's a lot to break down but it definitely seems like Cabal is passing -package-id polysemy-1.9.1.2-03b9744b5a6b09461e989de3b3caa039034922d5f496217d0a114f5184ee5eed (and so on) to GHC whereas Stack is not

mpilgrem commented 1 year ago

With GHC's verbosity turned on, I see in the GHC error message:

[warn] Locations searched:
[warn] .stack-work\dist\18095c9f\build\Polysemy\Plugin.hi
[warn] .stack-work\dist\18095c9f\build\Polysemy\Plugin.hi-boot

That does not exist in \18095c9f\build - but it does not exist in \74a2d300\build (GHC 9.4.7, Cabal 3.8.1.0) either.

mpilgrem commented 1 year ago

Looking earlier in Stack's verbose output:

GHC 9.4.7 (extracts only):

Run process within C:\Users\mikep\Documents\Code\Haskell\psymy\: 
  C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_9p6GVs8J_3.8.1.0_ghc-9.4.7.exe
configure
--with-ghc=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.4.7\bin\ghc-9.4.7.exe
--dependency=base=base-4.17.2.0 
--dependency=polysemy=polysemy-1.9.1.1-LLER3VJ1tZ5CADZn3WfGsD 
--dependency=polysemy-plugin=polysemy-plugin-0.4.5.0-1vYmXOH4oflIZAGBHzZ4KA

GHC 9.6.2 (extracts only):

Run process within C:\Users\mikep\Documents\Code\Haskell\psymy\: 
  C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_9p6GVs8J_3.10.1.0_ghc-9.6.2.exe 
configure 
--with-ghc=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.6.2\bin\ghc-9.6.2.exe
--dependency=base=base-4.18.0.0 
--dependency=polysemy=polysemy-1.9.1.2-KKWKK0OMtIqEc5FzcWRYKA 
--dependency=polysemy-plugin=polysemy-plugin-0.4.5.1-CCks1M1SJDYB4s9lu9bde

Stack seems to be configuring Cabal (the tool) in an identical way - save for the change in the versions of polysemy and polysemy-plugin.

mpilgrem commented 1 year ago

Thinking out loud: why is GHC 9.6.2 looking for (and, of course, failing to find) Plugin.hi in .stack-work when the package polysemy-plugin is immutable? The Plugin.hi file is to be found in the snapshot.

EDIT: I am stumped. ghc-pkg 9.6.2 knows where to find Polysemy.Plugin in the Stack environment, why does GHC 9.6.2 not know?

GHC 9.4.7:

❯ stack exec -- ghc-pkg find-module Polysemy.Plugin
C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.4.7\lib\package.conf.d
    (no packages)
C:\sr\snapshots\4ef858c3\pkgdb
    polysemy-plugin-0.4.5.0

C:\Users\mikep\Documents\Code\Haskell\psymy\.stack-work\install\c3501600\pkgdb
    (no packages)

GHC 9.6.2:

❯ stack exec -- ghc-pkg find-module Polysemy.Plugin
C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.6.2\lib\package.conf.d
    (no packages)
C:\sr\snapshots\d12bd16e\pkgdb
    polysemy-plugin-0.4.5.1

C:\Users\mikep\Documents\Code\Haskell\psymy\.stack-work\install\d9daf07f\pkgdb
    (no packages)
mpilgrem commented 1 year ago

I pinned some of the dependencies to rule that out as the cause:

resolver: nightly-2023-09-22 # alternatively lts-21.12

extra-deps:
- Cabal-3.8.1.0
- Cabal-syntax-3.8.1.0
- polysemy-1.9.1.2
- polysemy-plugin-0.4.5.1
andreabedini commented 12 months ago

It is the same issue as https://github.com/haskell/cabal/issues/9375, it was fixed two weeks ago in https://github.com/haskell/cabal/pull/9384.

There was an old workaround for an issue with ghc-cabal (which is now gone). The workaround was to not pass -package-db to GHC when calling --abi-hash. It used to be unnecessary but with GHC 9.6 the plugin initialisation happens earlier than before, and this means we do need to pass -package-db.

gergoerdi commented 9 months ago

So if this has been fixed in Cabal, what can I, as a user of Stack, do to apply that Cabal fix?

christiaanb commented 9 months ago

@gergoerdi as a work-around you can:

  1. add the latest 3.10 branch of cabal to the extra-deps in your stack.yaml: 
    - git: https://github.com/haskell/cabal.git
commit: a3865991986361b3a736007f620b6a8878d178e3
subdirs:
- Cabal
  2. Change the build-type of your package's .cabal file to Custom
  3. Add a custom-setup stanza containing: 
    custom-setup
setup-depends:
base,
Cabal >= 3.10.2.1

    Very ugly, but it works.

mbj commented 9 months ago

@christiaanb Thank you!

But it did not work for me, before I sink a lot of time (you probably have already spend), do you happen to have a public project here this worked for you so I can check whatever I have done differently? I checked your clash repos but did not see this specific solution (or I'm bad in searching which may very well be the case!).

christiaanb commented 9 months ago

@mbj we have a variant of the fix at this PR: https://github.com/clash-lang/clash-compiler/pull/2665

And it’s passing CI with GHC 9.6.4: https://github.com/clash-lang/clash-compiler/actions/runs/7886581662/job/21520111671?pr=2665

mbj commented 9 months ago

@christiaanb ty, I cannot yet spot the difference to what I tried, but will keep digging!

telser commented 8 months ago

It is the same issue as haskell/cabal#9375, it was fixed two weeks ago in haskell/cabal#9384.

There was an old workaround for an issue with ghc-cabal (which is now gone). The workaround was to not pass -package-db to GHC when calling --abi-hash. It used to be unnecessary but with GHC 9.6 the plugin initialisation happens earlier than before, and this means we do need to pass -package-db.

This would seem to say the fix is in Cabal. @mpilgrem Are there dragons lurking in updating the dependency to 3.10.2.1? This is the blocker for getting to 9.6 right now. Would be happy to give that update a shot, unless there is some reason to hold off.

mpilgrem commented 8 months ago

@telser, (EDIT: Unless a custom Setup.hs is specified) Stack uses the version of Cabal (the library) that is provided with the specified version of GHC - the boot library. I do not think that is configurable (currently) (EDIT: unless a custom Setup.hs is used). It may be work-around-able, but I would have to think about that. At the moment, there is no released version of GHC that comes with Cabal >= 3.10.2.1.

EDIT: I've not looked into the full history of that design choice (to use only GHC's Cabal boot library) but some of it is here: https://github.com/commercialhaskell/stack/issues/4070. The reason given was 'You shouldn't generally do this [change the boot library version of Cabal], since new Cabal versions may introduce incompatibilities with package sets'.

telser commented 8 months ago

@telser, Stack uses the version of Cabal (the library) that is provided with the specified version of GHC - the boot library. I do not think that is configurable (currently). It may be work-around-able, but I would have to think about that. At the moment, there is no released version of GHC that comes with Cabal >= 3.10.2.1.

EDIT: I've not looked into the full history of that design choice (to use only GHC's Cabal boot library) but some of it is here: #4070. The reason given was 'You shouldn't generally do this [change the boot library version of Cabal], since new Cabal versions may introduce incompatibilities with package sets'.

@mpilgrem I see. If I'm understanding correctly then it really is ghc-pkg isn't working correctly since 9.6.1. Meaning ghc has been shipping with a fundamentally broken configuration for the last 2 major releases. It would seem that we need then the next point releases of 9.6 and 9.8 to bump the cabal versions.

EDIT: That is to say the situation is certainly frustrating for users. I haven't been able to find an issue for this on the ghc side, so I'll raise one there. The optics for this are frustrating because for users they can easily get the impression the issue is with the plugin(s) they use or stack or stack's integration with Cabal. With the better understanding though it feels like it is ghc-pkg. I'd love to be able to add a test case for this to ghc itself, but where/how to do that is unclear to me at this point.

telser commented 8 months ago

For anyone watching this, there is a ghc issue: https://gitlab.haskell.org/ghc/ghc/-/issues/24518

mbj commented 7 months ago

For anyone watching this: The cabal version with the fix has a release now. 3.10.3.0

mpilgrem commented 7 months ago

GHC 9.10.0.20240328 comes with Cabal-3.11.0.0 >= 3.10.3.0, but ghc-tcplugins-extra, a direct dependency of polysemy-plugin, does not yet support GHC 9.10.

goertzenator commented 7 months ago

I've worked around this by just disabling the plugin (effectful-plugin in my case) for now. I had to add exactly one type annotation to get my code base to compile, so not a big deal. YMMV of course.

gergoerdi commented 7 months ago

@gergoerdi as a work-around you can:

  1. add the latest 3.10 branch of cabal to the extra-deps in your stack.yaml:
 - git: https://github.com/haskell/cabal.git
   commit: a3865991986361b3a736007f620b6a8878d178e3
   subdirs:
   - Cabal
  1. Change the build-type of your package's .cabal file to Custom
  2. Add a custom-setup stanza containing:
custom-setup
  setup-depends:
    base,
    Cabal >= 3.10.2.1

Very ugly, but it works.

Unfortunately, now that I've tried this, it still fails with the same Could not find module ‘GHC.TypeLits.KnownNat.Solver’ error message.

gergoerdi commented 7 months ago

Unfortunately, now that I've tried this, it still fails with the same Could not find module ‘GHC.TypeLits.KnownNat.Solver’ error message.

Unless... maybe this needs to be applied to all Clash-using dependencies as well, not just the main program? @christiaanb I'll try that out.

mpilgrem commented 7 months ago

I could also get @knightzmc's minimal example to work with GHC 9.6.4 by:

  1. Adding a Setup.hs:

    import Distribution.Simple
main = defaultMain

  2. Using a custom Setup.hs built against Cabal-3.10.3.0 and Cabal-syntax-3.10.3.0.

    package.yaml (extract):

    custom-setup:
  dependencies:
  - base
  - Cabal ==3.10.3.0
  - Cabal-syntax ==3.10.3.0

    stack.yaml:

    snapshot: lts-22.17 # GHC 9.6.4
extra-deps:
- Cabal-3.10.3.0
- Cabal-syntax-3.10.3.0
hasufell commented 7 months ago

If I'm understanding correctly then it really is ghc-pkg isn't working correctly since 9.6.1. Meaning ghc has been shipping with a fundamentally broken configuration for the last 2 major releases.

@telser can you expand on that? I don't see any mention of ghc-pkg in the GHC issue you linked.

telser commented 7 months ago

If I'm understanding correctly then it really is ghc-pkg isn't working correctly since 9.6.1. Meaning ghc has been shipping with a fundamentally broken configuration for the last 2 major releases.

@telser can you expand on that? I don't see any mention of ghc-pkg in the GHC issue you linked.

My, current, understanding is that ghc-pkg is what makes Cabal a boot library. But this was really my swapping of ghc-pkg and GHC in reading the following comment from earlier in this thread:

Thinking out loud: why is GHC 9.6.2 looking for (and, of course, failing to find) Plugin.hi in .stack-work when the package polysemy-plugin is immutable? The Plugin.hi file is to be found in the snapshot.

EDIT: I am stumped. ghc-pkg 9.6.2 knows where to find Polysemy.Plugin in the Stack environment, why does GHC 9.6.2 not know?

GHC 9.4.7:

❯ stack exec -- ghc-pkg find-module Polysemy.Plugin
C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.4.7\lib\package.conf.d
    (no packages)
C:\sr\snapshots\4ef858c3\pkgdb
    polysemy-plugin-0.4.5.0

C:\Users\mikep\Documents\Code\Haskell\psymy\.stack-work\install\c3501600\pkgdb
    (no packages)

GHC 9.6.2:

❯ stack exec -- ghc-pkg find-module Polysemy.Plugin
C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.6.2\lib\package.conf.d
    (no packages)
C:\sr\snapshots\d12bd16e\pkgdb
    polysemy-plugin-0.4.5.1

C:\Users\mikep\Documents\Code\Haskell\psymy\.stack-work\install\d9daf07f\pkgdb
    (no packages)
mpilgrem commented 6 months ago

I could get @knightzmc's minimal example to work with GHC 9.6.5, with:

stack.yaml:

snapshot: lts-22.17 # GHC 9.6.4
compiler: ghc-9.6.5 # Comes with Cabal-3.10.3.0

# Temporary, until GHC 9.6.5 is added to Stack's default setup-info dictionary:
setup-info:
  ghc:
    windows64:
      9.6.5:
        url: https://downloads.haskell.org/ghc/9.6.5/ghc-9.6.5-x86_64-unknown-mingw32.tar.xz