build fails on container haskell:9.6

vargonis commented 2 months ago

I'm trying (to no avail) to build hyphen. I saw that commit bff0df7 apparently introduced support for ghc 9.6 and decided to give it a try within a docker container (so as to have a clean environment). Within the container, I cloned this repo and followed the instructions on the BUILDING.md file. The script fails, with the following error message:

root@84adde6b2901:/home/hyphen# python3 hyphen/build-extn.py 
[ 1 of 12] Compiling HyphenBase       ( /home/hyphen/hyphen/lowlevel_src/HyphenBase.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenBase.o )
[ 2 of 12] Compiling HyphenKinds      ( /home/hyphen/hyphen/lowlevel_src/HyphenKinds.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenKinds.o )

/home/hyphen/hyphen/lowlevel_src/HyphenKinds.hs:9:1: error:
    Could not find module ‘Data.Hashable’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
9 | import Data.Hashable
  | ^^^^^^^^^^^^^^^^^^^^
[ 6 of 12] Compiling PythonBase       ( /home/hyphen/hyphen/lowlevel_src/PythonBase.hs, /home/hyphen/hyphen/lowlevel_inter/PythonBase.o )
[ 9 of 12] Compiling Pythonate        ( /home/hyphen/hyphen/lowlevel_src/Pythonate.hs, /home/hyphen/hyphen/lowlevel_inter/Pythonate.o )

/home/hyphen/hyphen/lowlevel_src/Pythonate.hs:12:1: error:
    Could not find module ‘Data.HashMap.Strict’
    Perhaps you meant
      Data.Map.Strict (from containers-0.6.7)
      Data.IntMap.Strict (from containers-0.6.7)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
12 | import Data.HashMap.Strict (HashMap)
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

/home/hyphen/hyphen/lowlevel_src/Pythonate.hs:18:1: error:
    Could not find module ‘Data.HashMap.Strict’
    Perhaps you meant
      Data.Map.Strict (from containers-0.6.7)
      Data.IntMap.Strict (from containers-0.6.7)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
18 | import qualified Data.HashMap.Strict  as HashMap
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
  File "hyphen/build-extn.py", line 136, in <module>
    subprocess.check_call(invocation + ['-no-link'])
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ghc', '-dynamic', '-shared', '-fPIC', '-no-hs-main', '-fwarn-unused-imports', '-cpp', '/home/hyphen/hyphen/lowlevel_src/Hyphen.hs', '/home/hyphen/hyphen/lowlevel_src/hyphen_c.c', '-i/home/hyphen/hyphen/lowlevel_src', '-hidir', '/home/hyphen/hyphen/lowlevel_inter', '-odir', '/home/hyphen/hyphen/lowlevel_inter', '-I/usr/include/python3.7m', '-package', 'ghc', '-flink-rts', '-threaded', '-lpython3.7m', '-no-link']' returned non-zero exit status 1.
root@84adde6b2901:/home/hyphen# exit

A quick search over the internet seems to indicate that GHC 7.6 was the last version to have such a module in base https://stackoverflow.com/questions/41662486/cabal-install-missingpy-can-t-find-data-hashtable. Any hint as to what I may be doing wrong here?

tbarnetlamb commented 2 months ago

Hello! I'm sorry you're having trouble building hyphen. I'll do my best to help.

I think that the problem is that cabal a while ago had a backwards-incompatible change and that following commands from BUILDING.md:

cabal install --reinstall --force-reinstalls --enable-shared text
cabal install --reinstall --force-reinstalls --enable-shared transformers
cabal install --reinstall --force-reinstalls --enable-shared mtl
cabal install --reinstall --force-reinstalls --enable-shared parsec
cabal install --reinstall --force-reinstalls --enable-shared hashable
cabal install --reinstall --force-reinstalls --enable-shared unordered-containers
cabal install --enable-shared --reinstall ghc-paths

should no longer read cabal install but instead read cabal v1-install. In fact, you can combine all the above into a single command, thusly:

cabal v1-install --reinstall --force-reinstalls --enable-shared text transformers mtl parsec hashable unordered-containers ghc-paths

as is done in https://github.com/tbarnetlamb/hyphen/blob/main/.github/workflows/ci.yml

If this works for you, I will update the instructions. (My apologies, when this change to cabal went in I forgot to update the BUILDING instructions.)

Two other things

FWIW, I think that all production users of hyphen use stack instead of cabal as a build system. There are examples of that kind of build in the ci.yml file linked above as well. I should probably update BUILDING.md to be stack-first at some point. You may want to use stack rather than cabal.
This issue reminded me that I have a WIP PR, #31, that I ought to finish and merge. I was hitting some issues where (I think) some recent python releases were unable to recognize hyphen. I didn't have time for a deep dive back then. I'll aim to get to the bottom of that in the next day or so...

vargonis commented 2 months ago

Thanks for all the info. I'm trying again and seem to be getting further, but now I have another issue:

root@6a43b5536636:/home/hyphen# python3 hyphen/build-extn.py 
[ 1 of 12] Compiling HyphenBase       ( /home/hyphen/hyphen/lowlevel_src/HyphenBase.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenBase.o )
[ 2 of 12] Compiling HyphenKinds      ( /home/hyphen/hyphen/lowlevel_src/HyphenKinds.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenKinds.o )
[ 3 of 12] Compiling HyphenTyCon      ( /home/hyphen/hyphen/lowlevel_src/HyphenTyCon.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenTyCon.o )
[ 4 of 12] Compiling HsType           ( /home/hyphen/hyphen/lowlevel_src/HsType.hs, /home/hyphen/hyphen/lowlevel_inter/HsType.o )
[ 5 of 12] Compiling HyphenUnify      ( /home/hyphen/hyphen/lowlevel_src/HyphenUnify.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenUnify.o )
[ 6 of 12] Compiling PythonBase       ( /home/hyphen/hyphen/lowlevel_src/PythonBase.hs, /home/hyphen/hyphen/lowlevel_inter/PythonBase.o )
[ 7 of 12] Compiling HsObjRaw         ( /home/hyphen/hyphen/lowlevel_src/HsObjRaw.hs, /home/hyphen/hyphen/lowlevel_inter/HsObjRaw.o )
[ 8 of 12] Compiling HyphenWrapping   ( /home/hyphen/hyphen/lowlevel_src/HyphenWrapping.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenWrapping.o )
[ 9 of 12] Compiling Pythonate        ( /home/hyphen/hyphen/lowlevel_src/Pythonate.hs, /home/hyphen/hyphen/lowlevel_inter/Pythonate.o )
[10 of 12] Compiling HyphenExceptions ( /home/hyphen/hyphen/lowlevel_src/HyphenExceptions.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenExceptions.o )
[11 of 12] Compiling HyphenGHC        ( /home/hyphen/hyphen/lowlevel_src/HyphenGHC.hs, /home/hyphen/hyphen/lowlevel_inter/HyphenGHC.o )

/home/hyphen/hyphen/lowlevel_src/HyphenGHC.hs:236:13: error: [GHC-39999]
    • No instance for ‘MC.MonadCatch GhcMonad.Ghc’
        arising from a use of ‘MC.handle’
    • In the first argument of ‘($)’, namely ‘MC.handle handler’
      In a stmt of a 'do' block:
        result <- MC.handle handler
                    $ do MC.bracket prep (const clean) . const $ do action
      In the expression:
        do logref <- liftIO $ newIORef ""
           dflags <- GhcMonad.getSessionDynFlags
           let prep = GHC.setSessionDynFlags $ dflags
               clean = GHC.setSessionDynFlags dflags
               ....
           result <- MC.handle handler
                       $ do MC.bracket prep (const clean) . const $ do ...
           ....
    |
236 |   result <- MC.handle handler $ do
    |             ^^^^^^^^^

/home/hyphen/hyphen/lowlevel_src/HyphenGHC.hs:237:5: error: [GHC-39999]
    • No instance for ‘MC.MonadMask GhcMonad.Ghc’
        arising from a use of ‘MC.bracket’
    • In the first argument of ‘(.)’, namely
        ‘MC.bracket prep (const clean)’
      In the first argument of ‘($)’, namely
        ‘MC.bracket prep (const clean) . const’
      In a stmt of a 'do' block:
        MC.bracket prep (const clean) . const $ do action
    |
237 |     MC.bracket prep (const clean) . const $ do
    |     ^^^^^^^^^^

/home/hyphen/hyphen/lowlevel_src/HyphenGHC.hs:798:31: error: [GHC-39999]
    • No instance for ‘MC.MonadCatch GhcMonad.Ghc’
        arising from a use of ‘MC.handle’
    • In the first argument of ‘($)’, namely ‘MC.handle handler’
      In the expression:
        MC.handle handler
          $ do result <- reportingGHCErrors Nothing
                           $ do tyths <- readGHCModule mname
                                let ...
                                ....
               return
                 $ case result of
                     Left errmsg -> (HashMap.empty, Nothing)
                     Right answer -> (answer, Just mname)
      In an equation for ‘readGHCModuleTycCanon’:
          readGHCModuleTycCanon mname
            = MC.handle handler
                $ do result <- reportingGHCErrors Nothing
                                 $ do tyths <- readGHCModule mname
                                      ....
                     return
                       $ case result of
                           Left errmsg -> ...
                           Right answer -> ...
            where
                handler ::
                  (Monad m) =>
                  MC.SomeException -> m (HashMap TyCon TyCon, Maybe Text)
                handler _ = return (HashMap.empty, Nothing)
    |
798 | readGHCModuleTycCanon mname = MC.handle handler $ do
    |                               ^^^^^^^^^
Traceback (most recent call last):
  File "hyphen/build-extn.py", line 136, in <module>
    subprocess.check_call(invocation + ['-no-link'])
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ghc', '-dynamic', '-shared', '-fPIC', '-no-hs-main', '-fwarn-unused-imports', '-cpp', '/home/hyphen/hyphen/lowlevel_src/Hyphen.hs', '/home/hyphen/hyphen/lowlevel_src/hyphen_c.c', '-i/home/hyphen/hyphen/lowlevel_src', '-hidir', '/home/hyphen/hyphen/lowlevel_inter', '-odir', '/home/hyphen/hyphen/lowlevel_inter', '-I/usr/include/python3.7m', '-package', 'ghc', '-flink-rts', '-threaded', '-lpython3.7m', '-no-link']' returned non-zero exit status 1.

I'm using the docker image haskell:9.6. Should I try with a different version?

tbarnetlamb commented 2 months ago

I must say I'm a little perplexed by this error. A MonadMask instance for GHC.Driver.Monad.Ghc (which in our context is called GhcMonad.Ghc is created here: https://github.com/ghc/ghc/blob/ghc-9.6.5-release/compiler/GHC/Driver/Monad.hs#L159 so it should be visible from here since we import GHC.Driver.Monad.Ghc, here: https://github.com/tbarnetlamb/hyphen/blob/main/hyphen/lowlevel_src/HyphenGHC.hs#L111

One thing to try: could you run

cabal v1-install --reinstall --force-reinstalls --enable-shared text transformers mtl parsec hashable unordered-containers ghc-paths exceptions

(adding exceptions at the end) and see if the build does any better?

tbarnetlamb commented 2 months ago

Ok. I can reproduce your issue now. I may not fix it today, but at least this is progress...

tbarnetlamb commented 2 months ago

This problem seems to exist on cabal but not on stack. This is why most of my automated testing hadn't seen it: most of the tests run using stack to build (like the clients of the library). I was running a single cabal build, but it ran against GHC 8.x and this problem only seems to arise on 9.0 and above.

Would it be possible for you to try building using stack instead of cabal? The following should build against the "resolver" lts-22.26 (which is GHC 9.6 based) , build everything, and then open a python session with hyphen available to test:

stack --arch x86_64 --resolver lts-22.26 build
stack --arch x86_64 --resolver lts-22.26 exec -- python3 hyphen/build-extn.py
stack --arch x86_64 --resolver lts-22.26 exec -- python3

You will be warned that stack is only tested against 9.4, but I think things will still work.

tbarnetlamb commented 2 months ago

Ok. I think I have a solution for cabal-based builds as well. Can you try starting from a clean container and, instead of doing

cabal v1-install --reinstall --force-reinstalls --enable-shared text transformers mtl parsec hashable unordered-containers ghc-paths

from my previous instructions, now do:

cabal v1-install --enable-shared text transformers mtl parsec hashable unordered-containers ghc-paths

In my tests, this fixed the issue.

vargonis commented 2 months ago

Hey there! I can confirm that, following your instructions, both builds (stack and cabal-based) work within a haskell:9.6 container. I'll probably stick to cabal, because having to launch python in that special way seems likely to produce trouble later, when integrating other stuff... Thank you for your quick support!

tbarnetlamb commented 2 months ago

That's great

I'll leave this issue open, if that's ok with you, until I've merged #31 (which will update the docs which caused this problem)
FWIW, the way that you have to launch python in a special way using stack is perhaps not as scary as it seems. All it's really doing is setting environment variables behind the scenes. You can do diff <(env | sort) <(stack exec -- env | sort) to see exactly what it's doing. See https://github.com/tbarnetlamb/hyphen/issues/7#issuecomment-560192722 for an example. Perhaps this demystification is helpful? That said, if you're happy with cabal, then that's great.

tbarnetlamb / hyphen

build fails on container haskell:9.6 #33