GHC 9.2 does not understand code produced by `alex -g`

[andreasabel]

The GHC-specific optimizations (alex option --ghc) are not compatible with GHC 9.2, it seems. I am getting errors like:

CPP/Lex.hs:452:50: error:
    • Couldn't match expected type ‘Int16#’ with actual type ‘Int#’
    • In the fifth argument of ‘alex_scan_tkn’, namely ‘sc’
      In the expression:
        alex_scan_tkn user__ input__ 0# input__ sc AlexNone
      In the expression:
        case alex_scan_tkn user__ input__ 0# input__ sc AlexNone of
          (AlexNone, input__')
            -> case alexGetByte input__ of
                 Nothing -> AlexEOF
                 Just _ -> AlexError input__'
          (AlexLastSkip input__'' len, _) -> AlexSkip input__'' len
          (AlexLastAcc k input__''' len, _)
            -> AlexToken input__''' len (alex_actions ! k)
    |
452 |   = case alex_scan_tkn user__ input__ 0# input__ sc AlexNone of
    |                                                  ^^

CPP/Lex.hs:482:1: error:
    Couldn't match type ‘Int#’ with ‘Int16#’
    Expected: t
              -> t1
              -> Int#
              -> AlexInput
              -> Int16#
              -> AlexLastAcc
              -> (AlexLastAcc, AlexInput)
      Actual: t
              -> t1
              -> Int#
              -> AlexInput
              -> Int#
              -> AlexLastAcc
              -> (AlexLastAcc, AlexInput)
    |
482 | alex_scan_tkn user__ orig_input len input__ s last_acc =
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...

CPP/Lex.hs:497:27: error:
    • Couldn't match expected type ‘Int#’ with actual type ‘Int32#’
    • In the first argument of ‘(+#)’, namely ‘base’
      In the expression: base +# ord_c
      In an equation for ‘offset’: offset = (base +# ord_c)
    |
497 |                 offset = (base +# ord_c)
    |                           ^^^^

CPP/Lex.hs:500:73: error:
    • Couldn't match expected type ‘Int#’ with actual type ‘Int16#’
    • In the first argument of ‘(==#)’, namely ‘check’
      In the first argument of ‘tagToEnum#’, namely ‘(check ==# ord_c)’
      In the second argument of ‘(&&)’, namely
        ‘(tagToEnum# (check ==# ord_c))’
    |
500 |                 new_s = if GTE(offset,0#) && EQ(check,ord_c)
    |                                                                         ^^^^^

CPP/Lex.hs:505:13: error:
    • Couldn't match expected type ‘Int16#’ with actual type ‘Int#’
    • In the pattern: -1#
      In a case alternative: -1# -> (new_acc, input__)
      In the expression:
        case new_s of
          -1# -> (new_acc, input__)
          _ -> alex_scan_tkn
                 user__ orig_input
                 (if c < 0x80 || c >= 0xC0 then (len +# 1#) else len) new_input
                 new_s new_acc
    |
505 |             -1# -> (new_acc, input__)
    |             ^^^

[andreasabel]

This could maybe fixed with some #if MIN_VERSION_ghc_prim(0,8,0) conditionals, cf. https://gitlab.haskell.org/ghc/ghc/-/issues/19099. I suppose work on this issue is already ongoing, maybe in the ghc-9.2 branch. However, at the present, the alex from the ghc-9.2 does not seem to bootstrap. It tried make sdist-test with allow-newer and it fails:

Build profile: -w ghc-9.2.0.20210422 -O1
In order, the following will be built (use -v for more details):
 - alex-3.2.6 (exe:alex) (first run)
 - alex-3.2.6 (test:tests) (first run)
 ...
[15 of 22] Compiling Scan             ( src/Scan.hs, /Users/abel/project/open-source/alex/dist-newstyle/sdist/alex-3.2.6/dist-newstyle/build/x86_64-osx/ghc-9.2.0.20210422/alex-3.2.6/x/alex/build/alex/alex-tmp/Scan.o )

src/Scan.hs:585:50: error:
    • Couldn't match expected type ‘Int16#’ with actual type ‘Int#’
    • In the fifth argument of ‘alex_scan_tkn’, namely ‘sc’
      In the expression:
        alex_scan_tkn user__ input__ 0# input__ sc AlexNone
      In the expression:
        case alex_scan_tkn user__ input__ 0# input__ sc AlexNone of
          (AlexNone, input__')
            -> case alexGetByte input__ of
                 Nothing -> AlexEOF
                 Just _ -> AlexError input__'
          (AlexLastSkip input__'' len, _) -> AlexSkip input__'' len
          (AlexLastAcc k input__''' len, _)
            -> AlexToken input__''' len (alex_actions ! k)
    |
585 |   = case alex_scan_tkn user__ input__ 0# input__ sc AlexNone of
    |                                                  ^^
...

[Ericson2314]

Does GHC itself no use --ghc? Otherwise I am confused why this is happening as I thought 3.2.6 was used in 9.2 and working just fine.

[andreasabel]

I thought 3.2.6 was used in 9.2 and working just fine.

Isn't it that GHC 9.2 is boot-strapped with GHC <= 9.0, so if Alex 3.2.6 outputs code that works with GHC <= 9.0 then GHC 9.2 compiles fine? (I don't know exactly how boot-strapping GHC works...)

[Ericson2314]

@andreasabel A regular GHC build builds GHC twice, bootstrapping the second new GHC with the first so that the ABIs of the produced code and the GHC itself match for GHCi and TH.

That means the version of Alex needs to work with with 9.0 and 9.2.

[andreasabel]

(Btw. what I find a bit confusing is that 3.2.6 is released, but the dev version is also called 3.2.6 atm. Usually, after the release I immediately bump the version. This need not be the final version for the next release, but at least it is strictly greater than the last released version. So, for now we have to live with the ambiguity what 3.2.6 means, but as I understand it, 3.2.6 (released) is supposed to work with GHC 9.2.)

I can only reiterate that currently GHC 9.2 does not accept files generated by alex 3.2.6 with option --ghc. It does work without that option.

I think responsible are changes in ghc-prim from 0.7 to 0.8.

UPDATE: Sorry, I think I had a version of alex that was called 3.2.6 but was not the released 3.2.6. :$ Sorry for the noice, after a reinstall of alex, it seems to work well. I also see things like #if __GLASGOW_HASKELL__ >= 901 in the generated .hs code which I weren't there before.

This issue can be closed.

[Ericson2314]

Yes I should bump the dev version; I'll go do that!

[Ericson2314]

Did in https://github.com/simonmar/alex/commit/fbcb2e0c44707671a21f81e1f63eb1f9ced26862

[andreasabel]

haskell-ci has a lot of users of the development version and active development, so there phadej seems to append the date as minor version number. https://github.com/haskell-CI/haskell-ci/blob/8f31dc87caf71d517b6ebfc0066389d686a7807a/haskell-ci.cabal#L3 This is most likely an overkill in many situations, but it is rather water proof. So that would be something like 3.2.6.20210614 for alex. I haven't adopted this for my own projects, but I should remember the trick in case I maintain a busy project some day...

[andreasabel]

After getting boot-strapping to run, I realize that make sdist-test fails on -Werror=noncanonical-monad-instances for case tests/default_typeclass.x.

ghc -Wall -fwarn-incomplete-uni-patterns -fno-warn-missing-signatures -fno-warn-unused-imports -fno-warn-tabs -Werror -package array -package bytestring  default_typeclass.n.hs -o default_typeclass.n.bin
[1 of 1] Compiling Main             ( default_typeclass.n.hs, default_typeclass.n.o )

default_typeclass.x:282:5: error: [-Wnoncanonical-monad-instances, -Werror=noncanonical-monad-instances]
    Noncanonical ‘return’ definition detected
    in the instance declaration for ‘Monad (StateT s m)’.
    ‘return’ will eventually be removed in favour of ‘pure’
    Either remove definition for ‘return’ (recommended) or define as ‘return = pure’
    See also: https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/monad-of-no-return
    |
282 | 
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

default_typeclass.x:308:5: error: [-Wnoncanonical-monad-instances, -Werror=noncanonical-monad-instances]
    Noncanonical ‘pure = return’ definition detected
    in the instance declaration for ‘Applicative (StateT s m)’.
    Move definition from ‘return’ to ‘pure’
    See also: https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/monad-of-no-return
    |
308 | 
    |     ^^^^^^^^^^^^^

Is this a known problem or am I doing something wrong (like make sdist-test isn't the right way to invoke the testsuite anymore)?

[andreasabel]

As the testsuite also fails with cabal test, I filed PR #188 with the canonical fix suggested by the error messages.