Build fails due to missing Alex data constructors

[ghost]

    /home/siddhu/code/yi/yi-misc-modes/src/Yi/Lexer/common.hsinc:75:17-27: error:
        Not in scope: data constructor ‘AlexAccPred’

    /home/siddhu/code/yi/yi-misc-modes/src/Yi/Lexer/common.hsinc:80:17-31: error:
        Not in scope: data constructor ‘AlexAccSkipPred’
        Perhaps you meant ‘AlexAccSkip’ (line 207)

Just did git clone and stack build.

$ stack exec -- alex --version
Alex version 3.2.1, (c) 2003 Chris Dornan and Simon Marlow
$ stack exec -- ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.0.2
$ cat stack.yaml | grep resolver
resolver: lts-8.18

[stites]

I was working on this last week -- the crazy part about this bug is that the data constructors are actually there for me (with alex 3.2.1), but aren't detected in some later phase of the build (hence the error). Another crazy thing is that, while common.hsinc is used in more than just yi-misc-modes, taking out misc-modes from your build will let compile fine.

The last thing I was doing was removing most of the .x files, duplicating common.hsinc in the remaining .xs, and building haskell files with alex manually, but I needed a mental break after realizing that this is working fine in other people's builds, and that the same structure exists in haskell and javascript mode and builds without a hitch.

[ghost]

Thanks! The offending lines are the following:

--- a/yi-misc-modes/src/Yi/Lexer/common.hsinc
+++ b/yi-misc-modes/src/Yi/Lexer/common.hsinc
@@ -71,18 +71,6 @@ alex_scan_tkn' user orig_input len input s last_acc =
     check_accs (AlexAccNone) = last_acc
     check_accs (AlexAcc a  ) = AlexLastAcc a input IBOX(len)
     check_accs (AlexAccSkip) = AlexLastSkip  input IBOX(len)
-#ifndef NO_ALEX_CONTEXTS
-    check_accs (AlexAccPred a predx rest)
-       | predx user orig_input IBOX(len) input
-       = AlexLastAcc a input IBOX(len)
-       | otherwise
-       = check_accs rest
-    check_accs (AlexAccSkipPred predx rest)
-       | predx user orig_input IBOX(len) input
-       = AlexLastSkip input IBOX(len)
-       | otherwise
-       = check_accs rest
-#endif

The common.hsinc files elsewhere might be referring to different lexer files.

[stites]

Yup! For me, I can confirm that running alex (via stack) on any of these files correctly generates these data constructors, they just don't exist in the common file. Furthermore, when I c/p'd this common.hsinc into one of the files and ran this, the error propagated to BasicTemplate.hs

Also, it seems like common.hsinc is copy-pasta'd into all of the Lexer modules (see: yi-mode-haskell, yi-mode-javascript, yi-language)

[stites]

Note that common.hsinc is a workaround for alex not exposing library files specifically for yi. See https://github.com/simonmar/alex/issues/45. Maybe you can help shed some light, @Fuuzetsu?

[Fuuzetsu]

I have no input except that we should strive to throw away all this Alex stuff at any cost.

[stites]

haha! In that case, I'll link #946 for posterity since that seems to be the preferred ticket.

[stites]

Nevermind, I found the bug and made a PR -- solved it by making a python-mode package (as a workaround), then slowly reintroducing new languages. @siddhanathan can you confirm that this fixes your issue?

[noughtmare]

Wait, I have discovered now that Yi.Lexer.BasicTemplate isn't imported by any other library and it is also not included in the cabal file of yi-misc-modes, so it shouldn't get compiled at all.

[noughtmare]

I can now confirm that it doesn't compile Yi.Lexer.BasicTemplate when I compile it seperately with cabal.

To reproduce:

git clone https://github.com/yi-editor/yi
cd yi/yi-misc-modes
cabal sandbox init
cabal install hpack
hpack
cabal install --dependencies-only
cabal build

[noughtmare]

I found the issue/cause! A newer hpack version (at least 0.18.1) generates a different cabal file which includes the Yi.Lexer.BasicTemplate as an other-module.

[noughtmare]

I also found that some lexers use the alex contexts (these do not contain #define NO_ALEX_CONTEXTS):

• Perl.x
• GNUMake.x
• GitCommit.x
• Cabal.x
• Haskell.x
• LiterateHaskell.x

[stites]

Nice going! Yes, I recall hpack (0.18.1) adding new other-modules to cabal, and I can confirm that this is happening on my box. Perhaps the right thing to do to account for these hpack /stack versions and keep things correct is to move src/Yi/Lexer/BasicTemplate.hs to template/Yi/Lexer/BasicTemplate.hs? Also to file a feature request on hpack for a hidden-modules field?

[stites]

I'm not sure where to put this (I remember seeing #include "SomeParserMixin.hs" in the alex parser via this issue) but I will point out that backpack is now in GHC and might be worth investigating to replace this kind of code. See the backpack proposal overview.

[noughtmare]

@stites I'm not sure if I understand what you're saying, but do you mean that backpack might solve https://github.com/yi-editor/yi/issues/914? I think it should also be possible to solve that issue with type classes, but I haven't had the time to try to fix it yet.

[noughtmare]

@stites After looking at #914 again I don't think that this can be solved with backpack or type classes. The problem lies in the fact that the lexers get generated by alex which declares its own data types during code generation, so it is necessary to include the common.hsinc file and it is not possible to make it a proper imported module.