search

ku-fpg / hermit-shell

HERMIT with GHCi shell
BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

Automated tests succeed if there is a GHC error (type error or name not in scope) #17

Closed roboguy13 closed 9 years ago

roboguy13 commented 9 years ago

I just made a change that breaks the nub example because it changes the name of lhs to either lhsT or lhsR resulting in a "Not in scope" error if I manually try this test (caac767 is the commit that has this error). I notice that if I run cabal test, it still shows all the tests as succeeding. I think this is because the hermit-shell itself doesn't fail with an error code at this point.

ecaustin commented 9 years ago

Yup, that's the current expected behavior (see: https://github.com/ku-fpg/hermit-shell/pull/13).

Right now we have too much debug output that changes from execution to execution to do a basic dump and diff test which I believe is the long term plan.

roboguy13 commented 9 years ago

I just added a temporary workaround to fix this. The tests now use ghc -fno-code to type check each script before it runs hermit-shell.

A side-effect of this is that all the scripts must have module names or else there is a "main" is not defined in module "Main" error (which probably shouldn't occur when ghc gets the -fno-code option, but that's a different issue for a different project). I think don't think that ghc-mod check has this issue, but I didn't want to bring in a large dependency for a temporary fix.

RyanGlScott commented 9 years ago

I didn't know about the -fno-code option. That's a pretty slick idea!

Honestly, ghc -fno-code is probably the best bet for ensuring that the script compiles. ghci is weird in that as long as it is given valid arguments, it won't abort upon a compilation error—it will just go back to the default REPL loop. To make things worse, I don't think there is any way you can force the REPL to exit with an error code.

And since we're calling ghci via a subprocess, just about everything in the test suite is a hack. But I'd like to think it's a pretty good hack. :)