Here's what I've managed to cobble together for a testing framework. This simply calls hermit-shell on each example and sees if it runs to completion. To run the tests, type
cabal test --show-details=streaming
The --show-details=streaming bit is optional, but it shows you the results of each test as it's running, which is nice.
You can filter which tests get run by searching against a pattern. For example:
cabal test --show-details=streaming --test-options="-p concatVanishes"
Current limitations:
At the moment, the only way for a test to fail is if hermit-shell fails to launch (e.g., if the .hs file it's run on isn't present). Lamentably, GHCi itself doesn't seem to have a mechanism for returning a non-zero exit code, so once GHCi starts, it's pretty much guaranteed to return ExitSuccess, even if something did go wrong (e.g., if the script didn't typecheck). This makes it less than ideal for running on Travis at the moment, since we wouldn't readily know if something went wrong.
In the future, we should probably diff the output against a golden file like the HERMIT tests do.
The cabal test options are pretty annoying to type. If my make-fu were stronger, I could alias the above commands with something more memorable.
Here's what I've managed to cobble together for a testing framework. This simply calls
hermit-shell
on each example and sees if it runs to completion. To run the tests, typeThe
--show-details=streaming
bit is optional, but it shows you the results of each test as it's running, which is nice.You can filter which tests get run by searching against a pattern. For example:
Current limitations:
At the moment, the only way for a test to fail is if
hermit-shell
fails to launch (e.g., if the.hs
file it's run on isn't present). Lamentably, GHCi itself doesn't seem to have a mechanism for returning a non-zero exit code, so once GHCi starts, it's pretty much guaranteed to returnExitSuccess
, even if something did go wrong (e.g., if the script didn't typecheck). This makes it less than ideal for running on Travis at the moment, since we wouldn't readily know if something went wrong.In the future, we should probably
diff
the output against a golden file like the HERMIT tests do.cabal test
options are pretty annoying to type. If mymake
-fu were stronger, I could alias the above commands with something more memorable.