CI-friendly test command

[ceedubs]

I thought that I might try to add a simple GitHub Actions CI for https://github.com/ceedubs/unison-foldl and I'm not sure how to run the tests in a way that will trigger a failed CI.

For example, I've added a failing test under foldl.failingtest. I'm not sure if there's a better way, but I'm using a heredoc to try to run the tests within ucm:

[nix-shell:~/code/unison]$ stack exec unison -- --new-runtime << EOF
cd foldl
test
EOF

  I've started a codebase API server at 

  http://127.0.0.1:65057?ajj7ur%2Bgi8MxFCWChsjhxSTsPs8FO5aa

   _____     _             
  |  |  |___|_|___ ___ ___ 
  |  |  |   | |_ -| . |   |
  |_____|_|_|_|___|___|_|_|

  Welcome to Unison!

  You are running version: devel/M1l-933-g5c65d3b25'

  I'm currently watching for changes to .u files under ~/code/unison

  Type help to get help. 😎

.> 
  I loaded scratch.u and didn't find anything.

.> .foldl> 
  Cached test results (`help testcache` to learn more)

  ◉ Fold.ap.tests.paired            : Proved.
  ◉ Fold.dimap.tests.paired         : Proved.
  ◉ Fold.foldSet.tests.sumNat       : Proved.
  ◉ folds.Nat.sum.tests.nonempty    : Proved.

  ✗ foldl.failingtest   

  🚫 1 test(s) failing,   4 test(s) passing

  Tip: Use view foldl.failingtest to view the source of a test.

.foldl> 
[nix-shell:~/code/unison]$ echo $?
0

As you can see, the return value is 0 even though there's a failing test. Is there a way to capture the fact that at least one test failed?

[aryairani]

There isn't, to my knowledge. My first thought is that we could change the ucm run command to pass through an exit code if the type of the specified program is Byte or {IO} Byte.

Or we could add a ucm test command that takes in a namespace, runs the tests, and returns 1 if any tests fail; the first is more general.

[ceedubs]

@aryairani my initial reaction is that a ucm test command makes a lot of sense. This is likely a common flow, so it probably makes sense to make it easy.

My afterthought is that maybe I'm used to other languages and this sort of thing isn't as necessary or there's a better approach with Unison. But I don't yet have any good ideas for what that would be :)

[aryairani]

Why not both! 😄

ucm test probably makes as much sense as the interactive test command, and we like that so far.