unisonweb / unison

A friendly programming language from the future
https://unison-lang.org
Other
5.81k stars 271 forks source link

io.test does not fail if term doesn't exist #5399

Open ceedubs opened 1 month ago

ceedubs commented 1 month ago

Describe and demonstrate the bug

In a transcript if you call io.test and a test fails, then it errors with a non-zero exit code. However, if you call io.test with a term that doesn't exist, it returns a success (0) exit code. This can mask errors (such as in CI) where you think that tests are passing but in fact they aren't running.

Input:

```ucm
fresh/main> io.test doesNotExist

Output:
fresh/main> io.test doesNotExist

    New test results:

  😶 No tests available.


You can see in the output that it detects that there are no tests. What you can't see from the transcript output is that the `transcript` command exits with 0.

I think that the following should all probably be considered errors:

- Passing in a nonexistent term
- Passing in a term with the wrong type
- Zero tests running (even if the term exists and has the right type, if it returns an empty test result list then something is probably off). But I'm less sure about this one.

**Environment (please complete the following information):**
 - `ucm --version` 572943993
 - OS/Architecture: x86 Linux

**Additional context**
Add any other context about the problem here.