unisonweb / unison

A friendly programming language from the future
https://unison-lang.org
Other
5.81k stars 271 forks source link

Add support for “known failures” to transcripts #5394

Open sellout opened 1 month ago

sellout commented 1 month ago

Overview

This allows issues that haven’t yet been resolved to be reflected in the transcripts. This makes it easier to close tickets as soon as issues have been fixed, and avoids having to manually re-try transcripts from a ticket to see if the situation is the same.

Fixes #5350.

Implementation notes

This adds two new transcript tags :failure and :incorrect. They behave similarly to :error and no tag, respectively, except the error messages are different when the results don’t match.

Test coverage

This adds three new transcripts for open tickets exercising the two new tags.

Loose ends

There are lots of open tickets that should also get transcripts added.

aryairani commented 1 month ago

I wanna think a little more about what the label names should be, because I think error failure and incorrect are hard to distinguish; but otherwise this looks great.

sellout commented 1 month ago

I agree. It might be worth doing #5214 before this.