codewars / runner

Issue tracker for Code Runner
32 stars 8 forks source link

Improve output of tests for frameworks which have no `<IT::>` #262

Open hobovsky opened 9 months ago

hobovsky commented 9 months ago

Initially asked on Discord:

Codewars runner supports some languages and testing frameworks which do not fit well into the BDD pattern and do not have constructs which map well into <DESCRIBE::> and <IT::> markers used by test output panel. As a consequence, some reporters emit artificially created <IT::> tags, with titles derived from some additional context data because authors cannot assign any title to them. This often results in an awkward output with confusing or misplaced titles (see #239 , #241 , https://github.com/codewars/FactCheck.jl/pull/3 ), or which spam <IT::> beyond reason.

My proposal would be to get rid of artificial <IT::>'s from all reporters for frameworks which do not have a corresponding construct, and add support for assertions which are not in scope of any <IT::>. Maybe it would be good enough to allow for assertions inside of <DESCRIBE::>, or maybe a better solution would be to provide some new marker, which would be used for frameworks which do not map well into BDD and use simpler concept of a test and a context, for exampel something like:

<TEST::>
  <TEST::>                     (if nesting is allowed)
    <ASSERTIONS_CONTEXT::>     (optional)
      <ASSERTIONS_CONTEXT::>   (if nesting is possible)
        <ASSERTION::>          (probably <PASSED::> or <FAILED::>)
        <ASSERTION::>          (if not stop on first failure)
      <COMPLETEDIN::>
    <COMPLETEDIN::>
  <COMPLETEDIN::>
<COMPLETEDIN::>

With such setup, Julia reporter would emit <TEST::> for @facts (which is optional), <ASSERTIONS_CONTEXT::> for @context (which is optional), and <PASSED::> or <FAILED::> for @fact. For Clojure, we'd get <TEST::> for deftest (whici is mandatory), <ASSERTIONS_CONTEXT::> for testing (which is optional), and <PASSED::> or <FAILED::> for is.

hobovsky commented 9 months ago

For what it's worth I admit that the main reason behind this ticket is the fact how discouraged I can sometimes be to fix some translations, because no matter what I try, there's simply no way to get a nice and clear output of tests for some languages. This can be also a problem for reviewers who can be not familiar with intricacies of setup of some languages on Codewars and sometimes request to make the output of tests clearer before a translation gets approved, even though it's not possible/not easy.

hobovsky commented 8 months ago

Another example: Perl

image

The test 1 blocks have no corresponding construct in code of tests and are most probably emitted by CW reporter to provide an obligatory <IT::>. It is not known how to provide a meaningful title for such it instead of the test N.

hobovsky commented 8 months ago

Another affected language: Raku