Feature Request: Test Groups

[maxsnew]

Right now when you run a test-suite one of 2 things happens

1. Everything Passes and you get 1 line of output.
2. At least 1 things fails and you get 1 line of output for each test case.

This doesn't scale to large test-suites. Preferably there should be a notion of a test group, which would control this printing at a finer scale. If everything in a test group passes you get 1 line but if anything fails you get only that test group expanding.

You can see the way HUnit handles this here: http://hackage.haskell.org/package/HUnit-1.2.5.2/docs/Test-HUnit-Base.html

[deadfoxygrandpa]

Yeah, I think this is the next obvious feature to add. I'll start thinking and trying out a few ideas.

[deadfoxygrandpa]

@maxsnew Check out the new suites branch. It lets you group tests into test suites and hides most of the output of a suite if it passes, only showing everything if the suite fails. It keeps most of the API the same.

[maxsnew]

Looks good for the most part. I'm going to try it out on the Elm test suite to see how it looks with tons of tests.

The one design decision I would question is only having one Test datatype. I think HUnit has a separate type for TestSuites and I think that would make sense:

data Test = TestCase Assertion | Suite TestSuite
data TestSuite = TestSuite String [Test]

Which I think would look nicer in that all of the runners could take TestSuites instead of a Test.

[maxsnew]

Here's the output from breaking a test that's deeply nested within test suites:

  22 suites run, containing 91 tests
  19 suites and 90 tests passed
  3 suites and 1 tests failed
Test Suite: Elm Standard Library Tests: FAILED
  Test Suite: List Tests: FAILED
      Test Suite: partition Tests: FAILED
            simple partition: passed.
            order check: passed.
            partition doc check 1: FAILED. Expected: ([0,0,2],[3,4,5]); got: ([0,1,2],[3,4,5])
            partition doc check 2: passed.
            partition stress test: passed.
      Test Suite: unzip Tests: all tests passed
      Test Suite: intersperse Tests: all tests passed
  Test Suite: String Tests: all tests passed
  Test Suite: Trampoline Tests: all tests passed
  Test Suite: Array Tests: all tests passed
  Test Suite: Dict Tests: all tests passed
  Test Suite: Set Tests: all tests passed

A huge improvement!

[deadfoxygrandpa]

Yeah, I was debating between having one or two datatypes, and I don't remember why I picked only one. After a couple weeks looking back at it, I think I made the wrong choice. I've basically turned every function into two functions mashed together with case

[deadfoxygrandpa]

Actually though, it looks like HUnit does use one data type:

data Test
    -- | A single, independent test case composed.
    = TestCase Assertion
    -- | A set of @Test@s sharing the same level in the hierarchy. 
    | TestList [Test]
    -- | A name or description for a subtree of the @Test@s.
    | TestLabel String Test

But I think I'll try splitting it up.

[maxsnew]

Hm maybe just leave it? It doesn't really matter.

[deadfoxygrandpa]

I guess so.

I still haven't implemented the Element runner in this branch. Is there anything else you think should improve before merging it in with master?

[maxsnew]

Nah just merge. Once you implement the Element runner and tag a release I'll send a pr to the Elm repo to use this.

[deadfoxygrandpa]

OK I'll try to get this done this weekend.

[deadfoxygrandpa]

I just implemented the Element runner, merged into master, and made a 0.3 release: https://github.com/deadfoxygrandpa/Elm-Test/releases/tag/0.3

[deadfoxygrandpa]

And of course, I forgot to bump the version number in elm_dependencies.json, hence: https://github.com/deadfoxygrandpa/Elm-Test/releases/tag/0.3.1

[maxsnew]

There's actually a new release of Elm coming this week ("0.12.1" I think, you can check the repo) so we need to update elm_dependencies.json again