[Discussion] testEach / describeEach

[andys8]

TL;DR

Should we add syntactic sugar to test input-to-expected-result tables?

Background

Jest recently included describe.each and test.each. Could it work for elm-test?

test.each([[1, 1, 2], [1, 2, 3], [2, 1, 3]])(
  '.add(%i, %i)',
  (a, b, expected) => {
    expect(a + b).toBe(expected);
  },
);

Examples

Example: Generic

Polymorph in its argument: testEach : String -> List a -> (a -> Expect.Expectation) -> List Test

module SearchTest exposing (all)

import Expect
import Search exposing (Filter(..), toFilters)
import Test exposing (Test, describe, test)

-- usage example

all : Test
all =
    describe "Search"
        [ describe "test toFilters"
            (testEach "toFilters"
                [ ( "", [] )
                , ( " ", [] )
                , ( "a", [ TextFilter "a" ] )
                , ( "a b", [ TextFilter "a", TextFilter "b" ] )
                ]
             <|
                \( a, b ) ->
                    a
                        |> toFilters
                        |> Expect.equal b
            )
        ]

-- test each implementation

testEach : String -> List a -> (a -> Expect.Expectation) -> List Test
testEach name table fn =
    let
        toTest item =
            let
                testName =
                    name ++ " for input " ++ toString item
            in
            test testName <|
                \_ ->
                    fn item
    in
    List.map toTest table

Example: Constrained

Would work with fixed tuples. testEach : String -> List ( a, b ) -> (a -> b) -> List Test

module SearchTest exposing (all)

import Expect
import Search exposing (Filter(..), toFilters)
import Test exposing (Test, describe, test)

-- usage example

all : Test
all =
    describe "Search"
        [ describe "test toFilters"
            (testEach "toFilters"
                [ ( "", [] )
                , ( " ", [] )
                , ( "a", [ TextFilter "a" ] )
                , ( "a b", [ TextFilter "a", TextFilter "b" ] )
                ]
                toFilters
            )
        ]

-- test each implementation

testEach : String -> List ( a, b ) -> (a -> b) -> List Test
testEach name table fn =
    let
        toTest (( input, expectedResult ) as tuple) =
            let
                testName =
                    name ++ " for input " ++ toString tuple
            in
            test testName <|
                \_ ->
                    fn input |> Expect.equal expectedResult
    in
    List.map toTest table

Open questions

• How would the api look like? Should it be restrained to List of tuples, which would lead to more variants?
• Should the api be constrained to Expect.equal only?
• What tests are possible that aren't better be solved with fuzz testing?
• Is it worth adding sugar? Will people be write less boilerplate and test suites that are easier to understand?
• How to create the test name, especially in 0.19?

[mgold]

I don't think this should be implemented.

• As you note, generating a name for each test is tricky. We use Debug.toString elsewhere under the hood, but I don't want to extend our reliance on it. elm-test requires that sibling names be unique, and this makes it harder to do that -- and harder to find which test has failed, since it's identified with an autogenerated name.
• It's super easy in Elm to extract a helper function, which can make the repeated tests not much more than the data and a name anyway.
• We already have looping and grouping constructs like Expect.all and describe.
• Compared to JS, Elm has a stronger ethos of "only one way to do it".

[andys8]

I found drathier/elm-test-tables and it looks great.

https://github.com/drathier/elm-test-tables https://package.elm-lang.org/packages/drathier/elm-test-tables/latest/Test-Table

It uses also toString. Can it be ported to elm 0.19 anyway?

Looking at the documentation of Debug.toString I'd think it's fine for testing libraries, because optimization is not used.

This is not available with elm make --optimize which gets rid of a bunch of runtime metadata. For example, it shortens record field names, and we need that info to toString the value! As a consequence, packages cannot use toString because they may be used in --optimize mode.

Can you confirm my assumptions?

[mgold]

elm-test is distributed on the package website, which enforces the requirements of being optimizable.

[andys8]

@drathier has been able to update elm-test-tables to 0.19. Therefore I don't see the need to get this feature in elm-test anymore.

https://package.elm-lang.org/packages/drathier/elm-test-tables/latest/Test-Table

[mgold]

Sounds good.

[Darker7]

The function

I was really surprised when I found that elm-test doesn't provide a test each function, so I sat down and wrote one for myself:

each : List a -> (a -> String) -> (a -> Expectation) -> List Test
each values name function =
    List.map (\val -> test (name val) <| \_ -> function val) values

Advantages of my version:

• The user is responsible for the test name generation ^{and as such can generate test names as descriptively as they want}
• Can handle inputs more complex than tuples ^{because it's based on records}
• No code duplication ^{as opposed to simply extracting a helper function}

Disadvantages of my version:

• Names have to be generated ^{which tends to look ugly in the code (though not in the test output)}
• requires an helper type alias ^{these can potentially be reused but are still not great}

Example

The tutorial that led me to this point wanted me to write this:

fullAdderTests =
    describe "Full adder"
        [ test "sum and carry-out are 0 when both inputs and carry-in are 0" <|
            \_ ->
                fullAdder 0 0 0
                    |> Expect.equal { carry = 0, sum = 0 }
        , test "sum is 1 and carry-out is 0 when both inputs are 0, but carry-in is 1" <|
            \_ ->
                fullAdder 0 0 1
                    |> Expect.equal { carry = 0, sum = 1 }
        , test "sum is 1 and carry-out is 0 when the 1st input is 0, the 2nd input is 1, and carry-in is 0" <|
            \_ ->
                fullAdder 0 1 0
                    |> Expect.equal { carry = 0, sum = 1 }
        , test "sum is 0 and carry-out is 1 when the 1st input is 0, the 2nd input is 1, and the carry-in is 1" <|
            \_ ->
                fullAdder 0 1 1
                    |> Expect.equal { carry = 1, sum = 0 }
        , test "sum is 1 and carry-out is 0 when the 1st input is 1, the 2nd input is 0, and the carry-in is 0" <|
            \_ ->
                fullAdder 1 0 0
                    |> Expect.equal { carry = 0, sum = 1 }
        , test "sum is 0 and carry-out is 1 when the 1st input is 1, the 2nd input is 0, and the carry-in is 1" <|
            \_ ->
                fullAdder 1 0 1
                    |> Expect.equal { carry = 1, sum = 0 }
        , test "sum is 0 and carry-out is 1 when the 1st input is 1, the 2nd input is 1, and the carry-in is 0" <|
            \_ ->
                fullAdder 1 1 0
                    |> Expect.equal { carry = 1, sum = 0 }
        , test "sum is 1 and carry-out is 1 when the 1st input is 1, the 2nd input is 1, and the carry-in is 1" <|
            \_ ->
                fullAdder 1 1 1
                    |> Expect.equal { carry = 1, sum = 1 }
        ]

which I turned into

type alias FullAdderValues =
    { expected :
        { sum : Int
        , carry : Int
        }
    , a : Int
    , b : Int
    , carryIn : Int
    }

fullAdderBehaviour : Test
fullAdderBehaviour =
    describe "Full adder" <|
        each
            [ FullAdderValues { sum = 0, carry = 0 } 0 0 0
            , FullAdderValues { sum = 1, carry = 0 } 0 0 1
            , FullAdderValues { sum = 1, carry = 0 } 0 1 0
            , FullAdderValues { sum = 0, carry = 1 } 0 1 1
            , FullAdderValues { sum = 1, carry = 0 } 1 0 0
            , FullAdderValues { sum = 0, carry = 1 } 1 0 1
            , FullAdderValues { sum = 0, carry = 1 } 1 1 0
            , FullAdderValues { sum = 1, carry = 1 } 1 1 1
            ]
            (\values ->
                "should output sum="
                    ++ String.fromInt values.expected.sum
                    ++ ", carry="
                    ++ String.fromInt values.expected.carry
                    ++ " for inputs "
                    ++ String.fromInt values.a
                    ++ ", "
                    ++ String.fromInt values.b
                    ++ " and carry-in "
                    ++ String.fromInt values.carryIn
            )
        <|
            \values ->
                fullAdder values.a values.b values.carryIn
                    |> Expect.equal values.expected

And for completions sake, here's with an extracted helper function:

fullAdderBehaviour : Test
fullAdderBehaviour =
    describe "Full adder"
        [ test "should output sum=0, carry=0 for inputs 0, 0 and carry-in 0" <|
            fullAdderTest { sum = 0, carry = 0 } 0 0 0
        , test "should output sum=1, carry=0 for inputs 0, 0 and carry-in 1" <|
            fullAdderTest { sum = 1, carry = 0 } 0 0 1
        , test "should output sum=1, carry=0 for inputs 0, 1 and carry-in 0" <|
            fullAdderTest { sum = 1, carry = 0 } 0 1 0
        , test "should output sum=0, carry=1 for inputs 0, 1 and carry-in 1" <|
            fullAdderTest { sum = 0, carry = 1 } 0 1 1
        , test "should output sum=1, carry=0 for inputs 1, 0 and carry-in 0" <|
            fullAdderTest { sum = 1, carry = 0 } 1 0 0
        , test "should output sum=0, carry=1 for inputs 1, 0 and carry-in 1" <|
            fullAdderTest { sum = 0, carry = 1 } 1 0 1
        , test "should output sum=0, carry=1 for inputs 1, 1 and carry-in 0" <|
            fullAdderTest { sum = 0, carry = 1 } 1 1 0
        , test "should output sum=1, carry=1 for inputs 1, 1 and carry-in 1" <|
            fullAdderTest { sum = 1, carry = 1 } 1 1 1
        ]

fullAdderTest : { sum : Int, carry : Int } -> Int -> Int -> Int -> (() -> Expectation)
fullAdderTest expected a b carryIn =
    \_ ->
        fullAdder a b carryIn
            |> Expect.equal expected