UnkindPartition / tasty

Modern and extensible testing framework for Haskell
639 stars 108 forks source link

`sequentialTestGroup` breaks `pattern` #411

Closed newhoggy closed 6 months ago

newhoggy commented 6 months ago

For the following test tree:

tests :: IO TestTree
tests = do
  pure $ T.testGroup "T"
    [ T.testGroup "A"
      [ T.sequentialTestGroup "a" T.AllFinish
        [ emptyTest "1"
        , emptyTest "2"
        ]
      , T.sequentialTestGroup "b" T.AllFinish
        [ emptyTest "1"
        , emptyTest "2"
        ]
      ]
    , T.testGroup "B"
      [ T.sequentialTestGroup "a" T.AllFinish
        [ emptyTest "1"
        , emptyTest "2"
        ]
      , T.sequentialTestGroup "b" T.AllFinish
        [ emptyTest "1"
        , emptyTest "2"
        ]
      ]
    ]

emptyTest :: T.TestName -> TestTree
emptyTest name = T.testCase name (pure ())

I try to filter by pattern:

$ tests --list-tests --pattern /1/
T.A.a.1
T.A.b.1
T.B.a.1
T.B.b.1

Looks good. But then I run a second pattern:

$ tests --list-tests --pattern /2/
T.A.a.1
T.A.a.2
T.A.b.1
T.A.b.2
T.B.a.1
T.B.a.2
T.B.b.1
T.B.b.2

Even though my pattern only specifies tests that contain 2, tests that include 1 are included.

Judging from the behaviour, I think it's because sequentialTestGroup sets up a dependency of each child test tree on its preceding sibling.

Whilst this way of sequentialising tests works for running tests, it breaks the way tests are selected with --pattern causing more tests to run than necessary.

newhoggy commented 6 months ago

I see it's a known behaviour https://github.com/UnkindPartition/tasty?tab=readme-ov-file#dependencies