Add concept of "pending" tests

[s9gf4ult]

Very useful concept when you have semi-ready code, and some of tests are expectedly broken, and you don't want to see them as "failed".

There is such a thing in hspec framework and it works well.

[adamgundry]

I agree this would be useful. I've actually hacked together a version that can be defined in downstream code, though making it part of the core could presumably lead to a nicer implementation that reports pending tests differently:

-- | Tasty test provider for pending tests, where the desired result
-- is the negation of the current one.  This causes a note to be made
-- in the test log.
newtype PendingTest t = PendingTest t
  deriving Typeable

instance IsTest t => IsTest (PendingTest t) where
  run os (PendingTest t) c = do
      r <- run os t c `catch` (\ e -> return $ testFailed $ show (e :: SomeException))
      let d = "(PENDING) " ++ resultDescription r
      return $ if resultSuccessful r && lookupOption os == StrictPending
               then testFailed d
               else testPassed d
  testOptions = return [Option (return LaxPending)]

-- | How to treat pending tests where the underlying test succeeds
data PendingLaxity = StrictPending -- ^ Treat as a test failure (the default)
                   | LaxPending    -- ^ Treat as a success, so pending tests always pass
  deriving (Eq, Ord, Typeable)

instance IsOption PendingLaxity where
  defaultValue = StrictPending
  parseValue s = case s of
                   "strict" -> Just StrictPending
                   "lax"    -> Just LaxPending
                   _        -> Nothing
  optionName   = return "pending"
  optionHelp   = return "Use 'lax' to allow successful pending tests to pass; 'strict' is the default"

-- | Mark all the tests in a tree as pending
pendingTestTree :: TestTree -> TestTree
pendingTestTree (SingleTest n t)      = SingleTest n (PendingTest t)
pendingTestTree (TestGroup n tts)     = TestGroup n (map pendingTestTree tts)
pendingTestTree (PlusTestOptions f t) = PlusTestOptions f (pendingTestTree t)
pendingTestTree (WithResource rs t)   = WithResource rs (pendingTestTree . t)
pendingTestTree (AskOptions t)        = AskOptions (pendingTestTree . t)
#if MIN_VERSION_tasty(1,2,0)
pendingTestTree (After d e t)         = After d e (pendingTestTree t)
#endif

-- | Create a pending HUnit test case
pendingTestCase :: TestName -> Assertion -> TestTree
pendingTestCase name = pendingTestTree . testCase name

[Bodigrim]

Tasty design principle is that everything which can be implemented outside of core should not be part of core.

[michaelpj]

For any future people: https://hackage.haskell.org/package/tasty-expected-failure exists and works okay.

[Bodigrim]

I'm inclined to close this as won't do.