`localOption` is ignored by `defaultMain`

[andreasabel]

I am trying to set a local option for a test but it seems to be ignored by the test runner (defaultMain) of tasty-silver.

#!/usr/bin/env GHC_ENVIRONMENT=- runhaskell -packagetasty-1.4.2 -packagetasty-silver-3.2.2

{-# LANGUAGE OverloadedStrings #-}

import Test.Tasty (TestTree, localOption)
import Test.Tasty.Silver
import Test.Tasty.Silver.Advanced
import Test.Tasty.Silver.Interactive as Silver

main :: IO ()
main = Silver.defaultMain $ localOption (Interactive true) testGolden
  where
  true = error "This error should be triggered!"

testGolden :: TestTree
testGolden =
  goldenTest1
    "wrongOutput"
    (return $ Just "golden value")
    (return "actual value")
    (DiffText Nothing)   -- always fail
    ShowText
    (const $ return ())  -- keep the golden file no matter what

When applying and then querying the option, this script should crash with error. But it simply runs the test non-interactively.

@phile314: You implemented the test runner. Where should PlusTestOption be usually handled?

[phile314]

The actual test running is done by launchTestTree from tasty: https://github.com/phile314/tasty-silver/blob/e910c06eab18e07318dad124333de9b24e08353b/Test/Tasty/Silver/Interactive.hs#L159 However, the tasty-silver code does some slightly "hackish" stuff to be able to customize running golden tests in interactive mode. I suspect that code does interfere somehow with the options. https://github.com/phile314/tasty-silver/blob/e910c06eab18e07318dad124333de9b24e08353b/Test/Tasty/Silver/Interactive/Run.hs#L32

For a normal tasty test case (e.g. hspec, hunit, ...), does this option cause the crash? What happens if you use the tasty default test runner?

EDIT: I think the interactive option is only checked at the top-level: https://github.com/phile314/tasty-silver/blob/e910c06eab18e07318dad124333de9b24e08353b/Test/Tasty/Silver/Interactive.hs#L325

[phile314]

For some more context, the basic idea was to have the interactive runner for the developer and the non-interactive for e.g. CI build servers or users who just want to run the test suite. The interactive runner assumes that e.g. git diff is available, whereas the non-interactive runner has no external dependencies. I think having the non-interactive runner suddenly pop up an interactive question would be rather strange. The other way round, using the interactive runner but reverting to non-interactive mode for certain test cases sounds more feasible and useful. It might still require some thinking to get the output to align nicely.

[andreasabel]

My aim is to test the interactive mode also in CI, esp. since there were problems on the Windows side. My current take is to assume answer "y" when run in interactive mode but stdin isn't connected to a terminal that could request interactive input from the user. This way, I can cover a lot of code from the interactive mode on CI as well. The easiest way to get this into the testsuite was (after trying other ways like calling runInteractiveTests directly) to pass option Interactive True to the desired tests. Alas, the option is ignored.

In general, I think that tasty-silver should preserve as much of the functionalities of tasty and tasty-golden as possible. It would have thus made sense to start with their testsuites, to not accidentially break features. Maybe it is not too late to adopt their testsuites, but it is of course easier to fix breakage while developing a fork, than to retroactively find out what broke tests.