test filter by name evals all tests somehow

[yaitskov]

Hi,

There is a project with lots of test trees. Some of them depend on external environment (sql server e.g.). I add another test tree, but it doesn't require anything like SQL server and I am trying to run just my new test tree, but I get error about SQL server not available.

cabal test  --test-options "-p /asdfasdfasdfasdfasdfs/"

Running 1 test suites...
Test suite spec: RUNNING...
spec: libpq: failed (FATAL:  database "app_dev" does not exist
)

Test suite spec: FAIL
Test suite logged to:
/home/dan/pro/napkin/dist-newstyle/build/x86_64-linux/ghc-8.10.6/app-0.3.8/t/spec/test/app-0.3.8-spec.log

test app tries to connect to db even none of test matches filter pattern. If I comment specific test tree inclusion then the error disappears.

appTestTree :: IO TestTree
appTestTree = do
  pure $
    testGroup
      "app"
      [
        -- Sql.testTree,
        Common.testTree
      ]

Is there any workaround to make test evaluation lazy?

[UnkindPartition]

Tasty doesn't run the tests that are filtered out.

Are you using tasty-hspec? That can lead to unexpected behavior due to various mismatches between tasty's and hspec's models of what a test or a test tree is (or possibly because of the way your spec is structured, like if you have some IO outside of it — I know that causes problems even in pure hspec, without tasty). You might want to raise this with tasty-hspec maintainers (and maybe give them more details about your tests). Or just write your tests using tasty-hunit.

Tasty is more restrictive w.r.t. where you can do IO than hspec precisely so that these things don't happen, but tasty-hspec has to circumvent that to accommodate hspec.