0 tests is executed sporadically when `--fail-fast` flag is on.

[Unisay]

Every once in a while I observe that no tests are executed when using the --fail-fast flag.

cabal run my-app -- --fail-fast
Tests:

My Awesome Test Suite 

  Passed:                       0
  Failed:                       0
  Sum of test runtimes:         0.00 seconds
  Test suite took:              0.77 seconds

This is non-deterministic, disappears upon retrying and then randomly happens again.

[NorfairKing]

@Unisay Damn. I think --fail-fast doesn't interact well with parallel tests. PR welcome Can you show me the test suite with which this happens?

[Unisay]

Sure,

walletSpec :: Spec
walletSpec = describe "Wallet Backend API" $ sequential do
    story "Wallet serves network info" testEnvironmentIsReady
    story "Created wallet is known" createdWallet

I ran cabal run wallet-e2e -- --fail-fast 15 times without this problem, 16th time it surfaced itself. No changes in between. 🤷🏼‍♂️

[NorfairKing]

@Unisay I meant the repository. This definitely isn't enough for anyone to reproduce the issue.

[Unisay]

My apologies for not making a minimal repro, but at least I can point out at a public repository that reproduces it: https://github.com/cardano-foundation/cardano-wallet/tree/yura/fail-fast-issue/lib/wallet-e2e

The branch name is yura/fail-fast-issue

you'd have to cd lib/wallet-e2e, make sure that direnv picks up the nix shell and run the command multiple times until the issue happens; This looks like it:

 ✘ yura@fractal  ~/projects/cardano-wallet/lib/wallet-e2e   yura/fail-fast-issue  cabal run wallet-e2e -- --fail-fast
Tests:

Wallet Backend API
  ✗ Wallet serves network info                                           769.55 ms
    Retries: 3 (does not look flaky)

Failures:

    src/Cardano/Wallet/Spec/Interpreters/Effectfully.hs:46
  ✗ 1 Wallet Backend API.Wallet serves network info
      Retries: 3 (does not look flaky)
      E2E test suite requires a running cardano-wallet instance 
      connected to a running cardano-node and listenting on "http://localhost:8090/v2"

      However, an exception happened when trying to retrieve 
      network information from the wallet backend: 

      HttpExceptionRequest Request {
        host                 = "localhost"
        port                 = 8090
        secure               = False
        requestHeaders       = []
        path                 = "/v2/network/information"
        queryString          = ""
        method               = "GET"
        proxy                = Nothing
        rawBody              = False
        redirectCount        = 10
        responseTimeout      = ResponseTimeoutDefault
        requestVersion       = HTTP/1.1
        proxySecureMode      = ProxySecureWithConnect
      }
       (ConnectionFailure Network.Socket.connect: <socket: 106>: does not exist (Connection refused))

  Examples:                     3
  Passed:                       0
  Failed:                       1
  Sum of test runtimes:         0.77 seconds
  Test suite took:              0.77 seconds

 ✘ yura@fractal  ~/projects/cardano-wallet/lib/wallet-e2e   yura/fail-fast-issue  cabal run wallet-e2e -- --fail-fast
Tests:

Wallet Backend API

  Passed:                       0
  Failed:                       0
  Sum of test runtimes:         0.00 seconds
  Test suite took:              0.77 seconds

 yura@fractal  ~/projects/cardano-wallet/lib/wallet-e2e   yura/fail-fast-issue  cabal run wallet-e2e -- --fail-fast
Tests:

Wallet Backend API
  ✗ Wallet serves network info                                           766.94 ms
    Retries: 3 (does not look flaky)

Failures:

    src/Cardano/Wallet/Spec/Interpreters/Effectfully.hs:46
  ✗ 1 Wallet Backend API.Wallet serves network info
      Retries: 3 (does not look flaky)
      E2E test suite requires a running cardano-wallet instance 
      connected to a running cardano-node and listenting on "http://localhost:8090/v2"

      However, an exception happened when trying to retrieve 
      network information from the wallet backend: 

      HttpExceptionRequest Request {
        host                 = "localhost"
        port                 = 8090
        secure               = False
        requestHeaders       = []
        path                 = "/v2/network/information"
        queryString          = ""
        method               = "GET"
        proxy                = Nothing
        rawBody              = False
        redirectCount        = 10
        responseTimeout      = ResponseTimeoutDefault
        requestVersion       = HTTP/1.1
        proxySecureMode      = ProxySecureWithConnect
      }
       (ConnectionFailure Network.Socket.connect: <socket: 106>: does not exist (Connection refused))

  Examples:                     3
  Passed:                       0
  Failed:                       1
  Sum of test runtimes:         0.77 seconds
  Test suite took:              0.77 seconds

Here we can observe that

1. a failure happens (as expected)
2. 0 tests discovered (unexpected)

3. the failure happens again (expected).

   The spec is here.