Open Unisay opened 1 year ago
@Unisay Damn. I think --fail-fast
doesn't interact well with parallel tests. PR welcome
Can you show me the test suite with which this happens?
Sure,
walletSpec :: Spec
walletSpec = describe "Wallet Backend API" $ sequential do
story "Wallet serves network info" testEnvironmentIsReady
story "Created wallet is known" createdWallet
I ran cabal run wallet-e2e -- --fail-fast
15 times without this problem, 16th time it surfaced itself. No changes in between. 🤷🏼♂️
@Unisay I meant the repository. This definitely isn't enough for anyone to reproduce the issue.
My apologies for not making a minimal repro, but at least I can point out at a public repository that reproduces it: https://github.com/cardano-foundation/cardano-wallet/tree/yura/fail-fast-issue/lib/wallet-e2e
The branch name is yura/fail-fast-issue
you'd have to cd lib/wallet-e2e
, make sure that direnv
picks up the nix shell and run the command multiple times until the issue happens; This looks like it:
✘ yura@fractal ~/projects/cardano-wallet/lib/wallet-e2e yura/fail-fast-issue cabal run wallet-e2e -- --fail-fast
Tests:
Wallet Backend API
✗ Wallet serves network info 769.55 ms
Retries: 3 (does not look flaky)
Failures:
src/Cardano/Wallet/Spec/Interpreters/Effectfully.hs:46
✗ 1 Wallet Backend API.Wallet serves network info
Retries: 3 (does not look flaky)
E2E test suite requires a running cardano-wallet instance
connected to a running cardano-node and listenting on "http://localhost:8090/v2"
However, an exception happened when trying to retrieve
network information from the wallet backend:
HttpExceptionRequest Request {
host = "localhost"
port = 8090
secure = False
requestHeaders = []
path = "/v2/network/information"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
proxySecureMode = ProxySecureWithConnect
}
(ConnectionFailure Network.Socket.connect: <socket: 106>: does not exist (Connection refused))
Examples: 3
Passed: 0
Failed: 1
Sum of test runtimes: 0.77 seconds
Test suite took: 0.77 seconds
✘ yura@fractal ~/projects/cardano-wallet/lib/wallet-e2e yura/fail-fast-issue cabal run wallet-e2e -- --fail-fast
Tests:
Wallet Backend API
Passed: 0
Failed: 0
Sum of test runtimes: 0.00 seconds
Test suite took: 0.77 seconds
yura@fractal ~/projects/cardano-wallet/lib/wallet-e2e yura/fail-fast-issue cabal run wallet-e2e -- --fail-fast
Tests:
Wallet Backend API
✗ Wallet serves network info 766.94 ms
Retries: 3 (does not look flaky)
Failures:
src/Cardano/Wallet/Spec/Interpreters/Effectfully.hs:46
✗ 1 Wallet Backend API.Wallet serves network info
Retries: 3 (does not look flaky)
E2E test suite requires a running cardano-wallet instance
connected to a running cardano-node and listenting on "http://localhost:8090/v2"
However, an exception happened when trying to retrieve
network information from the wallet backend:
HttpExceptionRequest Request {
host = "localhost"
port = 8090
secure = False
requestHeaders = []
path = "/v2/network/information"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
proxySecureMode = ProxySecureWithConnect
}
(ConnectionFailure Network.Socket.connect: <socket: 106>: does not exist (Connection refused))
Examples: 3
Passed: 0
Failed: 1
Sum of test runtimes: 0.77 seconds
Test suite took: 0.77 seconds
Here we can observe that
the failure happens again (expected).
The spec is here.
Every once in a while I observe that no tests are executed when using the
--fail-fast
flag.This is non-deterministic, disappears upon retrying and then randomly happens again.