Intermittant CI failure tracking issue II

I notice that all of the lightning-related test failures seem to happen in cli_test_rust_tests_bitcoind and never cli_test_rust_tests_electrs or cli_test_rust_tests_esplora ...

justinmoon commented 9 months ago

test_gateway_cannot_pay_expired_invoice & test_gateway_client_pay_unpayable_invoice https://github.com/fedimint/fedimint/actions/runs/5789072035/job/15689333467?pr=2911#step:9:7236

elsirion commented 9 months ago

cli_test_latency, cli_test_cli and cli_test_reconnect: https://github.com/fedimint/fedimint/actions/runs/5888250155/job/15969079807

elsirion commented 9 months ago

runs_consensus_if_tx_submitted
~fedimint_migration_tests::test_migrations: This one puzzles me a bit @m1sterc001guy, how could that be flaky?! The only thing I could imagine is that the file system is acting weirdly and not returning outdated data.~ Never mind, dpc already found it #2985.

dpc commented 9 months ago

Note: #2980 should make it much easier to debug failed runs.

elsirion commented 8 months ago

Something got a lot of the code coverage tests stuck here (search for has been).

elsirion commented 8 months ago

~All the backed_tests failed~ maybe not a flake

elsirion commented 8 months ago

Flakiness intensified, a lot of the code coverage tests just get stuck apparently.

bradleystachurski commented 6 months ago

ln-gateway::gatewayd-integration-tests test_gateway_configuration

justinmoon commented 6 months ago

https://github.com/fedimint/fedimint/actions/runs/6788652621/job/18461865546?pr=3547 had peg_out_fail_refund run too long

bradleystachurski commented 5 months ago

FAIL [  36.905s] fedimint-ln-gateway::gatewayd-integration-tests test_gateway_executes_swaps_between_connected_federations

https://github.com/fedimint/fedimint/actions/runs/7013415232/job/19079522681?pr=3746

Same failure in separate PR

https://github.com/fedimint/fedimint/actions/runs/7015530652/job/19085010433?pr=3756

okjodom commented 3 months ago

test_config_api possibly flaky

fedimint-workspace-lcov-ci> SLOW [>420.000s] fedimint-server config::api::tests::test_config_api[1m0 skipped
fedimint-workspace-lcov-ci> TERMINATING [>480.000s] fedimint-server config::api::tests::test_config_api[1m0 skipped
fedimint-workspace-lcov-ci> TIMEOUT [ 480.026s] fedimint-server config::api::tests::test_config_api skipped
fedimint-workspace-lcov-ci> --- STDOUT: fedimint-server config::api::tests::test_config_api --- fedimint-workspace-lcov-ci> running 1 test fedimint-workspace-lcov-ci> test config::api::tests::test_config_api has been running for over 60 seconds fedimint-workspace-lcov-ci> fedimint-workspace-lcov-ci> --- STDERR: fedimint-server config::api::tests::test_config_api --- fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.341974Z INFO consensus: Starting config gen fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.342039Z INFO net::peer::dkg: Created new config gen Api fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.343480Z INFO net::api: Starting api on ws://127.0.0.1:25031 fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.343510Z INFO consensus: Starting config gen fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.343601Z INFO net::peer::dkg: Created new config gen Api fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.343872Z INFO net::api: Starting api on ws://127.0.0.1:25033 fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.343892Z INFO consensus: Starting config gen fedimint-workspace-lcov-ci> 2024-01-22T19:41:15.343925Z INFO net::peer::dkg: Created new config gen Api fedimint-workspace-lcov-ci> 2024-01-22T19:41:16.342903Z INFO test: Test retrying server status fedimint-workspace-lcov-ci> 2024-01-22T19:41:16.345235Z INFO net::peer::dkg: Set password for config gen fedimint-workspace-lcov-ci> 2024-01-22T19:41:16.345542Z WARN net::api: api request error path="set_password" error=ApiError { code: 400, message: "Expected to be in AwaitingPassword state" } fedimint-workspace-lcov-ci> thread 'tokio-runtime-worker' panicked at fedimint-server/src/lib.rs:206:14: fedimint-workspace-lcov-ci> Could not build API server: API name: config-gen 2335 fedimint-workspace-lcov-ci> Caused by: fedimint-workspace-lcov-ci> 0: Bind address: 127.0.0.1:25035 fedimint-workspace-lcov-ci> 1: Networking or low-level protocol error: Address already in use (os error 98) fedimint-workspace-lcov-ci> 2: Address already in use (os error 98)

https://github.com/fedimint/fedimint/actions/runs/7616433363/job/20743174870#step:6:2315

okjodom commented 3 months ago

ci reconnect_test fails on merge queue

bradleystachurski commented 3 months ago

FAIL [   5.062s] fedimint-ln-gateway::gatewayd-integration-tests test_gateway_register_with_federation

https://github.com/fedimint/fedimint/actions/runs/7703671558/job/20994518252?pr=4166#step:6:2124

Timeouts in separate PR (merge queue)

Summary [ 505.141s] 200 tests run: 198 passed (2 slow), 2 timed out, 0 skipped
TIMEOUT [ 480.039s] fedimint-mint-tests::fedimint_mint_tests sends_ecash_out_of_band_cancel
TIMEOUT [ 480.027s] fedimint-server config::api::tests::test_restart_setup

https://github.com/fedimint/fedimint/actions/runs/7702842445/job/20991977876#step:6:38066

bradleystachurski commented 3 months ago

FAIL [   9.208s] fedimint-ln-gateway::gatewayd-integration-tests test_gateway_configuration

https://github.com/fedimint/fedimint/actions/runs/7704054812/job/20995614440?pr=4169#step:6:1835

FAIL [  10.728s] fedimint-ln-gateway::gatewayd-integration-tests test_gateway_register_with_federation

https://github.com/fedimint/fedimint/actions/runs/7835068918/job/21382888265?pr=4267#step:6:2189

fedimint / fedimint

Intermittant CI failure tracking issue II #2825