search

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.26k stars 337 forks source link

Flaky cli tests #4182

Open PSeitz opened 11 months ago

PSeitz commented 11 months ago

Executed cargo test

test test_delete_index_cli ... FAILED
test test_cmd_search_aggregation ... FAILED
test test_all_local_index ... FAILED
test test_ingest_docs_cli ... FAILED
test test_garbage_collect_index_cli ... FAILED
test test_garbage_collect_cli_no_grace ... FAILED

failures:

---- test_delete_index_cli stdout ----
thread 'test_delete_index_cli' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Failed to connect to `127.0.0.1:40785` failed, retrying 1/10
Failed to connect to `127.0.0.1:40785` failed, retrying 2/10
Failed to connect to `127.0.0.1:40785` failed, retrying 3/10
Failed to connect to `127.0.0.1:40785` failed, retrying 4/10
Failed to connect to `127.0.0.1:40785` failed, retrying 5/10
Failed to connect to `127.0.0.1:40785` failed, retrying 6/10
Failed to connect to `127.0.0.1:40785` failed, retrying 7/10
Failed to connect to `127.0.0.1:40785` failed, retrying 8/10
Failed to connect to `127.0.0.1:40785` failed, retrying 9/10
Failed to connect to `127.0.0.1:40785` failed, retrying 10/10
thread 'test_delete_index_cli' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:40785`', quickwit-cli/tests/cli.rs:608:35

---- test_cmd_search_aggregation stdout ----
thread 'test_cmd_search_aggregation' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:43953` failed, retrying 1/10
Failed to connect to `127.0.0.1:43953` failed, retrying 2/10
Failed to connect to `127.0.0.1:43953` failed, retrying 3/10
Failed to connect to `127.0.0.1:43953` failed, retrying 4/10
Failed to connect to `127.0.0.1:43953` failed, retrying 5/10
Failed to connect to `127.0.0.1:43953` failed, retrying 6/10
Failed to connect to `127.0.0.1:43953` failed, retrying 7/10
Failed to connect to `127.0.0.1:43953` failed, retrying 8/10
Failed to connect to `127.0.0.1:43953` failed, retrying 9/10
Failed to connect to `127.0.0.1:43953` failed, retrying 10/10
thread 'test_cmd_search_aggregation' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:43953`', quickwit-cli/tests/cli.rs:343:35

---- test_all_local_index stdout ----
thread 'test_all_local_index' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:40221` failed, retrying 1/10
Failed to connect to `127.0.0.1:40221` failed, retrying 2/10
Failed to connect to `127.0.0.1:40221` failed, retrying 3/10
Failed to connect to `127.0.0.1:40221` failed, retrying 4/10
Failed to connect to `127.0.0.1:40221` failed, retrying 5/10
Failed to connect to `127.0.0.1:40221` failed, retrying 6/10
Failed to connect to `127.0.0.1:40221` failed, retrying 7/10
Failed to connect to `127.0.0.1:40221` failed, retrying 8/10
Failed to connect to `127.0.0.1:40221` failed, retrying 9/10
Failed to connect to `127.0.0.1:40221` failed, retrying 10/10
thread 'test_all_local_index' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:40221`', quickwit-cli/tests/cli.rs:891:35

---- test_ingest_docs_cli stdout ----
thread 'test_ingest_docs_cli' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:44289` failed, retrying 1/10
Failed to connect to `127.0.0.1:44289` failed, retrying 2/10
Failed to connect to `127.0.0.1:44289` failed, retrying 3/10
Failed to connect to `127.0.0.1:44289` failed, retrying 4/10
Failed to connect to `127.0.0.1:44289` failed, retrying 5/10
Failed to connect to `127.0.0.1:44289` failed, retrying 6/10
Failed to connect to `127.0.0.1:44289` failed, retrying 7/10
Failed to connect to `127.0.0.1:44289` failed, retrying 8/10
Failed to connect to `127.0.0.1:44289` failed, retrying 9/10
Failed to connect to `127.0.0.1:44289` failed, retrying 10/10
thread 'test_ingest_docs_cli' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:44289`', quickwit-cli/tests/cli.rs:236:35

---- test_garbage_collect_index_cli stdout ----
thread 'test_garbage_collect_index_cli' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:34349` failed, retrying 1/10
Failed to connect to `127.0.0.1:34349` failed, retrying 2/10
Failed to connect to `127.0.0.1:34349` failed, retrying 3/10
Failed to connect to `127.0.0.1:34349` failed, retrying 4/10
Failed to connect to `127.0.0.1:34349` failed, retrying 5/10
Failed to connect to `127.0.0.1:34349` failed, retrying 6/10
Failed to connect to `127.0.0.1:34349` failed, retrying 7/10
Failed to connect to `127.0.0.1:34349` failed, retrying 8/10
Failed to connect to `127.0.0.1:34349` failed, retrying 9/10
Failed to connect to `127.0.0.1:34349` failed, retrying 10/10
thread 'test_garbage_collect_index_cli' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:34349`', quickwit-cli/tests/cli.rs:748:35

---- test_garbage_collect_cli_no_grace stdout ----
thread 'test_garbage_collect_cli_no_grace' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:36301` failed, retrying 1/10
Failed to connect to `127.0.0.1:36301` failed, retrying 2/10
Failed to connect to `127.0.0.1:36301` failed, retrying 3/10
Failed to connect to `127.0.0.1:36301` failed, retrying 4/10
Failed to connect to `127.0.0.1:36301` failed, retrying 5/10
Failed to connect to `127.0.0.1:36301` failed, retrying 6/10
Failed to connect to `127.0.0.1:36301` failed, retrying 7/10
Failed to connect to `127.0.0.1:36301` failed, retrying 8/10
Failed to connect to `127.0.0.1:36301` failed, retrying 9/10
Failed to connect to `127.0.0.1:36301` failed, retrying 10/10
thread 'test_garbage_collect_cli_no_grace' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:36301`', quickwit-cli/tests/cli.rs:637:35

failures:
    test_all_local_index
    test_cmd_search_aggregation
    test_delete_index_cli
    test_garbage_collect_cli_no_grace
    test_garbage_collect_index_cli
    test_ingest_docs_cli

test result: FAILED. 10 passed; 6 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.82s

error: test failed, to rerun pass `-p quickwit-cli --test cli`
fulmicoton commented 11 months ago

It would be nice to fix this of course, but could disable telemetry in your env variables? We should also disable it on debug mode probably?

PSeitz commented 11 months ago

It's still flaky with telemetry disabled

thread 'test_ingest_docs_cli' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `1`', quickwit-cli/tests/cli.rs:261:5

failures:
    test_ingest_docs_cli

test result: FAILED. 15 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 3.14s

error: test failed, to rerun pass `-p quickwit-cli --test cli`