rukai
commented
1 year ago and another:
redis_int_tests::cluster_handling
shotover 01:50:11.551156Z INFO shotover_proxy::runner: Starting Shotover 0.1.9
shotover 01:50:11.551248Z INFO shotover_proxy::runner: configuration=Config { main_log_level: "info,shotover_proxy=info", observability_interface: "0.0.0.0:9001" }
shotover 01:50:11.551276Z INFO shotover_proxy::runner: topology=Topology { sources: {"redis_prod": Redis(RedisConfig { listen_addr: "127.0.0.1:6379", connection_limit: None, hard_connection_limit: None, tls: None, timeout: None })}, chain_config: {"redis_chain": [RedisSinkCluster(RedisSinkClusterConfig { first_contact_points: ["127.0.0.1:2220", "127.0.0.1:2221", "127.0.0.1:2222", "127.0.0.1:2223", "127.0.0.1:2224", "127.0.0.1:2225"], direct_destination: Some("127.0.0.1:2220"), tls: None, connection_count: None, connect_timeout_ms: 3000 })]}, source_to_chain_mapping: {"redis_prod": "redis_chain"} }
shotover 01:50:11.556594Z INFO shotover_proxy::transforms::redis::sink_cluster: connected to upstream
shotover 01:50:11.556707Z INFO shotover_proxy::config::topology: Loaded chains ["redis_chain"]
shotover 01:50:11.556761Z INFO shotover_proxy::sources::redis: Starting Redis source on [127.0.0.1:6379]
shotover 01:50:11.556920Z INFO shotover_proxy::server: accepting inbound connections
shotover 01:50:11.557192Z INFO shotover_proxy::config::topology: Loaded sources [["redis_prod"]] and linked to chains
thread 'redis_int_tests::cluster_handling' panicked at 'called `Result::unwrap()` on an `Err` value: An error was signalled by the server: The specified node is not a master', shotover-proxy/tests/redis_int_tests/basic_driver_tests.rs:1005:10
stack backtrace:
0: rust_begin_unwind
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
2: core::result::unwrap_failed
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1790:5
3: core::result::Result<T,E>::unwrap
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1112:23
4: lib::redis_int_tests::basic_driver_tests::is_cluster_replicas_ready::{{closure}}
at ./tests/redis_int_tests/basic_driver_tests.rs:1000:15
5: lib::redis_int_tests::basic_driver_tests::test_cluster_ports_rewrite_nodes::{{closure}}
at ./tests/redis_int_tests/basic_driver_tests.rs:1027:61
6: lib::redis_int_tests::cluster_handling::{{closure}}::{{closure}}
at ./tests/redis_int_tests/mod.rs:256:55
7: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/future/future.rs:125:9
8: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:63
9: tokio::runtime::coop::with_budget
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
10: tokio::runtime::coop::budget
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
11: tokio::runtime::park::CachedParkThread::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:31
12: tokio::runtime::context::BlockingRegionGuard::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/context.rs:315:13
13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
14: tokio::runtime::runtime::Runtime::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/runtime.rs:284:45
15: lib::redis_int_tests::cluster_handling::{{closure}}
at ./tests/redis_int_tests/mod.rs:258:5
16: core::ops::function::FnOnce::call_once
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
17: serial_test::serial_code_lock::local_serial_core
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/serial_test-1.0.0/src/serial_code_lock.rs:25:5
18: lib::redis_int_tests::cluster_handling
at ./tests/redis_int_tests/mod.rs:240:1
19: lib::redis_int_tests::cluster_handling::{{closure}}
at ./tests/redis_int_tests/mod.rs:241:29
20: core::ops::function::FnOnce::call_once
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
21: core::ops::function::FnOnce::call_once
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test redis_int_tests::cluster_handling ... FAILEDand hit again:
shotover 03:15:58.426453Z INFO shotover_proxy::runner: Starting Shotover 0.1.9
shotover 03:15:58.426541Z INFO shotover_proxy::runner: configuration=Config { main_log_level: "info,shotover_proxy=info", observability_interface: "0.0.0.0:9001" }
shotover 03:15:58.426568Z INFO shotover_proxy::runner: topology=Topology { sources: {"redis_prod": Redis(RedisConfig { listen_addr: "127.0.0.1:6379", connection_limit: None, hard_connection_limit: None, tls: None, timeout: None })}, chain_config: {"redis_chain": [RedisSinkCluster(RedisSinkClusterConfig { first_contact_points: ["127.0.0.1:2220", "127.0.0.1:2221", "127.0.0.1:2222", "127.0.0.1:2223", "127.0.0.1:2224", "127.0.0.1:2225"], direct_destination: Some("127.0.0.1:2220"), tls: None, connection_count: None, connect_timeout_ms: 3000 })]}, source_to_chain_mapping: {"redis_prod": "redis_chain"} }
shotover 03:15:58.432166Z INFO shotover_proxy::transforms::redis::sink_cluster: connected to upstream
shotover 03:15:58.432287Z INFO shotover_proxy::config::topology: Loaded chains ["redis_chain"]
shotover 03:15:58.432341Z INFO shotover_proxy::sources::redis: Starting Redis source on [127.0.0.1:6379]
shotover 03:15:58.432459Z INFO shotover_proxy::config::topology: Loaded sources [["redis_prod"]] and linked to chains
shotover 03:15:58.433101Z INFO shotover_proxy::server: accepting inbound connections
thread 'redis_int_tests::cluster_handling' panicked at 'called `Result::unwrap()` on an `Err` value: An error was signalled by the server: The specified node is not a master', shotover-proxy/tests/redis_int_tests/basic_driver_tests.rs:1008:10
stack backtrace:
0: rust_begin_unwind
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
2: core::result::unwrap_failed
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1790:5
3: core::result::Result<T,E>::unwrap
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1112:23
4: lib::redis_int_tests::basic_driver_tests::is_cluster_replicas_ready::{{closure}}
at ./tests/redis_int_tests/basic_driver_tests.rs:1003:15
5: lib::redis_int_tests::basic_driver_tests::test_cluster_ports_rewrite_nodes::{{closure}}
at ./tests/redis_int_tests/basic_driver_tests.rs:1030:61
6: lib::redis_int_tests::cluster_handling::{{closure}}::{{closure}}
at ./tests/redis_int_tests/mod.rs:256:55
7: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/future/future.rs:125:9
8: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:63
9: tokio::runtime::coop::with_budget
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:102:5
10: tokio::runtime::coop::budget
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/coop.rs:68:5
11: tokio::runtime::park::CachedParkThread::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/park.rs:283:31
12: tokio::runtime::context::BlockingRegionGuard::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/context.rs:315:13
13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
14: tokio::runtime::runtime::Runtime::block_on
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.25.0/src/runtime/runtime.rs:284:45
15: lib::redis_int_tests::cluster_handling::{{closure}}
at ./tests/redis_int_tests/mod.rs:258:5
16: core::ops::function::FnOnce::call_once
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
17: serial_test::serial_code_lock::local_serial_core
at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/serial_test-1.0.0/src/serial_code_lock.rs:25:5
18: lib::redis_int_tests::cluster_handling
at ./tests/redis_int_tests/mod.rs:240:1
19: lib::redis_int_tests::cluster_handling::{{closure}}
at ./tests/redis_int_tests/mod.rs:241:29
20: core::ops::function::FnOnce::call_once
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
21: core::ops::function::FnOnce::call_once
at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test redis_int_tests::cluster_handling ... FAILED
justinweng-instaclustr
Intermittent test failures are now occurring pretty commonly, time to do some investigation and flush them out.
cassandra_int_tests::cluster_single_rack_v4::case_2_datastax
Oh oops, I wasnt thinking and overwrote this with a manually induced test failure.
## redis_int_tests::multiThis should be fixed by: https://github.com/shotover/shotover-proxy/pull/1080