redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.12k stars 556 forks source link

CI Failure (key symptom) in `RandomNodeOperationsTest.test_node_operations` #19825

Open vbotbuildovich opened 2 weeks ago

vbotbuildovich commented 2 weeks ago

https://buildkite.com/redpanda/redpanda/builds/50144

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 0,
    "enable_failures": false,
    "with_tiered_storage": true
}
test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   362.410 seconds

AssertionError()
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/random_node_operations_test.py", line 483, in test_node_operations
    regular_producer_consumer.verify()
  File "/root/tests/rptest/tests/random_node_operations_test.py", line 262, in verify
    self.consumer.wait()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/service.py", line 287, in wait
    if not self.wait_node(node, end - now):
  File "/root/tests/rptest/services/kgo_verifier_services.py", line 251, in wait_node
    return self._do_wait_node(node, timeout_sec)
  File "/root/tests/rptest/services/kgo_verifier_services.py", line 287, in _do_wait_node
    self._redpanda.wait_until(
  File "/root/tests/rptest/services/redpanda.py", line 1048, in wait_until
    wait_until(wrapped,
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 53, in wait_until
    raise e
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 44, in wait_until
    if condition():
  File "/root/tests/rptest/services/redpanda.py", line 1044, in wrapped
    assert self.all_up() or getattr(self, '_tolerate_crashes',
AssertionError

JIRA Link: CORE-4185