redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.2k stars 562 forks source link

CI Failure (key symptom) in `ManyPartitionsTest.test_many_partitions_compacted` #19904

Open vbotbuildovich opened 3 weeks ago

vbotbuildovich commented 3 weeks ago

https://buildkite.com/redpanda/vtools/builds/14811

Module: rptest.scale_tests.many_partitions_test
Class: ManyPartitionsTest
Method: test_many_partitions_compacted
Arguments: {
    "mib_per_partition": 4,
    "topic_partitions_per_shard": 1000
}
test_id:    ManyPartitionsTest.test_many_partitions_compacted
status:     FAIL
run time:   542.116 seconds

TimeoutError('Waiting for repeater messages')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 730, in test_many_partitions_compacted
    self._test_many_partitions(
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 968, in _test_many_partitions
    self._restart_stress(scale, topic_names, n_partitions,
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 399, in _restart_stress
    inter_restart_check()
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 949, in progress_check
    repeater.await_progress(
  File "/home/ubuntu/redpanda/tests/rptest/services/kgo_repeater_service.py", line 407, in await_progress
    self.redpanda.wait_until(check,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1048, in wait_until
    wait_until(wrapped,
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Waiting for repeater messages

JIRA Link: CORE-4231

vbotbuildovich commented 2 weeks ago

https://buildkite.com/redpanda/vtools/builds/14966 https://buildkite.com/redpanda/vtools/builds/15007

vbotbuildovich commented 1 week ago

https://buildkite.com/redpanda/vtools/builds/15137 https://buildkite.com/redpanda/vtools/builds/15156 https://buildkite.com/redpanda/vtools/builds/15177 https://buildkite.com/redpanda/vtools/builds/15299

vbotbuildovich commented 1 week ago

https://buildkite.com/redpanda/vtools/builds/15137 https://buildkite.com/redpanda/vtools/builds/15156 https://buildkite.com/redpanda/vtools/builds/15177 https://buildkite.com/redpanda/vtools/builds/15299

vbotbuildovich commented 1 week ago

*https://buildkite.com/redpanda/vtools/builds/15384

vbotbuildovich commented 15 hours ago

https://buildkite.com/redpanda/vtools/builds/15562 https://buildkite.com/redpanda/vtools/builds/15577 https://buildkite.com/redpanda/vtools/builds/15600 https://buildkite.com/redpanda/vtools/builds/15612