search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.12k stars 556 forks source link

CI Failure (key symptom) in `ManyPartitionsTest.test_many_partitions_compacted` #19828

Open vbotbuildovich opened 2 weeks ago

vbotbuildovich commented 2 weeks ago

https://buildkite.com/redpanda/vtools/builds/14609 https://buildkite.com/redpanda/vtools/builds/14624

Module: rptest.scale_tests.many_partitions_test
Class: ManyPartitionsTest
Method: test_many_partitions_compacted
Arguments: {
    "mib_per_partition": 4,
    "topic_partitions_per_shard": 1000
}
test_id:    ManyPartitionsTest.test_many_partitions_compacted
status:     FAIL
run time:   586.460 seconds

TimeoutError('Waiting for traffic soak')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 730, in test_many_partitions_compacted
    self._test_many_partitions(
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 987, in _test_many_partitions
    repeater.await_progress(soak_await_msgs,
  File "/home/ubuntu/redpanda/tests/rptest/services/kgo_repeater_service.py", line 407, in await_progress
    self.redpanda.wait_until(check,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1048, in wait_until
    wait_until(wrapped,
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Waiting for traffic soak

JIRA Link: CORE-4187

vbotbuildovich commented 1 week ago

https://buildkite.com/redpanda/vtools/builds/14692 https://buildkite.com/redpanda/vtools/builds/14660 https://buildkite.com/redpanda/vtools/builds/14734 https://buildkite.com/redpanda/vtools/builds/14709 https://buildkite.com/redpanda/vtools/builds/14753 https://buildkite.com/redpanda/vtools/builds/14748 https://buildkite.com/redpanda/vtools/builds/14762 https://buildkite.com/redpanda/vtools/builds/14768 https://buildkite.com/redpanda/vtools/builds/14778 https://buildkite.com/redpanda/vtools/builds/14793 https://buildkite.com/redpanda/vtools/builds/14847 https://buildkite.com/redpanda/vtools/builds/14865

vbotbuildovich commented 12 hours ago

https://buildkite.com/redpanda/vtools/builds/14881 https://buildkite.com/redpanda/vtools/builds/14930 https://buildkite.com/redpanda/vtools/builds/14952 https://buildkite.com/redpanda/vtools/builds/14975 https://buildkite.com/redpanda/vtools/builds/14983 https://buildkite.com/redpanda/vtools/builds/14990 https://buildkite.com/redpanda/vtools/builds/14997 https://buildkite.com/redpanda/vtools/builds/15021 https://buildkite.com/redpanda/vtools/builds/15023 https://buildkite.com/redpanda/vtools/builds/15041 https://buildkite.com/redpanda/vtools/builds/15081 https://buildkite.com/redpanda/vtools/builds/15057

vbotbuildovich commented 13 minutes ago

*https://buildkite.com/redpanda/vtools/builds/15094