CI Failure (key symptom) in `HighThroughputTest.test_add_and_decommission`

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15105

Module: rptest.redpanda_cloud_tests.high_throughput_test
Class: HighThroughputTest
Method: test_add_and_decommission

test_id:    HighThroughputTest.test_add_and_decommission
status:     FAIL
run time:   3817.710 seconds

TimeoutError('Unable to deterimine readiness of cloud cluster rp-ducktape-cluster-befb2a9a; last state creating_agent')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 177, in _do_run
    self.test = self.test_context.cls(self.test_context)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/high_throughput_test.py", line 242, in __init__
    super(HighThroughputTest, self).__init__(test_context=test_ctx,
  File "/home/ubuntu/redpanda/tests/rptest/utils/test_mixins.py", line 29, in __init__
    super().__init__(**kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_cloud_test.py", line 25, in __init__
    self.redpanda = make_redpanda_cloud_service(test_context)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 5087, in make_redpanda_cloud_service
    return RedpandaServiceCloud(context,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1689, in __init__
    cluster_id = self._cloud_cluster.create(superuser=self._superuser)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 925, in create
    self._create_new_cluster()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 715, in _create_new_cluster
    wait_until(lambda: self._cluster_ready(),
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Unable to deterimine readiness of cloud cluster rp-ducktape-cluster-befb2a9a; last state creating_agent

JIRA Link: CORE-5038

vbotbuildovich commented 3 weeks ago

*https://buildkite.com/redpanda/vtools/builds/15105

vbotbuildovich commented 3 weeks ago

https://buildkite.com/redpanda/vtools/builds/15491 https://buildkite.com/redpanda/vtools/builds/15571

vbotbuildovich commented 2 weeks ago

*https://buildkite.com/redpanda/vtools/builds/15653

vbotbuildovich commented 2 weeks ago

*https://buildkite.com/redpanda/vtools/builds/15724

vbotbuildovich commented 2 weeks ago

*https://buildkite.com/redpanda/vtools/builds/15771

vbotbuildovich commented 1 week ago