redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.39k stars 577 forks source link

CI Failure (key symptom) in `RedpandaCloudSelfTest.test_healthy` #21448

Open vbotbuildovich opened 1 month ago

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15646

Module: rptest.redpanda_cloud_tests.cloud_self_test
Class: RedpandaCloudSelfTest
Method: test_healthy
test_id:    RedpandaCloudSelfTest.test_healthy
status:     FAIL
run time:   131.228 seconds

RuntimeError('Failed to get initial cluster spec')
Traceback (most recent call last):
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 682, in _create_new_cluster
    _cluster_id = self._wait_for_cluster_id(r['id'])
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 620, in _wait_for_cluster_id
    wait_until(lambda: self._cluster_id_updated(uuid),
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Failed to get proper id of cloud cluster rp-ducktape-cluster-4b779d56

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 177, in _do_run
    self.test = self.test_context.cls(self.test_context)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/cloud_self_test.py", line 24, in __init__
    super().__init__(test_context, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_cloud_test.py", line 25, in __init__
    self.redpanda = make_redpanda_cloud_service(test_context)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 5093, in make_redpanda_cloud_service
    return RedpandaServiceCloud(context,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1695, in __init__
    cluster_id = self._cloud_cluster.create(superuser=self._superuser)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 926, in create
    self._create_new_cluster()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 688, in _create_new_cluster
    raise RuntimeError("Failed to get initial cluster spec") from e
RuntimeError: Failed to get initial cluster spec

JIRA Link: CORE-5657

rpdevmp commented 1 month ago

Buildkite CI job that was added by Pandatriage shows: https://buildkite.com/redpanda/vtools/builds/15646

INSERT INTO buildkite_test_analytics

psycopg2.errors.StringDataRightTruncation: value too long for type character varying(500)

rpdevmp commented 1 month ago

Looks like there was an issue with creating this item.. Included error in description doesn't match Buildkite issue..

Going by https://buildkite.com/redpanda/vtools/builds/15646 We now have several PRs to fix this issue: value too long for type character varying(500)

  1. limit produce analytics report per branch https://github.com/redpanda-data/vtools/pull/2984

  2. Update DB schema to use TEXT instead of VARCHAR(500) to avoid char limits https://github.com/redpanda-data/qa-infra/pull/80 https://github.com/redpanda-data/vtools/pull/2994