redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.44k stars 580 forks source link

CI Failure (key symptom) in `OMBValidationTest.test_common_workload` #20296

Closed vbotbuildovich closed 1 month ago

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15048

Module: rptest.redpanda_cloud_tests.omb_validation_test
Class: OMBValidationTest
Method: test_common_workload
test_id:    OMBValidationTest.test_common_workload
status:     FAIL
run time:   58.250 seconds

CalledProcessError(1, ['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@cptnlj3kho8umi3mjsm0-agent', 'kubectl', 'get', 'pods', '-n', 'redpanda', '-o', 'json'], '{\n    "apiVersion": "v1",\n    "items": [],\n    "kind": "List",\n    "metadata": {\n        "resourceVersion": ""\n    }\n}\n', 'Unable to connect to the server: dial tcp 34.145.56.100:443: i/o timeout\n\x1b[31mERROR: \x1b[0mProcess exited with status 1\n\n')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 177, in _do_run
    self.test = self.test_context.cls(self.test_context)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 156, in __init__
    super().__init__(test_ctx, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_cloud_test.py", line 25, in __init__
    self.redpanda = make_redpanda_cloud_service(test_context)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 5087, in make_redpanda_cloud_service
    return RedpandaServiceCloud(context,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1700, in __init__
    self.rebuild_pods_classes()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1728, in rebuild_pods_classes
    for p in self.get_redpanda_pods()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1787, in get_redpanda_pods
    pods = json.loads(self.kubectl.cmd('get pods -n redpanda -o json'))
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 276, in cmd
    return self._ssh_cmd(cmd, capture=capture)
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 252, in _ssh_cmd
    return self._local_cmd(local_cmd)
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 232, in _local_cmd
    raise subprocess.CalledProcessError(process.returncode, cmd, s_out,
subprocess.CalledProcessError: Command '['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@cptnlj3kho8umi3mjsm0-agent', 'kubectl', 'get', 'pods', '-n', 'redpanda', '-o', 'json']' returned non-zero exit status 1.

JIRA Link: CORE-4559

michael-redpanda commented 1 month ago

Automatically closing issue to match current state of CORE-4559