search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.08k stars 554 forks source link

CI Failure (key symptom) in `SimpleSelfTest.test_cloud` #18729

Open vbotbuildovich opened 2 weeks ago

vbotbuildovich commented 2 weeks ago

https://buildkite.com/redpanda/vtools/builds/14177

Module: rptest.tests.services_self_test
Class: SimpleSelfTest
Method: test_cloud
test_id:    SimpleSelfTest.test_cloud
status:     FAIL
run time:   22.999 seconds

CalledProcessError(1, ['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@cpc0mb06mfavf68d5ir0-agent', './breakglass-tools.sh'], '', '\x1b[31mERROR: \x1b[0mfailed connecting to host cpc0mb06mfavf68d5ir0-agent:0: failed to receive cluster details response\n\tfailed to dial target host\n\tTeleport proxy failed to connect to "node" agent "@local-node" over reverse tunnel:\n\n  no tunnel connection found: no node reverse tunnel for d289877f-b34a-43e4-a7bf-58454c76b97a.proxy.tp.redpanda.com found\n\nThis usually means that the agent is offline or has disconnected. Check the\nagent logs and, if the issue persists, try restarting it or re-registering it\nwith the cluster.\n\n')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 177, in _do_run
    self.test = self.test_context.cls(self.test_context)
  File "/home/ubuntu/redpanda/tests/rptest/tests/services_self_test.py", line 286, in __init__
    self.redpanda = make_redpanda_mixed_service(test_context,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 4988, in make_redpanda_mixed_service
    return make_redpanda_cloud_service(context, min_brokers=min_brokers)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 4967, in make_redpanda_cloud_service
    return RedpandaServiceCloud(context,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1649, in __init__
    self.rebuild_pods_classes()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1677, in rebuild_pods_classes
    for p in self.get_redpanda_pods()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1736, in get_redpanda_pods
    pods = json.loads(self.kubectl.cmd('get pods -n redpanda -o json'))
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 252, in cmd
    self._install()
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 122, in _install
    self._ssh_cmd(['./breakglass-tools.sh'])
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 235, in _ssh_cmd
    return self._local_cmd(local_cmd)
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 215, in _local_cmd
    raise subprocess.CalledProcessError(process.returncode, cmd, s_out,
subprocess.CalledProcessError: Command '['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@cpc0mb06mfavf68d5ir0-agent', './breakglass-tools.sh']' returned non-zero exit status 1.

JIRA Link: CORE-3142

vbotbuildovich commented 1 week ago

*https://buildkite.com/redpanda/vtools/builds/14407