redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.72k stars 591 forks source link

CI Failure (key symptom) in `RollingRestartTest.test_rolling_restart` #20160

Closed vbotbuildovich closed 2 months ago

vbotbuildovich commented 5 months ago

https://buildkite.com/redpanda/vtools/builds/15055

Module: rptest.redpanda_cloud_tests.rolling_restart_test
Class: RollingRestartTest
Method: test_rolling_restart
test_id:    RollingRestartTest.test_rolling_restart
status:     FAIL
run time:   964.626 seconds

CalledProcessError(1, ['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@cptqbj965nolbbtg7t00-agent', 'kubectl', 'cp', 'pod_log_extract.sh', 'ip-10-1-7-108.us-west-2.compute.internal-pshell:/tmp/pod_log_extract.sh'])
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 177, in _do_run
    self.test = self.test_context.cls(self.test_context)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/rolling_restart_test.py", line 19, in __init__
    super().__init__(test_context, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_cloud_test.py", line 25, in __init__
    self.redpanda = make_redpanda_cloud_service(test_context)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 5087, in make_redpanda_cloud_service
    return RedpandaServiceCloud(context,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1700, in __init__
    self.rebuild_pods_classes()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1726, in rebuild_pods_classes
    self.pods = [
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1727, in <listcomp>
    CloudBroker(p, self.kubectl, self.logger)
  File "/home/ubuntu/redpanda/tests/rptest/services/cloud_broker.py", line 64, in __init__
    self.inject_script("pod_log_extract.sh")
  File "/home/ubuntu/redpanda/tests/rptest/services/cloud_broker.py", line 90, in inject_script
    res = subprocess.check_output(_cp_cmd)
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@cptqbj965nolbbtg7t00-agent', 'kubectl', 'cp', 'pod_log_extract.sh', 'ip-10-1-7-108.us-west-2.compute.internal-pshell:/tmp/pod_log_extract.sh']' returned non-zero exit status 1.

JIRA Link: CORE-4453

michael-redpanda commented 5 months ago

Failed on Weekly CDT on Redpanda Cloud BYOC tier-3-aws-v2-arm. Not sure if I close this or not?

piyushredpanda commented 2 months ago

Closing older-bot-filed CI issues as we transition to a more reliable system.