Open vbotbuildovich opened 1 month ago
Many tests failed due to the same infra issue remote commands kept failing
RemoteCommandError({'ssh_config': {'host': 'ip-172-31-3-159', 'hostname': '172.31.3.159', 'user': 'root', 'port': 22, 'password': None, 'identityfile': '/home/ubuntu/.ssh/id_rsa'}, 'hostname': 'ip-172-31-3-159', 'ssh_hostname': '172.31.3.159', 'user': 'root', 'externally_routable_ip': '54.184.188.225',
Example Buildkite Job: https://buildkite.com/redpanda/vtools/builds/15928
Going to close others as duplicate of this issue
Also, in some tests additonal error is present:
Example Buildkite Job: https://buildkite.com/redpanda/vtools/builds/15928
ClientError('An error occurred (AuthenticationRequired) when calling the ListBuckets operation: Authentication required.')
Traceback (most recent call last):
File "/home/ubuntu/redpanda/tests/rptest/archival/s3_client.py", line 530, in list_objects
res = self._list_objects(bucket=bucket,
File "/home/ubuntu/redpanda/tests/rptest/archival/s3_client.py", line 47, in do_retry
return fn(*args, **kwargs)
File "/home/ubuntu/redpanda/tests/rptest/archival/s3_client.py", line 503, in _list_objects
return client.list_objects_v2(Bucket=bucket,
File "/opt/.ducktape-venv/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/.ducktape-venv/lib/python3.10/site-packages/botocore/client.py", line 964, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AuthenticationRequired) when calling the ListObjectsV2 operation: Authentication required.
We need to investigate and fix both of these and close all other GT issues that were opened due to 15126 or 15128
P.S. Also, going to open a Jira task to improve PandaTriage logging to make it easier to map test error with GH issue AND related Buildkite Job
Example of how this should be logged:
opening issue for test failure StreamVerifierTest.test_simple_produce_consume_txn_with_add_node created issue: https://github.com/redpanda-data/redpanda/issues/21625 Based on CI job: https://buildkite.com/redpanda/vtools/builds/15926
Moved Jira issue to https://redpandadata.atlassian.net/browse/PESDLC-1717
https://buildkite.com/redpanda/vtools/builds/15926
JIRA Link: CORE-5780