CI Failure (mand 'python3 /tmp/compute_storage.py) in `EndToEndShadowIndexingTest.test_write`

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15117

Module: rptest.tests.e2e_shadow_indexing_test
Class: EndToEndShadowIndexingTest
Method: test_write
Arguments: {
    "cloud_storage_type": 1
}

test_id:    EndToEndShadowIndexingTest.test_write
status:     FAIL
run time:   69.396 seconds

RemoteCommandError({'ssh_config': {'host': 'ip-172-31-9-197', 'hostname': '172.31.9.197', 'user': 'root', 'port': 22, 'password': None, 'identityfile': '/home/ubuntu/.ssh/id_rsa'}, 'hostname': 'ip-172-31-9-197', 'ssh_hostname': '172.31.9.197', 'user': 'root', 'externally_routable_ip': '18.246.33.146', '_logger': <Logger rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3-391 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f9e367d5b10>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f9e367ff1c0>, '_custom_ssh_exception_checks': None}, 'python3 /tmp/compute_storage.py --sizes --md5 --print-flat --data-dir /var/lib/redpanda/data', 1, b'')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/e2e_shadow_indexing_test.py", line 591, in test_write
    for file in self.redpanda.data_checksum(node):
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 4304, in data_checksum
    lines = node.account.ssh_output(cmd, timeout_sec=120)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 41, in wrapper
    return method(self, *args, **kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 397, in ssh_output
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-9-197: Command 'python3 /tmp/compute_storage.py --sizes --md5 --print-flat --data-dir /var/lib/redpanda/data' returned non-zero exit status 1.

JIRA Link: CORE-5035

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15138 https://buildkite.com/redpanda/vtools/builds/15152 https://buildkite.com/redpanda/vtools/builds/15165 https://buildkite.com/redpanda/vtools/builds/15198 https://buildkite.com/redpanda/vtools/builds/15241 https://buildkite.com/redpanda/vtools/builds/15299

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15138 https://buildkite.com/redpanda/vtools/builds/15152 https://buildkite.com/redpanda/vtools/builds/15165 https://buildkite.com/redpanda/vtools/builds/15198 https://buildkite.com/redpanda/vtools/builds/15241 https://buildkite.com/redpanda/vtools/builds/15299

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15370

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15384

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15401

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15457

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15488

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15512 https://buildkite.com/redpanda/vtools/builds/15562 https://buildkite.com/redpanda/vtools/builds/15601 https://buildkite.com/redpanda/vtools/builds/15614

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15627

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15654

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15690

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15738

dotnwat commented 1 month ago

No apparent problem in the logs. Could be a race condition. If good output from the compute_storage.py tool is not being produced (perhaps i'm misisng it) then we should add that at a minimum so future errors with this tool are easier to dianose.

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15789

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15828

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15841

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15853

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15893

vbotbuildovich commented 3 weeks ago

*https://buildkite.com/redpanda/vtools/builds/16118