CI Failure (key symptom) in `OMBValidationTest.test_retention`

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15481

Module: rptest.redpanda_cloud_tests.omb_validation_test
Class: OMBValidationTest
Method: test_retention

test_id:    OMBValidationTest.test_retention
status:     FAIL
run time:   748.177 seconds

AssertionError("['Metric aggregatedEndToEndLatency99pct, value 175.932, Expected to be <= 60.0, check failed.', 'Metric aggregatedEndToEndLatency999pct, value 277.553, Expected to be <= 100.0, check failed.']")
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 750, in test_retention
    self.run_benchmark_with_retries(benchmark, max_retries)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 99, in run_benchmark_with_retries
    self.handle_failed_attempt(try_count, max_retries, benchmark,
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 130, in handle_failed_attempt
    benchmark.check_succeed()
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 381, in check_succeed
    OMBSampleConfigurations.validate_metrics(self._metrics,
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark_configs.py", line 132, in validate_metrics
    assert is_valid, str(results)
AssertionError: ['Metric aggregatedEndToEndLatency99pct, value 175.932, Expected to be <= 60.0, check failed.', 'Metric aggregatedEndToEndLatency999pct, value 277.553, Expected to be <= 100.0, check failed.']

JIRA Link: CORE-5564

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15544 https://buildkite.com/redpanda/vtools/builds/15590 https://buildkite.com/redpanda/vtools/builds/15609 https://buildkite.com/redpanda/vtools/builds/15610 *https://buildkite.com/redpanda/vtools/builds/15621

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15643

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15673 https://buildkite.com/redpanda/vtools/builds/15674 *https://buildkite.com/redpanda/vtools/builds/15675

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15730 https://buildkite.com/redpanda/vtools/builds/15729

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15771 https://buildkite.com/redpanda/vtools/builds/15781 *https://buildkite.com/redpanda/vtools/builds/15783

vbotbuildovich commented 1 month ago