search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.41k stars 577 forks source link

CI Failure (key symptom) in `TieredStorageCacheStressTest.garbage_objects_test` #19921

Open vbotbuildovich opened 2 months ago

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/14692 https://buildkite.com/redpanda/vtools/builds/14660 https://buildkite.com/redpanda/vtools/builds/14734 https://buildkite.com/redpanda/vtools/builds/14709 https://buildkite.com/redpanda/vtools/builds/14753 https://buildkite.com/redpanda/vtools/builds/14748 https://buildkite.com/redpanda/vtools/builds/14762 https://buildkite.com/redpanda/vtools/builds/14768 https://buildkite.com/redpanda/vtools/builds/14778 https://buildkite.com/redpanda/vtools/builds/14793 https://buildkite.com/redpanda/vtools/builds/14811 https://buildkite.com/redpanda/vtools/builds/14847 https://buildkite.com/redpanda/vtools/builds/14865

Module: rptest.scale_tests.tiered_storage_cache_stress_test
Class: TieredStorageCacheStressTest
Method: garbage_objects_test
test_id:    TieredStorageCacheStressTest.garbage_objects_test
status:     FAIL
run time:   79.988 seconds

MetricCheckFailed('redpanda_cloud_storage_cache_trim_fast_trims_total', 0.0, 1.0)
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/tiered_storage_cache_stress_test.py", line 502, in garbage_objects_test
    self.run_test_with_cache_prefilled(
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/tiered_storage_cache_stress_test.py", line 472, in run_test_with_cache_prefilled
    m.expect([('redpanda_cloud_storage_cache_trim_fast_trims_total',
  File "/home/ubuntu/redpanda/tests/rptest/services/metrics_check.py", line 141, in expect
    raise error
rptest.services.metrics_check.MetricCheckFailed: MetricCheckFailed<redpanda_cloud_storage_cache_trim_fast_trims_total old=0.0 new=1.0>

JIRA Link: CORE-4244

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/14897 https://buildkite.com/redpanda/vtools/builds/14930 https://buildkite.com/redpanda/vtools/builds/14952 https://buildkite.com/redpanda/vtools/builds/14966 https://buildkite.com/redpanda/vtools/builds/14975 https://buildkite.com/redpanda/vtools/builds/14983 https://buildkite.com/redpanda/vtools/builds/15021 https://buildkite.com/redpanda/vtools/builds/15041 *https://buildkite.com/redpanda/vtools/builds/15057

vbotbuildovich commented 2 months ago

*https://buildkite.com/redpanda/vtools/builds/15094

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15106 https://buildkite.com/redpanda/vtools/builds/15115 https://buildkite.com/redpanda/vtools/builds/15117 https://buildkite.com/redpanda/vtools/builds/15127 https://buildkite.com/redpanda/vtools/builds/15138 https://buildkite.com/redpanda/vtools/builds/15137 https://buildkite.com/redpanda/vtools/builds/15152 https://buildkite.com/redpanda/vtools/builds/15156 https://buildkite.com/redpanda/vtools/builds/15150 https://buildkite.com/redpanda/vtools/builds/15165 https://buildkite.com/redpanda/vtools/builds/15177 https://buildkite.com/redpanda/vtools/builds/15164 https://buildkite.com/redpanda/vtools/builds/15198 https://buildkite.com/redpanda/vtools/builds/15195 https://buildkite.com/redpanda/vtools/builds/15239 https://buildkite.com/redpanda/vtools/builds/15283 https://buildkite.com/redpanda/vtools/builds/15299 https://buildkite.com/redpanda/vtools/builds/15296

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15106 https://buildkite.com/redpanda/vtools/builds/15115 https://buildkite.com/redpanda/vtools/builds/15127 https://buildkite.com/redpanda/vtools/builds/15117 https://buildkite.com/redpanda/vtools/builds/15138 https://buildkite.com/redpanda/vtools/builds/15137 https://buildkite.com/redpanda/vtools/builds/15152 https://buildkite.com/redpanda/vtools/builds/15150 https://buildkite.com/redpanda/vtools/builds/15156 https://buildkite.com/redpanda/vtools/builds/15165 https://buildkite.com/redpanda/vtools/builds/15164 https://buildkite.com/redpanda/vtools/builds/15177 https://buildkite.com/redpanda/vtools/builds/15195 https://buildkite.com/redpanda/vtools/builds/15198 https://buildkite.com/redpanda/vtools/builds/15239 https://buildkite.com/redpanda/vtools/builds/15283 https://buildkite.com/redpanda/vtools/builds/15299 https://buildkite.com/redpanda/vtools/builds/15296

vbotbuildovich commented 2 months ago

*https://buildkite.com/redpanda/vtools/builds/15338

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15352 https://buildkite.com/redpanda/vtools/builds/15368 *https://buildkite.com/redpanda/vtools/builds/15370

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15384 https://buildkite.com/redpanda/vtools/builds/15383

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15390 https://buildkite.com/redpanda/vtools/builds/15401 *https://buildkite.com/redpanda/vtools/builds/15399

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15454 https://buildkite.com/redpanda/vtools/builds/15457

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15473 https://buildkite.com/redpanda/vtools/builds/15488 *https://buildkite.com/redpanda/vtools/builds/15485

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15496 https://buildkite.com/redpanda/vtools/builds/15512 https://buildkite.com/redpanda/vtools/builds/15509 https://buildkite.com/redpanda/vtools/builds/15562 https://buildkite.com/redpanda/vtools/builds/15559 https://buildkite.com/redpanda/vtools/builds/15577 https://buildkite.com/redpanda/vtools/builds/15601 https://buildkite.com/redpanda/vtools/builds/15600 https://buildkite.com/redpanda/vtools/builds/15612 https://buildkite.com/redpanda/vtools/builds/15617 *https://buildkite.com/redpanda/vtools/builds/15614

vbotbuildovich commented 2 months ago

https://buildkite.com/redpanda/vtools/builds/15627 https://buildkite.com/redpanda/vtools/builds/15626

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15639 https://buildkite.com/redpanda/vtools/builds/15654 *https://buildkite.com/redpanda/vtools/builds/15650

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15665 https://buildkite.com/redpanda/vtools/builds/15690 *https://buildkite.com/redpanda/vtools/builds/15687

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15718 https://buildkite.com/redpanda/vtools/builds/15734 *https://buildkite.com/redpanda/vtools/builds/15738

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15774 https://buildkite.com/redpanda/vtools/builds/15789 *https://buildkite.com/redpanda/vtools/builds/15788

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15817 https://buildkite.com/redpanda/vtools/builds/15828 *https://buildkite.com/redpanda/vtools/builds/15826

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15840 https://buildkite.com/redpanda/vtools/builds/15841

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15844 https://buildkite.com/redpanda/vtools/builds/15851

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15879 https://buildkite.com/redpanda/vtools/builds/15890 *https://buildkite.com/redpanda/vtools/builds/15893

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15913 https://buildkite.com/redpanda/vtools/builds/15925

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15934