redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.38k stars 577 forks source link

CI Failure (no topic manifest anomaly) in `SIAdminApiTest.test_manifest_dump` #10932

Closed VladLazar closed 1 year ago

VladLazar commented 1 year ago

https://buildkite.com/redpanda/redpanda/builds/29529#018837b4-f2b8-4742-b098-86a29623a733

Module: rptest.tests.shadow_indexing_admin_api_test
Class:  SIAdminApiTest
Method: test_manifest_dump
test_id:    rptest.tests.shadow_indexing_admin_api_test.SIAdminApiTest.test_manifest_dump
status:     FAIL
run time:   31.092 seconds

    RuntimeError("Object storage scrub detected fatal anomalies of type {'ntr_no_topic_manifest'}")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 103, in wrapped
    self.redpanda.stop_and_scrub_object_storage()
  File "/root/tests/rptest/services/redpanda.py", line 3284, in stop_and_scrub_object_storage
    raise RuntimeError(
RuntimeError: Object storage scrub detected fatal anomalies of type {'ntr_no_topic_manifest'}
michael-redpanda commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/8053#0188a97e-103b-4222-a146-9e5360524691 https://buildkite.com/redpanda/vtools/builds/8040#0188a1c4-e577-42ce-8641-5eadd5124787

ztlpn commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/8065#0188b139-3f45-49fb-9528-0ce49fefce98

michael-redpanda commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/8095#0188bb87-3f08-4e99-a21d-dcac231c8252

andijcr commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/8434#0189416c-3f23-492c-bc9b-b458c0b62f92

abhijat commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/8434#0189416c-3f23-492c-bc9b-b458c0b62f92 https://ci-artifacts.dev.vectorized.cloud/vtools/8434/0189416c-3f23-492c-bc9b-b458c0b62f92/vbuild/ducktape/results/2023-07-10--001/SIAdminApiTest/test_manifest_dump/1079/

andrwng commented 1 year ago

The issue here is that we disable tiered storage and restart before getting a chance to upload the topic manifest, despite being able to upload the partition manifest. The test should wait for the topic manifest to show up, or we should be lenient in these kinds of failures given this test disabled tiered storage.

mmaslankaprv commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/9006

abhijat commented 1 year ago

https://buildkite.com/redpanda/redpanda/builds/36183#018a4d7d-4ccc-48d4-b586-924806621bb3

RuntimeError: Object storage scrub detected fatal anomalies of type {'ntr_no_topic_manifest'}