redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.39k stars 577 forks source link

CI Failure (key symptom) in `EndToEndShadowIndexingTest.test_reset_from_cloud` #17408

Closed vbotbuildovich closed 1 month ago

vbotbuildovich commented 5 months ago

https://buildkite.com/redpanda/redpanda/builds/46794

Module: rptest.tests.e2e_shadow_indexing_test
Class: EndToEndShadowIndexingTest
Method: test_reset_from_cloud
Arguments: {
    "cloud_storage_type": 1
}
test_id:    EndToEndShadowIndexingTest.test_reset_from_cloud
status:     FAIL
run time:   163.769 seconds

<BadLogLines nodes=docker-rp-15(1) example="ERROR 2024-03-26 16:29:40,647 [shard 1:au  ] cloud_storage - partition_manifest.cc:1036 - ["90000000/meta/kafka/panda-topic/0_24/manifest.bin"] New segment does not line up with previous segment: {anomaly_types: {{offset_gap}}, new_segment: {o=15897-16410 t={timestamp: 1711470563708}-{timestamp: 1711470563817}}, previous_segment: {{is_compacted: false, size_bytes: 1091437, base_offset: 13339, committed_offset: 13848, base_timestamp: {timestamp: 1711470562580}, max_timestamp: {timestamp: 1711470562714}, delta_offset: 12, ntp_revision: 24, archiver_term: 1, segment_term: 1, delta_offset_end: 12, sname_format: {v3}, metadata_size_hint: 0}}}">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 173, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1569, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-15(1) example="ERROR 2024-03-26 16:29:40,647 [shard 1:au  ] cloud_storage - partition_manifest.cc:1036 - ["90000000/meta/kafka/panda-topic/0_24/manifest.bin"] New segment does not line up with previous segment: {anomaly_types: {{offset_gap}}, new_segment: {o=15897-16410 t={timestamp: 1711470563708}-{timestamp: 1711470563817}}, previous_segment: {{is_compacted: false, size_bytes: 1091437, base_offset: 13339, committed_offset: 13848, base_timestamp: {timestamp: 1711470562580}, max_timestamp: {timestamp: 1711470562714}, delta_offset: 12, ntp_revision: 24, archiver_term: 1, segment_term: 1, delta_offset_end: 12, sname_format: {v3}, metadata_size_hint: 0}}}">

JIRA Link: CORE-1937

nvartolomei commented 2 months ago

Logs are long gone and can't reproduce. Waiting for another occurrence.

WillemKauf commented 1 month ago

Possibly fixed by https://github.com/redpanda-data/redpanda/pull/21348.

WillemKauf commented 1 month ago

Closing due to potentially fixed issue and lack of logs.