redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.41k stars 577 forks source link

CI Failure (key symptom) in `RandomNodeOperationsTest.test_node_operations` #22718

Open vbotbuildovich opened 1 month ago

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/16155

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 0,
    "enable_failures": true,
    "with_tiered_storage": true
}
test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   2714.540 seconds

<BadLogLines nodes=ducktape-node-14-quickly-better-racer(1) example="ERROR 2024-08-03 09:45:55,870 [shard 2:au  ] archival - [fiber18 kafka/fuzzy-operator-5546-zksxaq/0] - ntp_archiver_service.cc:3017 - Failed to make reupload candidate to match the run, candidate: {source segment offsets: {term:5, base_offset:33, committed_offset:492, dirty_offset:492}, exposed_name: 33-5-v1.log, starting_offset: 33, file_offset: 0, content_length: 5552, final_offset: 492, final_file_offset: 5552, term: 5, source names: {/var/lib/redpanda/data/kafka/fuzzy-operator-5546-zksxaq/0_3732/33-5-v1.log}}, run: {is_compacted: false, size_bytes: 29248, base_offset: 33, committed_offset: 492, base_timestamp: {timestamp: 1722678092044}, max_timestamp: {timestamp: 1722678092058}, delta_offset: 33, ntp_revision: 68, archiver_term: 5, segment_term: 5, delta_offset_end: 51, sname_format: {v3}, metadata_size_hint: 0}">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 190, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1581, in raise_on_bad_logs
    lsearcher.search_logs(_searchable_nodes)
  File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 197, in search_logs
    raise BadLogLines(bad_loglines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ducktape-node-14-quickly-better-racer(1) example="ERROR 2024-08-03 09:45:55,870 [shard 2:au  ] archival - [fiber18 kafka/fuzzy-operator-5546-zksxaq/0] - ntp_archiver_service.cc:3017 - Failed to make reupload candidate to match the run, candidate: {source segment offsets: {term:5, base_offset:33, committed_offset:492, dirty_offset:492}, exposed_name: 33-5-v1.log, starting_offset: 33, file_offset: 0, content_length: 5552, final_offset: 492, final_file_offset: 5552, term: 5, source names: {/var/lib/redpanda/data/kafka/fuzzy-operator-5546-zksxaq/0_3732/33-5-v1.log}}, run: {is_compacted: false, size_bytes: 29248, base_offset: 33, committed_offset: 492, base_timestamp: {timestamp: 1722678092044}, max_timestamp: {timestamp: 1722678092058}, delta_offset: 33, ntp_revision: 68, archiver_term: 5, segment_term: 5, delta_offset_end: 51, sname_format: {v3}, metadata_size_hint: 0}">

JIRA Link: CORE-6782

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/16160 https://buildkite.com/redpanda/vtools/builds/16160 https://buildkite.com/redpanda/vtools/builds/16167 https://buildkite.com/redpanda/vtools/builds/16167 https://buildkite.com/redpanda/vtools/builds/16168 https://buildkite.com/redpanda/vtools/builds/16168