search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.31k stars 572 forks source link

CI Failure (key symptom) in `RandomNodeOperationsTest.test_node_operations` #21279

Open vbotbuildovich opened 1 month ago

vbotbuildovich commented 1 month ago

https://buildkite.com/redpanda/vtools/builds/15368

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 3,
    "enable_failures": true,
    "with_tiered_storage": true
}
test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   2951.624 seconds

<BadLogLines nodes=ip-172-31-15-221(1) example="ERROR 2024-07-06 12:08:14,218 [shard 0:au  ] archival - [fiber185 kafka/tp-workload-compaction/16] - ntp_archiver_service.cc:1266 - failed to upload segment "54e8c4d2/kafka/tp-workload-compaction/16_61/17373-23312-557126-1-v1.log.2": seastar::nested_exception (seastar::nested_exception)">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 190, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1570, in raise_on_bad_logs
    lsearcher.search_logs(_searchable_nodes)
  File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 197, in search_logs
    raise BadLogLines(bad_loglines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-15-221(1) example="ERROR 2024-07-06 12:08:14,218 [shard 0:au  ] archival - [fiber185 kafka/tp-workload-compaction/16] - ntp_archiver_service.cc:1266 - failed to upload segment "54e8c4d2/kafka/tp-workload-compaction/16_61/17373-23312-557126-1-v1.log.2": seastar::nested_exception (seastar::nested_exception)">

JIRA Link: CORE-5533

vbotbuildovich commented 1 month ago

*https://buildkite.com/redpanda/vtools/builds/15626