redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.41k stars 577 forks source link

CI Failure (crash) in `ManyPartitionsTest.test_many_partitions` #13452

Open andrwng opened 1 year ago

andrwng commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/9435

Module: rptest.scale_tests.many_partitions_test
Class: ManyPartitionsTest
Method: test_many_partitions
test_id:    ManyPartitionsTest.test_many_partitions
status:     FAIL
run time:   620.644 seconds

<NodeCrash ip-172-31-11-122: Segmentation fault on shard 3.
>
Traceback (most recent call last):
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 82, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 887, in test_many_partitions
    self._test_many_partitions(compacted=False)
  File "/home/ubuntu/redpanda/tests/rptest/scale_tests/many_partitions_test.py", line 1102, in _test_many_partitions
    repeater.await_progress(soak_await_msgs, soak_timeout)
  File "/home/ubuntu/redpanda/tests/rptest/services/kgo_repeater_service.py", line 301, in await_progress
    self.redpanda.wait_until(check, timeout_sec=timeout_sec, backoff_sec=1)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1149, in wait_until
    wait_until(wrapped,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 53, in wait_until
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 44, in wait_until
    if condition():
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1146, in wrapped
    assert self.all_up() or self._tolerate_crashes
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    redpanda.raise_on_crash(log_allow_list=log_allow_list)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2514, in raise_on_crash
    raise NodeCrash(crashes)
rptest.services.utils.NodeCrash: <NodeCrash ip-172-31-11-122: Segmentation fault on shard 3.
>

JIRA Link: CORE-4141

andrwng commented 1 year ago

Merging with https://github.com/redpanda-data/redpanda/issues/13449 since the backtrace is the same

vbotbuildovich commented 3 months ago

*https://buildkite.com/redpanda/vtools/builds/9495

github-actions[bot] commented 3 days ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.