search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.48k stars 580 forks source link

CI Failure (Redpanda service failed to start within 20 sec) in `RedpandaFIPSStartupTest.test_startup` #18845

Closed vbotbuildovich closed 2 months ago

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/vtools/builds/14464 https://buildkite.com/redpanda/vtools/builds/14463

Module: rptest.tests.redpanda_startup_test
Class: RedpandaFIPSStartupTest
Method: test_startup
test_id:    RedpandaFIPSStartupTest.test_startup
status:     FAIL
run time:   29.292 seconds

TimeoutError('Redpanda service ducktape-node-12-possibly-handy-doberman failed to start within 20 sec')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 182, in _do_run
    self.setup_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 267, in setup_test
    self.test.setup()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/test.py", line 91, in setup
    self.setUp()
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 39, in setUp
    self.__redpanda.start()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2716, in start
    raise e
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2708, in start
    self.for_nodes(to_start, start_one)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1478, in for_nodes
    return list(executor.map(cb, nodes))
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2700, in start_one
    self.start_node(node,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 3052, in start_node
    self.start_service(node, start_rp)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 3128, in start_service
    start()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 3043, in start_rp
    wait_until(
ducktape.errors.TimeoutError: Redpanda service ducktape-node-12-possibly-handy-doberman failed to start within 20 sec

JIRA Link: CORE-3241

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/vtools/builds/14509 https://buildkite.com/redpanda/vtools/builds/14521

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/vtools/builds/14548 https://buildkite.com/redpanda/vtools/builds/14567 https://buildkite.com/redpanda/vtools/builds/14562 https://buildkite.com/redpanda/vtools/builds/14576

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/vtools/builds/14576 https://buildkite.com/redpanda/vtools/builds/14589 *https://buildkite.com/redpanda/vtools/builds/14598

vbotbuildovich commented 3 months ago

*https://buildkite.com/redpanda/vtools/builds/14598

vbotbuildovich commented 3 months ago

*https://buildkite.com/redpanda/vtools/builds/14598

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/vtools/builds/14598 https://buildkite.com/redpanda/vtools/builds/14609 https://buildkite.com/redpanda/vtools/builds/14625 https://buildkite.com/redpanda/vtools/builds/14624

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/vtools/builds/14659 https://buildkite.com/redpanda/vtools/builds/14692 https://buildkite.com/redpanda/vtools/builds/14660 https://buildkite.com/redpanda/vtools/builds/14710 https://buildkite.com/redpanda/vtools/builds/14734 https://buildkite.com/redpanda/vtools/builds/14709 https://buildkite.com/redpanda/vtools/builds/14753 https://buildkite.com/redpanda/vtools/builds/14747 https://buildkite.com/redpanda/vtools/builds/14748 https://buildkite.com/redpanda/vtools/builds/14763 https://buildkite.com/redpanda/vtools/builds/14762 https://buildkite.com/redpanda/vtools/builds/14768 https://buildkite.com/redpanda/vtools/builds/14777 https://buildkite.com/redpanda/vtools/builds/14778 https://buildkite.com/redpanda/vtools/builds/14793 https://buildkite.com/redpanda/vtools/builds/14812 https://buildkite.com/redpanda/vtools/builds/14811 https://buildkite.com/redpanda/vtools/builds/14847 https://buildkite.com/redpanda/vtools/builds/14864 https://buildkite.com/redpanda/vtools/builds/14865

travisdowns commented 3 months ago

This is failing a lot in nightly CDT both AWS and GCP. Maybe 20s is too short in FIPS mode (self-tests take extra time?)?

travisdowns commented 3 months ago

See also: https://github.com/redpanda-data/redpanda/issues/19959

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/vtools/builds/14990 https://buildkite.com/redpanda/vtools/builds/14998 *https://buildkite.com/redpanda/vtools/builds/14997

michael-redpanda commented 2 months ago

Automatically closing issue to match current state of CORE-3241