redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.64k stars 587 forks source link

test: fix ducktape test failures in clustered bare metal environment #3680

Closed dotnwat closed 2 years ago

dotnwat commented 2 years ago

When running ducktape tests against a bare metal clustered environment a large percentage of the tests pass out-of-the-box. Most tests that don't pass appear to be related to an interaction with the allowed_log feature in ducktape that asserts unknown errors do not appear in redpanda logs. Finally, there are a few tests that have real failures that may be related to being run in the clustered environment.

This ticket covers ensuring that all the tests pass.

ajfabbri commented 2 years ago

As discussed w/ Noah, I've been taking the time to add more automation around this case to eliminate the manual setup steps. Adding support for our ansible inventory files to ducktape here.

ajfabbri commented 2 years ago

Update: Automation took longer than expected but we learned a lot and made great progress. The clustered bug fixing was also time consuming, but we're wrapping it with a big team effort. 🙌

ajfabbri commented 2 years ago

Our set of successful tests between clustered and containerized DT runs is very similar now. Thanks everyone who helped.