Enable allocation failure injection in debug builds

[jcsp]

We've talked from time to time about the need to do failure injection to make us more robust against bad_alloc exceptions -- depending on exactly where they come up, they often exercise rarely-taken error handling/destruction code paths -- this ticket is for suggesting a specific plan of action.

Seastar has a failure injection hook wrapped in include/seastar/util/alloc_failure_injector.hh but we don't use it.

• Set SEASTAR_ENABLE_ALLOC_FAILURE_INJECTION in our cmake files for debug mode only
• Add a configuration property for enabling failure injection and the proportion of allocations to fail. This needs to be a config rather than an admin API call, because we want to cover failures even during startup (especially during e.g. replaying logs).
• On startup, if we should be failing some proportion of allocations, call alloc_failure_injection::fail_after with a pseudorandom parameter based on the expected failure proportion. e.g. if failure proportion is 0.01, then we should pick a number between 1 and 200 for the fail_after (i.e. average of 100 successful allocations between failures)
• Extend our custom @cluster decorator (this is being added in https://github.com/vectorizedio/redpanda/pull/3226) to have a parameter for failure injection. If true, then the decorator should add two parametrize decorators to the wrapped functions, to toggle this off and on.
• Identify a representative set of existing tests to cover the major redpanda subsystems (i.e. basic produce/consume, transactions, consumer groups, admin api, pandaproxy, schema registry). Add the failure injection flag to these tests' @cluster definitions.
• Ensure the tests have enough fault tolerance to handle e.g. a single kafka request failure, a single HTTP request failure. The expectation should be that our failure injection rate is low enough that a retry works. If a test is requesting a particularly large structure then the request may be impossible to fulfil -- this will require tweaks. If existing tests aren't amenable to this, create new ones that cover the same subsystems' functionality but written with failure injection in mind.

JIRA Link: CORE-817

[jcsp]

@mmaslankaprv I know you had thoughts about bad_alloc failure injection -- is this anything like what you had in mind?

[mmaslankaprv]

This is exactly what i had in mind. I saw that the allocation failure injector was introduced in seastar recently

[rockwotj]

Closing this - we abort on allocation failures at the moment so it's not helpful to enable this.