Add configurable retries for target writes

[pondzix]

PDP-1346

Currently retrying configuration is hardcoded to 5 attempts with exponential delay (inital 1 second) for each type of an error.

This commit brings a couple of improvements to retrying:

• Make retrying settings configurable from HCL. Max attempts and delay can now be customized.
• Split retrying strategies into 2 categories: transient and setup. Target can now signal what kind of an error has happened and what kind of retrying strategy should be applied.
• HTTP target can now return setup errors by the new response rules configuration. These rules also allows the target to match invalid data. Previously it was either failure (retried) or oversized data.
• Eventually setup errors will also trigger another actions like toggling application health status and sending monitoring alerts. For now behaviour for setup errors is basically the same as for transient errors, but extending behaviour for setup errors should be relatively easy with this structure.

[pondzix]

General question - what does this implementation mean for the idea of relying on the SDK retries for other targets?

Not much. Retrying in SDKs is still there 😁

Whatever transient error leaks from the SDK (like throttling error for kinesis) will handled here + from now retrying details (delay, max attempts) could also be configured. In the future we can also extend other target to catch and return setup errors to use different retry strategy. But we don't have to do anything now.

With current state of things I just think this solution is a nice middle ground.

[colmsnowplow]

With current state of things I just think this solution is a nice middle ground.

I don't disagree!

Overall looks good, just two things I think need tweaking before I approve (possible value leakage and placement of parsing respose) :)