Configurable error handling and retrying in HTTP target

[pondzix]

Jira ref: PDP-1203

Drafting:

• Flexible configuration allowing us to categorize HTTP responses and match specific retrying strategies.
• Very simple application health store as global variable.
• Very simple monitoring interface used to send alerts during retries in HTTP target.

It doesn't compile now! It's just to show how such configuration could look like and validate such approach.

In general the goal is to make default error handling and retries similar to what we have in our new streaming loaders.

[colmsnowplow]

This is great @pondzix!

I left some thoughts - broadly love the ideas & direction here. Two things I'm thinking about on this design:

• Can we use this or similar mechanism for all the targets, and replace the retry in sourceWriteFunc? (and what does design look like, if we can't there's implications etc).

• Can marking as unhealthy work similarly to loaders for our use case? (maybe for some cases and not others)

I'm not sure I have a firm opinion on the second yet, just that I would be keen to understand it and think it through. :)