handle errors in hops along the chain properly

[fkleedorfer]

Forseeable errors (internal errors in a chain node controller, or unreachable chain nodes) should be propagated back to the originator so it can retry sending the request.

[fkleedorfer]

Suggestion: add an status code field and an error message field to the Message class, and add a MessageResult class with status code and error message. The MessageResult class is used as for providing a JSON result for the '/request' and '/response'.

When an error occurs during message handling on a chain node there are two options:

1. it happens before the http response is sent, which means it happens before the http request is asynchronously sent to the next chain node or the destination service. In that case, the MessageResult object in the http result contains an error code identifying the problem (we should list possible errors and assign a code).
2. when the message is sent to the next chain node in line, the call to /request returns a MessageResult that indicates an error. In that case, the message sent in the /response call on the previous-in-line chain node gets the status code and error message copied from the MessageResult.

That way, the only problem we can't propagate back to the originator is an error sending the response back - as there is no way of bypassing a chain node that works ok for the 'request' direction but not for the 'response' direction.

That means we have to implement a timeout for the originator after which we have to assume that the call was made but the response was not propagated back through the chain correctly. That timeout could be guaranteed by enforcing a timeout on all http calls we make, so we avoid confusing a slow server somewhere along the way with a problem during back-propagation.

[fkleedorfer]

If we go with the deferred request approach, this task is greatly simplified. Let's wait until we get a decision.