Dispatcher error handler and retrier features

[kamikazechaser]

All the below depend on the correct RPC response. An error handler should be written to correctly propagate the error type from the RPC node. This is a bit difficult for sidechains e.g. Celo which implement custom response error messages.

• [x] RPC error handler with core error return types

Gas bump

Resign a transaction and bump the gas price by 15%. On the otx table, the old transaction should be invalidated and a new one created in a single tx. Gas limit issues can also be included here but ideally, we set a generous gas limit that on the gas oracle itself.

• [ ] Invalidated flag on otx table
• [ ] Resigner

Low nonce issues

Ideally this kind of error should be near-impossible to get if the signer implementation is correct as we maintain the nonce sequence offline. However, in the rare event that it does occur, a special noop tx which attempt a random low gas sign should be attempted to close the gap.

• [ ] Special noop tx tyoe signer
• [x] Periodic health check to chech if a tx is mined/reverted. If not, we may have any of the issues described here.

Out of balance

Ideally, we never face this issue. A lock should be applied on the account to prevent further tx sign requests. There should always be a pre-check on the dispatcher before it is attempted.

• [ ] Gas lock implementation
• [ ] Balance check middleware.

[kamikazechaser]

A retrier logger and health check has been added to confirm if the dispatch error handler works in the first place before we can proceed to write complex retrying logic.