New resiliency features

[hanapedia]

What

In order to reproduce more complex failure patterns such as metastable failures, local resiliency features should be implemented. Aligned with the objective of Hexagon, these features should be configurable.

Specifications

Request Retry

Different types of request retry policies must be configurable for all secondary adapters when applicable. It may not be applicable for secondary adapters that are not bound by network I/O For starters, following retry policy should be implemented:

• Simple retries: configurable wait duration and max number of attempts
• Exponential backoff: configurable number of max attempts / duration

other retry policies to consider:

• linkerd implements budget based retry policy.
• gRPC client provides similar budget based approaches, where the number of retries are capped by success rate.

Request Timeout

Arbitral duration of request timeout must be configurable for all secondary adapters when applicable. It may not be applicable for secondary adapters that are not bound by network I/O It is assumed that the same amount of timeout will be applied to the retries.

Error Handling

Error Handling must be configurable for all primary adapters. Whether the primary adapter return error upon encountering secondary adapter error should be configurable based on which secondary adapter has failed, so the user can design cases where partial failures are ignored. It is assumed that the secondary adapter produces error when all retries are expired

[hanapedia]

Work plan

• [x] 1. Error propagation
• [x] 2. Retries and timeouts for synchronous protocols
• [x] 3. Consider the implementaions for other protocols

[hanapedia]

Note

on gRPC retry policy

gRPC specifies quite strict retry parameter constraints in the specification, meaning that there is little to no room for retry misconfigurations. It enforces exponential backoffs with 5 max attempts. Also, the configuration is done through service config, meaning that they are enforced on serverside.

on regular http (REST) retry policy

hashicorp provides thin wrapper for net/http package that provides API for custom retry function. The default retry policy is set to exponential backoff. Retries can only be configured per client basis.

consideration

as the approach to the retry implementation variates between client libraries, it might makes sense to implement a common interface for retry mechanism in hexagon's service unit runtime implementation.