tarantool / vshard

The new generation of sharding based on virtual buckets
Other
99 stars 30 forks source link

Allow to backoff replica if request fails #485

Open Serpentian opened 3 months ago

Serpentian commented 3 months ago

In mission critical systems requests cannot fail at all. We have automatic backoff procedure, which forbids to make requests to some instance for REPLICA_BACKOFF_INTERVAL, when storage is disabled or privileges weren't granted. I propose to extend its usage and allow users to configure it.

We'll introduce two new configuration options: can_backoff_after and backoff_timeout. can_backoff_after is a trigger, which accepts error, we got from a client, and a func name we tried to call. It returns true or false. If true is returned, replica goes into backoff state for backoff_timeout, no requests can be done to this replica, while it's in the backoff state.

This will allow users to minimize the number of failing requests a lot. Backoff procedure will be triggered when request to the instance fails or when ping in failover fails.