apache / bookkeeper

Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads
https://bookkeeper.apache.org/
Apache License 2.0
1.91k stars 903 forks source link

Handle too_many_requests gracefully #1369

Open ArvinDevel opened 6 years ago

ArvinDevel commented 6 years ago

FEATURE REQUEST

  1. Please describe the feature you are requesting. Add more policy to deal too_many_requests error

  2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue? nice-to-have

  3. Provide any additional detail on your proposed use case for this feature. Currently BookKeeper client use general error dealing logic to handle error, including too_many_requests error sent from bookie. The handle logic is remaking ensemble with excluding 'failed' bookie, but this is unfriendly. For example, when the cluster is small, remaking action will fail and the ledger will be closed, and following add-request will throw error. Maybe we can add other handling error policy: retry connection to 'failed' bookie; block request when resource is limited(one case is when adding to previous 'failed' bookie fail again). And these two policy should be used after current handling logic, and we need to log some useful information, such blocking state when using blocking i/o. I know this blocking and retring 'failed' bookie logic may bring in difficulty to low-latency goal, although it bring some out-of-box more easy-using feature in.

eolivelli commented 6 years ago

Totally +1