search

arunaengine / proxy

DEPRECATED: See https://github.com/ArunaStorage/aruna/tree/main/components/data_proxy for latest release
Apache License 2.0
5 stars 0 forks source link

[bug] Stale connection retries #37

Closed St4NNi closed 11 months ago

St4NNi commented 1 year ago

Bug

If the internal notifier connection was not used for approx. 1 minute it goes stale and needs to be retried on the next occurrence, this results in an "Internal Notifier Error" error.

To fix this we should implement 2 measures:

  1. Add retry / backoff logic for random transport errors in tonic / h2
  2. Open a new backend connection for each user request
St4NNi commented 1 year ago

This will be mitigated by: https://github.com/ArunaStorage/ArunaServer/commit/6ffb29a2d70952ab8eab4be8a0e2b8fcef9bd605

For a proper retry logic we should wait for: https://github.com/hyperium/tonic/issues/733 and: https://github.com/tower-rs/tower/issues/682