Catch cache errors and raise errors where necessary

jaywhy13 commented 3 years ago

Waffle currently tries to retrieve a setting from the cache, then goes to the DB if there is a cache miss. This doesn't account for issues accessing the cache. If there is a cache issue (e.g. a TimeoutError), this error is thrown to the caller. In prod, we've encountered several of these issues for a small percentage of our traffic.

I'd suggest making the following changes:

By default, catch exceptions thrown by the cache and go directly to the database.
Add a setting that allows the dev to configure the number of retries.
Allow the dev to configure the number of retries when calling each function (e.g. switch_is_active)

RenDelaCruz commented 3 years ago

We're facing the exact same problem. If the cache is not available, there's no way for waffle to go directly to the DB.

This https://github.com/django-waffle/django-waffle/issues/350#issuecomment-630376602 is a possible solution to always read from the DB, but it will hurt performance. It looks like going to the DB upon cache failures by default is a good idea.

clintonb commented 3 years ago

I'm happy to review a PR if folks want to implement these changes.

jazzband / django-waffle

Catch cache errors and raise errors where necessary #399