search

sciencehistory / scihist_digicoll

Science History Institute Digital Collections
Other
13 stars 0 forks source link

Try using Redis instead of Memcached for Rails cache #2143

Open jrochkind opened 1 year ago

jrochkind commented 1 year ago

to be a store for rack-attack.

Can we keep using rack-attack but get away from these annoying periodic dalli/memcached errors? What if we try using a redis instance of a memcached instance? Still provided by a heroku plugin. Cost might be similar?

A different redis than the one we're using for activejob queue, since it should have different settings -- to use as a cache it should be set as a cache to evict old values when it gets full (which the ActiveJob queue should not!)

jrochkind commented 1 year ago

OK, I'm realizing that all those annoying Dalli::Ring errors we get.... aren't actually interupting our users at all.

Rails does just move on from the cache failure, rescuing the exception.

It just also reports the error in a way that Honeybadger hooks into. Note that in honeybadger the Dalli::Ring errors have tags [severity:warning ], [handled:true].

Which is coming from here.

It's already a silent ignore -- but with error logging.

Maybe we just want this error to never be reported to honeybadger at all, so it doesn't annoy us?

We could do that just by adding to our honeybadger before_notify config something like:

config.before_notify do |notice|
  if notice.exception.class < MyError && notice.tags.include?("severity:warning") && notice.tags.include?("handled:true")
    notice.halt!
  end
end

Not sure if we want to just totally ignore these or not.... are we bothered that memcached could be super duper misbehaving and we wouldn't know it?

Alternately, can we get honeybadger to group these better, so we only get ONE honeybadger error instead of multiple....

jrochkind commented 1 year ago

OK, I think the number of errors we get is actually pretty low -- like one every few weeks or so, which is FINE, especially for this kind of cache. I don't know if it justifies changing to a whole different data store, the one we have is actually working fine.

We may want to do something to reduce the HoneyBadger alerting when it does happen. We could have HB ignore them completely (not even logged by HB), or we could adjust the way HoneyBadger decides they are the "same" error to only get one per occurence instead of 4-8. Or probably other ideas.