sasa1977 / con_cache

ets based key/value cache with row level isolated writes and ttl support
MIT License
910 stars 71 forks source link

Crashed cache won't work anymore after restart #31

Closed alfredbaudisch closed 6 years ago

alfredbaudisch commented 7 years ago

Unfortunately I don't have much more information to try to reproduce, but if ConCache crashes, after the Supervisor restarts it keeps repeating:

** (MatchError) no match of right hand side value: []
      (con_cache) lib/con_cache/registry.ex:18: ConCache.Registry.get/1
      (con_cache) lib/con_cache.ex:143: ConCache.get/2
    (fred_engine) lib/engine/message.ex:5: Fred.Engine.Message.work/1
         (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
         (stdlib) erl_eval.erl:228: :erl_eval.expr/5
         (elixir) lib/enum.ex:1628: anonymous fn/3 in Enum.reduce/3
         (elixir) lib/range.ex:92: Enumerable.Range.reduce/5
         (elixir) lib/enum.ex:1627: Enum.reduce/3
sasa1977 commented 7 years ago

Judging by this output, the cache owner process is not running. Can you confirm that it has in fact restarted?

I also wonder how did the cache owner process crash in the first place. That process does fairly simple work which is related only to expiring of items, so it shouldn't crash (unless there's a bug of course). Can you find a log record of the cache process crashing?

alfredbaudisch commented 7 years ago

Sorry for the delay. It was something like this:

Application
|
+--Some Supervisor
      |
      +-- con_cache
      +-- something else

Some Supervisor crashed, and after restarting, con_cache was in that state. Unfortunately after my report I wasn't able to reproduce anymore.

sasa1977 commented 7 years ago

This doesn't leave much to go on :(

In any case, you may want to upgrade you con_cache (assuming you're on Elixir 1.4). It now uses the official Registry (previously a custom one has been used). It also nests lock processes in the cache hierarchy (previously it was global). Perhaps both of these changes eliminate the problem.