search

bwarren2 / datadrivendota

Codebase for dota analytics
Other
0 stars 0 forks source link

Memcache throwing "Server Dead" errors? #505

Closed bwarren2 closed 8 years ago

bwarren2 commented 9 years ago

Ex:

Nov 14 10:02:10 datadrivendota app/web.1:  Traceback (most recent call last): 
Nov 14 10:02:10 datadrivendota app/web.1:    File "/app/.heroku/python/lib/python2.7/site-packages/django_pylibmc/memcached.py", line 124, in get 
Nov 14 10:02:10 datadrivendota app/web.1:      return super(PyLibMCCache, self).get(key, default, version) 
Nov 14 10:02:10 datadrivendota app/web.1:    File "/app/.heroku/python/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 84, in get 
Nov 14 10:02:10 datadrivendota app/web.1:      val = self._cache.get(key) 
Nov 14 10:02:10 datadrivendota app/web.1:    File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/api/datastore_trace.py", line 70, in _nr_datastore_trace_wrapper_ 
Nov 14 10:02:10 datadrivendota app/web.1:      return wrapped(*args, **kwargs) 
Nov 14 10:02:10 datadrivendota app/web.1:  Error: error 31 from memcached_get(:1:views.decorators.cache.cache_): (0x7fb734007d40) A TIMEOUT OCCURRED, (number_of == 0),  host: mc1.dev.ec2.memcachier.com:11211 -> libmemcached/connect.cc:133 
Nov 14 10:04:27 datadrivendota app/web.1:  ERROR:django.pylibmc:MemcachedError: error 35 from memcached_get(:1:views.decorators.cache.cache_): (0x7fb728005b80) SERVER IS MARKED DEAD,  host: mc1.dev.ec2.memcachier.com:11211 -> libmemcached/connect.cc:696

See papertrail.

bwarren2 commented 9 years ago

Variant error:

Nov 16 10:37:56 datadrivendota app/web.2:  Traceback (most recent call last): 
Nov 16 10:37:56 datadrivendota app/web.2:    File "/app/.heroku/python/lib/python2.7/site-packages/django_pylibmc/memcached.py", line 124, in get 
Nov 16 10:37:56 datadrivendota app/web.2:      return super(PyLibMCCache, self).get(key, default, version) 
Nov 16 10:37:56 datadrivendota app/web.2:    File "/app/.heroku/python/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 84, in get 
Nov 16 10:37:56 datadrivendota app/web.2:      val = self._cache.get(key) 
Nov 16 10:37:56 datadrivendota app/web.2:    File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/api/datastore_trace.py", line 70, in _nr_datastore_trace_wrapper_ 
Nov 16 10:37:56 datadrivendota app/web.2:      return wrapped(*args, **kwargs) 
Nov 16 10:37:56 datadrivendota app/web.2:  ServerDown: error 47 from memcached_get(:1:views.decorators.cache.cache_): (0x7eff80005710) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, mc1.dev.ec2.memcachier.com:11211,  host: mc1.dev.ec2.memcachier.com:11211 -> libmemcached/connect.cc:811
bwarren2 commented 9 years ago

@wlonk Do you know off the top of your head what these are? If not, I can dive in.

bwarren2 commented 9 years ago

I have opened a ticket with the memcachier folks about this.

wlonk commented 9 years ago

Nope, no idea. Looks like server reliability issues?