search

azavea / climate-change-api

Apache License 2.0
0 stars 0 forks source link

memcached_set fails with error 37: SUCCESS #286

Closed rmartz closed 7 years ago

rmartz commented 7 years ago

When attempting to access some resources on the API, the server will respond with a 500 error. This appears to be because of an exception thrown by Memcache, but the error claims it succeeded:

Feb 22 09:10:40 52.202.188.214 django:  Internal Server Error: /api/climate-data/8/RCP85/
Feb 22 09:10:40 52.202.188.214 django:  Traceback (most recent call last):
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
Feb 22 09:10:40 52.202.188.214 django:      response = self.process_exception_by_middleware(e, request)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
Feb 22 09:10:40 52.202.188.214 django:      response = wrapped_callback(request, *callback_args, **callback_kwargs)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
Feb 22 09:10:40 52.202.188.214 django:      return view_func(*args, **kwargs)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
Feb 22 09:10:40 52.202.188.214 django:      return self.dispatch(request, *args, **kwargs)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 474, in dispatch
Feb 22 09:10:40 52.202.188.214 django:      response = self.handle_exception(exc)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 434, in handle_exception
Feb 22 09:10:40 52.202.188.214 django:      self.raise_uncaught_exception(exc)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 471, in dispatch
Feb 22 09:10:40 52.202.188.214 django:      response = handler(request, *args, **kwargs)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/rest_framework_extensions/cache/decorators.py", line 48, in inner
Feb 22 09:10:40 52.202.188.214 django:      kwargs=kwargs,
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/rest_framework_extensions/cache/decorators.py", line 72, in process_cache_response
Feb 22 09:10:40 52.202.188.214 django:      self.cache.set(key, response, self.timeout)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/django_elasticache/memcached.py", line 18, in wrapper
Feb 22 09:10:40 52.202.188.214 django:      return f(self, *args, **kwds)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/django_elasticache/memcached.py", line 109, in set
Feb 22 09:10:40 52.202.188.214 django:      return super(ElastiCache, self).set(*args, **kwargs)
Feb 22 09:10:40 52.202.188.214 django:    File "/usr/local/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 83, in set
Feb 22 09:10:40 52.202.188.214 django:      if not self._cache.set(key, value, self.get_backend_timeout(timeout)):
Feb 22 09:10:40 52.202.188.214 django:  Error: error 37 from memcached_set: SUCCESS
CloudNiner commented 7 years ago

This is the pylibmc error for attempting to set too large of a value into the cache.

Currently our cache value size limit is set to 8mb: https://github.com/azavea/climate-change-api/blob/develop/docker-compose.yml#L37

I thought this setting was also set via a setting in the elasticache memcached parameter group, but I don't see it there ATM. It looks like that got lost in the refactor here: https://github.com/azavea/climate-change-api/pull/264/files#diff-66d1d41d2fb4d0845b5a67e53f5cca21L59

So a first pass attempt at this issue would be to re-add that config parameter, setting it to 8mb.

http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ParameterGroups.Memcached.html#ParameterGroups.Memcached.V1-4-5