[memcached] 500 error POSTing to REST API

[larsbutler]

Produced with version ca915156fb2ce4fe4356f54fb2cee7bd01185af5 of Swift, on Vagrant with DevStack. (See https://github.com/zerovm/zerocloud/pull/161.)

Here's the error, from the s-proxy console in DevStack:

Error: An error occurred: #012Traceback (most recent call last):#012  
File "/opt/stack/swift/swift/common/middleware/catch_errors.py", line 41, in handle_request#012    resp = self._app_call(env)#012  
File "/opt/stack/swift/swift/common/wsgi.py", line 578, in _app_call#012    resp = self.app(env, self._start_response)#012  
File "/opt/stack/swift/swift/common/middleware/gatekeeper.py", line 90, in __call__#012    return self.app(env, gatekeeper_response)#012  
File "/opt/stack/swift/swift/common/middleware/healthcheck.py", line 57, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/proxy_logging.py", line 289, in __call__#012    iterable = self.app(env, my_start_response)#012  
File "/opt/stack/swift/swift/common/middleware/memcache.py", line 85, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/swob.py", line 1291, in _wsgify_self#012    return func(self, Request(env))(env, start_response)#012  
File "/opt/stack/swift/swift/common/swob.py", line 1291, in _wsgify_self#012    return func(self, Request(env))(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/tempurl.py", line 295, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/ratelimit.py", line 291, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/crossdomain.py", line 82, in __call__#012    return self.app(env, start_response)#012  
File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 646, in __call__#012    return self._call_app(env, start_response)#012  
File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 624, in _call_app#012    return self._app(env, _fake_start_response)#012  
File "/opt/stack/swift/swift/common/middleware/keystoneauth.py", line 119, in __call__#012    return self.app(environ, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/tempauth.py", line 238, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/formpost.py", line 352, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/staticweb.py", line 479, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/swob.py", line 1291, in _wsgify_self#012    return func(self, Request(env))(env, start_response)#012  
File "/opt/stack/swift/swift/common/swob.py", line 1291, in _wsgify_self#012    return func(self, Request(env))(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/slo.py", line 749, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/dlo.py", line 282, in __call__#012    return self.app(env, start_response)#012  
File "/opt/stack/swift/swift/common/middleware/proxy_logging.py", line 220, in __call__#012    return self.app(env, start_response)#012  
File "/zerocloud-root/zerocloud/chain.py", line 83, in __call__#012    return context.handle_chain(env, start_response)#012  
File "/zerocloud-root/zerocloud/chain.py", line 25, in handle_chain#012    resp = self._app_call(env)#012  
File "/opt/stack/swift/swift/common/wsgi.py", line 578, in _app_call#012    resp = self.app(env, self._start_response)#012  
File "/opt/stack/swift/swift/common/swob.py", line 1291, in _wsgify_self#012    return func(self, Request(env))(env, start_response)#012  
File "/zerocloud-root/zerocloud/proxyquery.py", line 621, in __call__#012    res = handler(req)#012  
File "/opt/stack/swift/swift/proxy/controllers/base.py", line 96, in wrapped#012    return func(*a, **kw)#012  
File "/opt/stack/swift/swift/proxy/controllers/base.py", line 259, in wrapped#012    return func(*a, **kw)#012  
File "/zerocloud-root/zerocloud/proxyquery.py", line 1703, in POST#012    resp = self.handle_request(req)#012  
File "/zerocloud-root/zerocloud/proxyquery.py", line 1771, in handle_request#012    error = self.load_config(req, swift_path)#012  
File "/zerocloud-root/zerocloud/proxyquery.py", line 1883, in load_config#012    time=float(self.middleware.zerovm_cache_config_timeout))#012  
File "/opt/stack/swift/swift/common/memcached.py", line 287, in set#012    value = json.dumps(value)#012  
File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 366, in dumps#012    return _default_encoder.encode(obj)#012  
File "/usr/local/lib/python2.7/dist-packages/simplejson/encoder.py", line 271, in encode#012    chunks = list(chunks)#012  
File "/usr/local/lib/python2.7/dist-packages/simplejson/encoder.py", line 629, in _iterencode#012    for chunk in _iterencode_list(o, _current_indent_level):#012  
File "/usr/local/lib/python2.7/dist-packages/simplejson/encoder.py", line 481, in _iterencode_list#012    for chunk in chunks:#012  
File "/usr/local/lib/python2.7/dist-packages/simplejson/encoder.py", line 642, in _iterencode#012    o = _default(o)#012  
File "/usr/local/lib/python2.7/dist-packages/simplejson/encoder.py", line 246, in default#012    raise TypeError(repr(o) + " is not JSON serializable")#012TypeError: <zerocloud.common.SwiftPath instance at 0x2125f38> is not JSON serializable (txn: txa95f17d3111d404fb82e8-0054213430)

It seems like there's a problem storing a SwiftPath object in memcache. ZeroCloud expects it to be pickled, but DevStack is trying to use JSON instead for encoding. There is a hint about this in a Swift commit from two years ago.

I have no idea why it's broken now. Regardless, we should probably move to JSON encoding instead of relying on pickling.

This only appears to be a problem with REST APIs using api/1.0; services using open/1.0 seem to be unaffected.