search

gholt / swauth

This is the historical location of Swauth; active development is now at https://github.com/openstack/swauth
45 stars 29 forks source link

I am getting "503 server unavailable" and "503 internal server error" while uploading large amount of files (small or big) simultaneously. #32

Closed robbinparrish closed 12 years ago

robbinparrish commented 12 years ago

Essex with Swauth support On CentOS6 with single node.

May 17 23:38:52 localhost proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last):#012 File "/root/swauth/middleware.py", line 404, in handle#012 return self.handle_request(req)(env, start_response)#012 File "/root/swauth/middleware.py", line 471, in handle_request#012 req.response = handler(req)#012 File "/root/swauth/middleware.py", line 1183, in handle_gettoken#012 'exists: %s %s' % (path, resp.status))#012Exception: Could not detect whether a token already exists: /v1/AUTH.auth/.token_d/AUTH_tk69d81d080626426bb6c2b2aa8d2e267d 503 Internal Server Error#012: {'SCRIPT_NAME': '/auth', 'webob.adhoc_attrs': {'start_time': 1337278117.1463921, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/v1.0', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': '', 'HTTP_X_STORAGE_PASS': 'password', 'HTTP_USER_AGENT': '' '', 'HTTP_CONNECTION': 'Keep-Alive', 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swauth.middleware.Swauth object at 0x145f250>>, (<Request at 0x6461cd0 GET http://192.168.1.10/auth/v1.0>,), {})], 'SERVER_NAME': '192.168.1.10', 'REMOTE_ADDR': '192.168.1.167', 'eventlet.input': <eventlet.wsgi.Input object at 0x2f4ae90>, 'HTTP_X_STORAGE_USER': 'one:one', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '443', 'wsgi.input': <eventlet.wsgi.Input object at 0x2f4ae90>, 'HTTP_HOST': '192.168.1.10', 'wsgi.multithread': True, 'HTTP_ACCEPT': 'application/xml', 'wsgi.version': (1, 0), 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x1436590>, 'wsgi.multiprocess': False, 'CONTENT_TYPE': None, 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x145f410>} May 17 23:39:40 localhost proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last):#012 File "/root/swauth/middleware.py", line 404, in handle#012 return self.handle_request(req)(env, start_response)#012 File "/root/swauth/middleware.py", line 471, in handle_request#012 req.response = handler(req)#012 File "/root/swauth/middleware.py", line 1223, in handle_gettoken#012 (path, resp.status))#012Exception: Could not obtain services info: /v1/AUTH.auth/umesh/.services 503 Internal Server Error#012: {'SCRIPT_NAME': '/auth', 'webob.adhoc_attrs': {'start_time': 1337278155.8839321, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/v1.0', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': '', 'HTTP_X_STORAGE_PASS': 'password', 'HTTP_USER_AGENT': '' '', 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swauth.middleware.Swauth object at 0x145f250>>, (<Request at 0x4514c50 GET http://192.168.1.10/auth/v1.0>,), {})], 'SERVER_NAME': '192.168.1.10', 'REMOTE_ADDR': '192.168.1.167', 'eventlet.input': <eventlet.wsgi.Input object at 0x4863e10>, 'HTTP_X_STORAGE_USER': 'one:one', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '443', 'wsgi.input': <eventlet.wsgi.Input object at 0x4863e10>, 'HTTP_HOST': '192.168.1.10', 'wsgi.multithread': True, 'HTTP_ACCEPT': 'application/xml', 'wsgi.version': (1, 0), 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x1436590>, 'wsgi.multiprocess': False, 'CONTENT_TYPE': None, 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x145f410>} May 17 23:40:32 localhost proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last):#012 File "/root/swauth/middleware.py", line 404, in handle#012 return self.handle_request(req)(env, start_response)#012 File "/root/swauth/middleware.py", line 471, in handle_request#012 req.response = handler(req)#012 File "/root/swauth/middleware.py", line 1162, in handle_gettoken#012 (path, resp.status))#012Exception: Could not obtain user details: /v1/AUTH.auth/umesh/umesh 503 Internal Server Error#012: {'SCRIPT_NAME': '/auth', 'webob.adhoc_attrs': {'start_time': 1337278232.3465359, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/v1.0', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': '', 'HTTP_X_STORAGE_PASS': 'password', 'HTTP_USER_AGENT': '' '', 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swauth.middleware.Swauth object at 0x145f250>>, (<Request at 0x1a8d150 GET http://192.168.1.10/auth/v1.0>,), {})], 'SERVER_NAME': '192.168.1.10', 'REMOTE_ADDR': '192.168.1.167', 'eventlet.input': <eventlet.wsgi.Input object at 0x70086d0>, 'HTTP_X_STORAGE_USER': 'one:one', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '443', 'wsgi.input': <eventlet.wsgi.Input object at 0x70086d0>, 'HTTP_HOST': '192.168.1.10', 'wsgi.multithread': True, 'HTTP_ACCEPT': 'application/xml', 'wsgi.version': (1, 0), 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x1436590>, 'wsgi.multiprocess': False, 'CONTENT_TYPE': None, 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x145f410>}

gholt commented 12 years ago

The "Could not detect whether a token already exists: /v1/AUTH.auth/.tokend/AUTH_tk69d81d080626426bb6c2b2aa8d2e267d 503 Internal Server Error" and the other "Could not ... 503 ..." indicates that the Swift cluster itself is returning 503s to the Swauth middleware. There should probably be log entries somewhere from Swift indicating what problems it's having.

robbinparrish commented 12 years ago

Thanks for reply. Yes its also showing timeout (Timeout (10s). But it only happened with high load.

gholt commented 12 years ago

Ah okay, makes perfect sense then.