For example, during a get_match_list run in a python shell with a large amount of IDs, a riot_api task that retries 3x after an initial 503 error only to get a 503 on each retry, execution will halt.
This may not affect things in normal operations (e.g. non-shell operation), but something needs to allow bulk operations like the above to continue even when Riot's servers are misbehaving.
Example log output follows:
l[2015-11-30 18:59:41,570: INFO/MainProcess] Received task: lol_stats2.celery.riot_api[84fffb7d-cec2-4180-9812-d436ea935b7b] eta:[2015-11-30 18:59:43.566973-05:00] u
x[2015-11-30 18:59:41,573: INFO/MainProcess] Task lol_stats2.celery.riot_api[84fffb7d-cec2-4180-9812-d436ea935b7b] retry: Retry in 2s: LoLException('Internal server error', <Response [500]>) x
m[2015-11-30 18:59:45,009: DEBUG/Worker-6] kwargs: {'season': None, 'ranked_queues': 'RANKED_SOLO_5x5', 'champion_ids': None, 'begin_index': None, 'region': 'na', 'begin_time': None, 'end_time': None, 'end_index': None, 'method': 'get_mx
`[2015-11-30 18:59:45,009] DEBUG [lol_stats2.celery:76] riot_api: kwargs: {'season': None, 'ranked_queues': 'RANKED_SOLO_5x5', 'champion_ids': None, 'begin_index': None, 'region': 'na', 'begin_time': None, 'end_time': None, 'end_index':x
l[2015-11-30 18:59:45,011: INFO/Worker-6] Starting new HTTPS connection (1): na.api.pvp.net x
x[2015-11-30 18:59:45,969: ERROR/Worker-6] LoLException occurred, Internal server error u
xTraceback (most recent call last): x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api x
x except LoLException as e: x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list x
x endIndex=end_index x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request x
x **kwargs x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request u
x raise_status(r) x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 212, in raise_status x
x raise LoLException(error_500, response) u
mriotwatcher.riotwatcher.LoLException: Internal server error u
l[2015-11-30 18:59:45,969] ERROR [lol_stats2.celery:97] riot_api: LoLException occurred, Internal server error x
xTraceback (most recent call last): x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api u
x except LoLException as e: u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list x
x endIndex=end_index u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request x
x **kwargs x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request x
x raise_status(r) u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 212, in raise_status x
x raise LoLException(error_500, response) x
mriotwatcher.riotwatcher.LoLException: Internal server error
l[2015-11-30 18:59:45,970] ERROR [lol_stats2.celery:108] riot_api: 500 error, retrying in 2 sec u
xTraceback (most recent call last): x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api x
x except LoLException as e: x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list x
x endIndex=end_index x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request u
x **kwargs u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request x
x raise_status(r) x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 212, in raise_status x
x raise LoLException(error_500, response) x
mriotwatcher.riotwatcher.LoLException: Internal server error u
l[2015-11-30 18:59:45,973: INFO/MainProcess] Received task: lol_stats2.celery.riot_api[84fffb7d-cec2-4180-9812-d436ea935b7b] eta:[2015-11-30 18:59:47.971452-05:00] x
x[2015-11-30 18:59:45,975: INFO/MainProcess] Task lol_stats2.celery.riot_api[84fffb7d-cec2-4180-9812-d436ea935b7b] retry: Retry in 2s: LoLException('Internal server error', <Response [500]>) x
m[2015-11-30 18:59:49,012: DEBUG/Worker-10] kwargs: {'season': None, 'ranked_queues': 'RANKED_SOLO_5x5', 'champion_ids': None, 'begin_index': None, 'region': 'na', 'begin_time': None, 'end_time': None, 'end_index': None, 'method': 'get_x
`[2015-11-30 18:59:49,012] DEBUG [lol_stats2.celery:76] riot_api: kwargs: {'season': None, 'ranked_queues': 'RANKED_SOLO_5x5', 'champion_ids': None, 'begin_index': None, 'region': 'na', 'begin_time': None, 'end_time': None, 'end_index':x
l[2015-11-30 18:59:49,014: INFO/Worker-10] Starting new HTTPS connection (1): na.api.pvp.net u
x[2015-11-30 18:59:49,456: ERROR/Worker-10] LoLException occurred, Service unavailable x
xTraceback (most recent call last): x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api x
x except LoLException as e: x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list x
x endIndex=end_index x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request x
x **kwargs u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request x
x raise_status(r) x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 214, in raise_status u
x raise LoLException(error_503, response) u
mriotwatcher.riotwatcher.LoLException: Service unavailable x
l[2015-11-30 18:59:49,456] ERROR [lol_stats2.celery:97] riot_api: LoLException occurred, Service unavailable x
xTraceback (most recent call last): u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api u
x except LoLException as e: x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list u
x endIndex=end_index x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request x
x **kwargs x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request u
x raise_status(r) x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 214, in raise_status x
x raise LoLException(error_503, response) x
mriotwatcher.riotwatcher.LoLException: Service unavailable
l[2015-11-30 18:59:49,457: ERROR/Worker-10] 503 error, retrying in 2 sec x
xTraceback (most recent call last): x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api u
x except LoLException as e: x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list x
x endIndex=end_index x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request u
x **kwargs x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request x
x raise_status(r) x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 214, in raise_status x
x raise LoLException(error_503, response) x
mriotwatcher.riotwatcher.LoLException: Service unavailable u
l[2015-11-30 18:59:49,457] ERROR [lol_stats2.celery:113] riot_api: 503 error, retrying in 2 sec u
xTraceback (most recent call last): x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api x
x except LoLException as e: x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list x
x endIndex=end_index u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request x
x **kwargs x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request x
x raise_status(r) x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 214, in raise_status u
x raise LoLException(error_503, response) x
mriotwatcher.riotwatcher.LoLException: Service unavailable x
l[2015-11-30 18:59:49,462: ERROR/MainProcess] Task lol_stats2.celery.riot_api[84fffb7d-cec2-4180-9812-d436ea935b7b] raised unexpected: LoLException('Service unavailable', <Response [503]>) x
xTraceback (most recent call last): x
x File "/home/chen/.virtualenvs/lol_stats2_dev/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task x
x R = retval = fun(*args, **kwargs) x
x File "/home/chen/.virtualenvs/lol_stats2_dev/lib/python3.4/site-packages/celery/app/trace.py", line 438, in __protected_call__ x
x return self.run(*args, **kwargs) u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 116, in riot_api x
x raise self.retry(exc=e) x
x File "/home/chen/.virtualenvs/lol_stats2_dev/lib/python3.4/site-packages/celery/app/task.py", line 670, in retry x
x maybe_reraise() u
x File "/home/chen/.virtualenvs/lol_stats2_dev/lib/python3.4/site-packages/celery/utils/__init__.py", line 242, in maybe_reraise x
x reraise(exc_info[0], exc_info[1], exc_info[2]) x
x File "/home/chen/.virtualenvs/lol_stats2_dev/lib/python3.4/site-packages/celery/five.py", line 91, in reraise u
x raise value u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/lol_stats2/celery.py", line 93, in riot_api x
x except LoLException as e: u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 538, in get_match_list x
x endIndex=end_index x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 524, in _match_list_request x
x **kwargs u
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 269, in base_request x
x raise_status(r) x
x File "/home/chen/python-projects/lol_stats2/lol_stats2/riotwatcher/riotwatcher.py", line 214, in raise_status x
x raise LoLException(error_503, response) x
xriotwatcher.riotwatcher.LoLException: Service unavailable
chenwardT
commented
8 years ago
For example, during a
get_match_listrun in a python shell with a large amount of IDs, ariot_apitask that retries 3x after an initial 503 error only to get a 503 on each retry, execution will halt.This may not affect things in normal operations (e.g. non-shell operation), but something needs to allow bulk operations like the above to continue even when Riot's servers are misbehaving.
Example log output follows: