ASFHyP3 / hyp3

A processing environment for HyP3 Plugins in AWS.
BSD 3-Clause "New" or "Revised" License
37 stars 8 forks source link

/user requests fail with "504 Server Error: Gateway Timeout" for users who have submitted many past jobs #1312

Open asjohnston-asf opened 2 years ago

asjohnston-asf commented 2 years ago

Jira: https://asfdaac.atlassian.net/browse/TOOL-2041

The /user endpoint returns a list of all job names ever submitted by the logged in user. For users that have submitted many (hundreds of thousands) of past jobs, the query for past job names can take longer than the 30 second API Gateway Integration timeout, resulting in a HTTP 504 Gateway Timeout error being returned to the user.

The troublesome query is get_names_for_user() invoked by the /user handler at https://github.com/ASFHyP3/hyp3/blob/develop/apps/api/src/hyp3_api/handlers.py#L84

Here is an example traceback generated for this error when calling hyp3_sdk.hyp3.my_info() as reported by @cirrusasf :

Traceback (most recent call last):
  File "/home/jiangzhu/apps/anaconda3/envs/its-live-operations/lib/python3.10/site-packages/hyp3_sdk/exceptions.py", line 29, in _raise_for_hyp3_status
    response.raise_for_status()
  File "/home/jiangzhu/apps/anaconda3/envs/its-live-operations/lib/python3.10/site-packages/requests/models.py", line 1022, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: https://hyp3-its-live.asf.alaska.edu/user

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jiangzhu/apps/anaconda3/envs/its-live-operations/lib/python3.10/site-packages/hyp3_sdk/hyp3.py", line 410, in my_info
    _raise_for_hyp3_status(response)
  File "/home/jiangzhu/apps/anaconda3/envs/its-live-operations/lib/python3.10/site-packages/hyp3_sdk/exceptions.py", line 33, in _raise_for_hyp3_status
    raise ServerError
hyp3_sdk.exceptions.ServerError
jtherrmann commented 5 months ago

Proposed implementation: