Exception: Could not parse the result of the push health as json. Url: https://treeherder.mozilla.org/api/pr...

Sentry Issue: UPDATEBOT-PROD-1T

JSONDecodeError: Expecting value: line 2 column 1 (char 1)
  File "__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value: line 2 column 1 (char 1)
  File "apis/taskcluster.py", line 390, in get_push_health
    push_health = r.json()

Exception: Could not parse the result of the push health as json. Url: https://treeherder.mozilla.org/api/project/try/push/health/?revision=a6955c475a24c2acc1f6804ad1bf5713c8c0b0d9 Response:

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

(5 additional frame(s) were not displayed)
...
  File "/builds/worker/updatebot/tasktypes/vendoring.py", line 559, in _process_job_details_for_awaiting_retrigger_results
    no_build_failures, results, comment_lines = self._get_comments_on_push(library, existing_job)
  File "components/logging.py", line 38, in func_wrapper
    ret = func(*args, **kwargs)
  File "/builds/worker/updatebot/tasktypes/vendoring.py", line 307, in _get_comments_on_push
    this_push_health = self.taskclusterProvider.get_push_health(t.revision)
  File "components/logging.py", line 38, in func_wrapper
    ret = func(*args, **kwargs)
  File "apis/taskcluster.py", line 392, in get_push_health
    raise Exception("Could not parse the result of the push health as json. Url: %s Response:\n%s" % (push_health_url, r.text))

It looks like we expect JSON and do not properly handle sever errors. We should find out how we can (silently) retrigger this request in case of transient server errors. Distinguishing between transient and permanent server errors is likely going to be a bit of an issue that I'm not entirely sure how to tackle. Maybe we should keep track of retries until we reach a maximum?

mozilla-services / updatebot

Exception: Could not parse the result of the push health as json. Url: https://treeherder.mozilla.org/api/pr... #360