search

smarkets / marge-bot

A merge-bot for GitLab
BSD 3-Clause "New" or "Revised" License
687 stars 134 forks source link

"Too Many Requests" #126

Open Jellby opened 5 years ago

Jellby commented 5 years ago

I got this log:

2018-08-31 16:23:55,992 INFO Waiting for CI to pass for MR !12
2018-08-31 18:11:34,929 ERROR Unexpected Exception
Traceback (most recent call last):
  File "/home/molcas-test/marge-bot/marge/single_merge_job.py", line 24, in execute
    self.update_merge_request_and_accept(approvals)
  File "/home/molcas-test/marge-bot/marge/single_merge_job.py", line 73, in update_merge_request_and_accept
    self.wait_for_ci_to_pass(merge_request, actual_sha)
  File "/home/molcas-test/marge-bot/marge/job.py", line 191, in wait_for_ci_to_pass
    ci_status = self.get_mr_ci_status(merge_request, commit_sha=commit_sha)
  File "/home/molcas-test/marge-bot/marge/job.py", line 129, in get_mr_ci_status
    pipelines = Pipeline.pipelines_by_branch(pid, ref, self._api)
  File "/home/molcas-test/marge-bot/marge/pipeline.py", line 29, in pipelines_by_branch
    params,
  File "/home/molcas-test/marge-bot/marge/gitlab.py", line 58, in call
    raise error(response.status_code, err_message)
marge.gitlab.UnexpectedError: (429, 'Too Many Requests')

as you can see, after waiting a couple of hours for the CI to finish (this is not a problem, the pipeline was running fine, it just takes a long time), marge got 429, 'Too Many Requests' from gitlab.com and crashed.

A better protection against this kind of error would be welcome, or maybe it's just a matter of increasing waiting_time_in_secs.

jcpetruzza commented 5 years ago

If gitlab sends a 'Retry-After' header in the 429 response, marge could then try to honour that

Jellby commented 5 years ago

I don't know if it does (I can't reproduce it), but could marge assume a default of say 60 seconds?

jcpetruzza commented 5 years ago

The current throttling behaviour is documented in one of the comments of this thread.