search

debops / ansible-gitlab_runner

Manage GitLab Runner instances
GNU General Public License v3.0
20 stars 19 forks source link

500 errors while registering runners #7

Open sidewinder12s opened 7 years ago

sidewinder12s commented 7 years ago

I seem to be getting 500 errors somewhat randomly when I am running a playbook with serial: +1 (multiple runners are getting configured at the same time since it is running the tasks in parallel). I am not sure how prevalent it is, but I have seen it at least a few times with my GitLab instances though it is solved by just re-running the playbook.

Would be make sense to add some retry/wait logic around registering with GitLab or would it be better to just deal with it at the playbook level?

drybjed commented 7 years ago

Ansible only allows to set serial on the play level. I suppose that this could be set in the debops.gitlab_runner playbook by default if it works.

Did you try running with small amount of forks?

debops service/gitlab_runner --forks=1

Or, we could write a custom module to handle the registration and retry if necessary.

sidewinder12s commented 7 years ago

I didn't try the small amount of forks. I assume anything that limits the amount of runners that try to register at the same time would do as this seems like a GitLab Server issue though more of a corner case since most people won't be registering multiple runners at the same time. Maybe just a rescue block for the register to re-try/wait & retry?

drybjed commented 7 years ago

Sure, we could check if a rescue block would help. BTW, with what amount of hosts at once you see these issues? Are there any changes if you increase number of threads on the GitLab server?

sidewinder12s commented 7 years ago

This was with trying to register just 2 runners at the same time, just implementing the role separately from any other debops roles. I have not tried increasing the number of threads on the GitLab server since just re-running the playbook registered the second instance (So I've done no further troubleshooting 😄).

sidewinder12s commented 7 years ago

This could all also just be a bug in GitLab proper, but I haven't opened anything with them or looked at the production.log yet. I'll see if I can look at that next time I'm registering more runners.