Our github API requests do not get retried on failure. In one instance, a temporary network blip prevented Zuul from updating the status after a check job finished. From the PR POV, the job never completed and was still running (a yellow pending button on the check). The user was left wondering what was going on since the Zuul status page correctly did not report it as running. We should probably retry API commands that fail for network related issues, and probably some others?
There's a newer library in use in openstack that may make this super easy, just wrap the functions in connection/github.py https://github.com/jd/tenacity
Our github API requests do not get retried on failure. In one instance, a temporary network blip prevented Zuul from updating the status after a check job finished. From the PR POV, the job never completed and was still running (a yellow pending button on the check). The user was left wondering what was going on since the Zuul status page correctly did not report it as running. We should probably retry API commands that fail for network related issues, and probably some others?
There's a newer library in use in openstack that may make this super easy, just wrap the functions in connection/github.py https://github.com/jd/tenacity