connection retry/backoffs

fedora-infra / bodhi

Bodhi is a web-system that facilitates the process of publishing updates for a Fedora-based software distribution.

https://bodhi.fedoraproject.org

GNU General Public License v2.0

152 stars 194 forks source link

connection retry/backoffs #1143

Open nirik opened 7 years ago

nirik commented 7 years ago

Currently, when the bodhi masher is going it talks to several other systems. Whenever one of these things stops answering it almost immediately fails that thread.

These things include:

koji
kerberos server (but there may be more)

It would be nice if it would detect the connection failed and wait and retry with some backoff. We don't want it to keep trying forever, but if it tried even say 5 times at immediate/30seconds/60seconds/120seconds, etc it would help make it more robust.

nirik commented 7 years ago

Also, bugzilla breaks things:

bodhi-backend01.phx2.fedoraproject.org fedmsg-hub[14214]: [2017-03-13 23:58:13][ fedmsg ERROR]f25-updates MasherThread failed. Transaction rolled back. ... bodhi-backend01.phx2.fedoraproject.org fedmsg-hub[14214]: HTTPError: 502 Server Error: Proxy Error for url: https://bugzilla.redhat.com/xmlrpc.cgi

bowlofeggs commented 6 years ago

python-backoff could be useful for this.