rust-lang / homu

A bot that integrates with GitHub and your favorite continuous integration service
MIT License
182 stars 57 forks source link

GitHub Webhooks Delivery Instability #178

Open pinkforest opened 2 years ago

pinkforest commented 2 years ago

There are r+'s that Bors never receives and it has led to #174 but I thought it's best to raise a ticket for the general Webhook reliance

The fixes iterated on #174 - for commit mixups - will not solve this one - related Zulip thread here: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Bors.20is.20asleep.3F

Related recent PRs where Bors missed up webhook - no commit mixup involved tho - and no general outage involved

The plan I believe was to migrate to some GH feature preview on merge queue

However it will not solve general instability with Webhooks related to Homu or other bots that rely on WebHooks

Notably these occurences happen where there is no general outage.

I will adjust my analysis on to look for occurences where bors / olther bot never reacted when called to see impact over time.

I commented on GitHub Webhooks deliveries API but I wonder can we see these missed deliveries in there and that could be simply re-attempted in automated way? https://github.com/rust-lang/homu/issues/174#issuecomment-1172201585