search

kostmo / circleci-failure-tracker

A log analyzer for CircleCI. Note that this project is now hosted at pytorch/dr-ci
https://github.com/pytorch/dr-ci
5 stars 2 forks source link

Mandate a rebase if upstream failures have since been fixed in master #82

Open kostmo opened 4 years ago

kostmo commented 4 years ago

Currently, Dr. CI indicates that failed CI jobs in one's PR that are also failing upstream at the merge base are "not your fault". Developers currently take this as license to merge.

However, this runs the risk of the upstream failure masking newly-introduced failures of the same CI job. A possible example is here: https://github.com/pytorch/pytorch/pull/36290#issuecomment-613240682

In this instance, Dr. CI helpfully suggested (also see image below) that the failures were "not your fault", but the merged PR went on to break the same job in master.

Sometimes the upstream failure is not fixed by the time the developer needs to merge their PR (due to time pressure). However, if the upstream failure has indeed been fixed, we should indicate this to the user and insist that they rebase. More technically, Dr. CI should not interpret the CI results as "qualified green" if any of the upstream breakages have been subsequently fixed.

Screen Shot 2020-04-13 at 11 02 54 PM
ezyang commented 4 years ago

Yes, please do this. This brings us inline with fbcode where we distinguish between "Green" and "No Signal". If there is too much "No Signal" we should ask to rebase.