Feature: Using git bisect skip to ignore compilation errors

[PhilLab]

Depending on you commit policy, git bisect may have a hard time to identify which commit actually is the faulty one. Take this history:

1. Known good commit, passing tests
2. Good commit
3. Good commit but with compilation errors
4. Good commit but with compilation errors
5. Good commit fixing compilation of 3 and 4
6. bad commit causing a test failure git bisect should find
7. known bad commit, failing tests

In scenarios where you e.g. start a big refactoring job, you might actually have commits in-between where the build fails. But if you want to identify when a test started failed, you want to skip these. Otherwise, in the above git history, the Jenkins git blame plugin would return the third commit as the test jobs fails there.

So an option to ignore compilation failures but not test failures in the Jenkins git bisect plugin would be really helpful

[dorav]

There is an issue with input for the policy.

How does the plugin know if a job failed because of a build failure, or an actual failure?

Let me expand on your example with commits that build successfully but sanity tests don't pass. Or any other critical section breaks.

The good practice is to not allow such commits in the trunk, using a gated check-in process. But there are many systems that don't use such process. Also, there is an inherent problem with looking for problems in retrospect, you could not predict all failure cases.

I think that this is an important feature to implement, but I'm not sure how it should be implemented.

[pkpio]

This can be worked around by fixing the downstream job to report failure only when the tests fail. Most of these cases can be addressed by redefining the notion of failure in the downstream job.