ocurrent / opam-repo-ci

An OCurrent pipeline for testing submissions to opam-repository
Apache License 2.0
20 stars 22 forks source link

Improve reliability of revdeps checks #388

Open shonfeder opened 3 weeks ago

shonfeder commented 3 weeks ago

When new package releases encounter revdeps errors, it can be really time consuming (and probably pretty unreliable) to try to audit the failures to see if the build/test failures have any relationship to the new release. We can use this issue to gather ideas on how to make the reported results more reliable and easier to navigate.

Two ideas to try to improve the relevance of revdeps errors:

Both of these could benefit from reviewer input during review time, allowing reviewers to mark revdeps failures as irrelevant, e.g., with the addition of a regex on certain errors, or by marking problem packages.

Another improvement that is already planned is to not report errors for packages that fail due to an intermediate dependency failure: e.g., if we have a dependency chain like A <- B <- C and B fails, we do not also want a report on the inevitable failure of C due to B.

maiste commented 3 weeks ago

Related to the context where I encountered this issue, I'm relatively biased here. This is definitely a great idea, and it would save time to maintainers. About the first proposition, the heuristics need to be added carefully to prevent broken releases to make their way to opam. It is easier to ignore CI errors than not having them reported.