Packages with intermittent failures

[hadley]

One challenge when submitting a package to CRAN with many revdeps is that some packages will fail because their tests are flaky and fail stochastically (typically due to reliance on a flaky external resource like a server). These packages also make extra work when making changes to R itself, since each failure has to be manually inspected to confirm it's a false positive.

Could the WG figure out some way to help improve these packages?

[llrs]

Some of these problems might also come from using a different seed not only requesting a resource. But, are you suggesting a new check for R CMD check or a way to detect when such cases are random?

If the later it might be relatively easy to monitor which packages haven't had any change on them or dependencies and reported an error on the release and/or old release of R. Probably this can be done via tools::CRAN_check_results and tools::package_dependencies(, reverse = TRUE).

[llrs]

I created a website that should update every day with the packages that have intermittent failures, and there hasn't been any change on the package or its dependencies: https://llrs.github.io/intermittent_failures/ with the list of packages with intermittent issues at the end.

I haven't made it to store permanently the packages with flaky code (check https://github.com/llrs/intermittent_failures/issues/1) but that could be an easy addition.

[hadley]

Nice, thanks for working on this!

[llrs]

Glad to help! Is this something similar to what you had in mine when you opened the issue?

[hadley]

Yeah, I mostly want issue 1 so we have a historical source of data for further analysis.

[llrs]

Ok, I thought that was just for specific updates of R or submission to CRAN. A historic view of the errors can be done via the cchecks package and the cranchecks API.

[hadley]

I thought R CMD check was re-run more often than that. My recollection is that it's run every 24 hours on at least windows and linux, so I'm most interested in tracking those changes over time (in order to detect intermittent failures).

[llrs]

cchecks::cch_pkgs_history reports check data updated daily back to 30 days ago and cchecks::cch_history returns checks for all packages back to 2018-12-20, I think there is enough data to detect those packages and there is no need for a new package or way to retrieve that data.

However, I think just knowing which packages have done so in the past doesn't get us closer to how to improve those packages that fail intermittently. I thought that page could help the CRAN team and package maintainers but at the moment I don't know how else we could help them.

[hadley]

At least it helps me to know which packages fail intermittently when I run a revdep check; then I don’t need to spend more time looking into the failure.

To help others, we could investigate more the causes for the underlying failures and put together some docs on common causes.