search

Bioconductor / BiocCheck

http://bioconductor.org/packages/BiocCheck
8 stars 26 forks source link

Reproducibility of timings in BiocCheck/Single Package Builder #81

Closed kokitsuyuzaki closed 4 years ago

kokitsuyuzaki commented 4 years ago

Hi,

When I tried to register an R package called scTensor to Bioconductor Contributions before, I was troubled by the rule of the Single Package Builder to finish BiocCheck in 5 minutes.

I also noticed that how long the calculation take depends on the specs and the computing situation of the machines performing BiocCheck.

For example, as follows, compared to the Linux/Mac machines, the Windows machine took about twice the time, and after git push of almost the same code over and over again, the check was passed. https://github.com/Bioconductor/Contributions/issues/884#issuecomment-436216127

For the above reason, I would like to suggest the following changes

1. Loosen the 5-minute rule (e.g., 10 minutes)
2. BiocCheck with cloud environment

Since there is a movement to execute the CI (GitHub Actions) for R packages (https://github.com/r-lib/actions/issues/84), if you make Single Package Builder based on this, the reproducibility to perform BiocCheck will be guaranteed.

Koki

lcolladotor commented 4 years ago

Hi!

Since I made https://github.com/r-lib/actions/issues/84 let me know if you have any questions.

Best, Leo

nturaga commented 4 years ago

@lshep will leave more thoughts about this issue. We think this might already be implemented.

lshep commented 4 years ago

The documentation should be updated - The BiocCheck is actually already extended out to 10 min on the builders (20 for windows because we test on the two arches) - We don't want to extend much more than that because if we are going to extend this to the daily builder besides the single package builder we would have to run it on all packages. We recognize that the timing will depend on the individual machine and load but realistically they shouldn't be drastically different and why they appear as warnings and can be argued or justified (except in the case of TIMINGOUT) but again the TIMEOUT is already fairly generous.

kokitsuyuzaki commented 4 years ago

10min on the builders

I didn't know that. Ok, 10 or 20 min sounds sufficient. Thank you.

lshep commented 4 years ago

I'll make sure to update the documentation so it is reflected.