Closed HughParsonage closed 6 years ago
This is great -- thanks for this! 148 spurious package installation errors to be worked through then! Welcome to my world. I tend to get down to under 10 quite quickly. Then the last 4 or 5 can take a few days to resolve. You're on Windows though which should help a lot given compiled packages. (On Ubuntu, I have to compile them all!)
There are lots of errors like this one :
"Packages required but not available: 'MassSpecWavelet' 'impute'"
Which command did you use to run this, and why doesn't it install the required packages?
The code I use is here : https://github.com/Rdatatable/data.table/blob/master/CRAN_Release.cmd#L310
I can see two non-spurious by the looks of it, both fread related. So there's at least some work to do then: either communication with those maintainers, or fixes to avoid the breakage.
This was simply devtools::revdep(dependencies = c("Depends", "Imports"))
. I think those packages are on Bioconductor. Sometimes a package just fails too: connection drops out at an inopportune time.
I'll go through it and keep culling the spurious ones: some other main ones I see are due to ITime
and ..x
.
Bravo sir. I've got some time I could spare today (Singapore time = GMT+8) if you'd like an extra pair of hands/eyes 👍
Sure! You go from the bottom? My plan is to just edit the issue OP until only the genuine ones remain.
I ran the code in CRAN_Release.cmd overnight and got this :
> status()
CRAN:
ERROR : 16 : AFM easycsv expss fst iml mlr PhenotypeSimulator popEpi prediction sdcMicro SIRItoGTFS socialmixr SpaDES.core splitstackshape tstools yCrypticRNAs
WARNING : 2 : optiSel xgboost
NOTE : 148
OK : 307
TOTAL : 473 / 473
RUNNING : 0 :
BIOC:
ERROR : 30 : CAGEr ccmap cellbaseR ChIPexoQual COMPASS CONFESS crossmeta CytoML fgsea flowWorkspace genomation ggcyto GGtools GOTHiC ImmuneSpaceR LowMACA maftools metagene metavizr methylPipe minfi MinimumDistance pepStat PGA psichomics rTANDEM RTCGA SNPhood TCGAbiolinks VanillaICE
WARNING : 56 : ABAEnrichment amplican anamiR AUCell basecallQC BASiCS biobroom BPRMeth branchpointer bsseq CEMiTool Chicago CopywriteR CRISPRseek dada2 DEFormats diffloop geneXtendeR GenoGAM GenomicDataCommons GenomicTuples GUIDEseq iGC IrisSpatialFeatures LOLA MAST MethylMix MIGSA MIRA MSnID MSstats OmicsMarkeR Onassis OncoSimulR openCyto paxtoolsr phyloseq Prostar PureCN Pviz QUALIFIER R3CPET r3Cseq rfPred RiboProfiling RTN S4Vectors scater SCnorm SISPA SWATH2stats TCGAbiolinksGUI TitanCNA TPP transcriptogramer yamss
NOTE : 23
OK : 2
TOTAL : 111 / 111
RUNNING : 0 :
Took 7.5 hours using 8 cores on laptop. The error rate on CRAN is very low. Given that, the Bioc result is likely mostly spurious. Will look.
Logs from the 18 CRAN fails attached :
find . -name 00check.log | grep -E
'AFM|easycsv|expss|fst|iml|mlr|PhenotypeSimulator|popEpi|prediction|sdcMicro|SIRItoGTFS|socialmixr|SpaDES.core|splitstackshape|tstools|yCrypticRNAs|optiSel|xgboost'
| xargs grep -H . > /tmp/out.log
I installad CRAN version data.table_1.10.4-3 and reran CRAN fails.
> status()
CRAN:
ERROR : 5 : AFM expss mlr prediction yCrypticRNAs
WARNING : 2 : optiSel xgboost
NOTE : 154
OK : 310
TOTAL : 471 / 471
RUNNING : 0 :
NB: 2 packages removed from CRAN overnight (readability
and syllable
) hence 473 => 471.
So those 7 are either spurious local fails to be resolved, or already in fail status on CRAN.
That leaves 18-7 = 11 CRAN packages that are broken by the data.table update. 11/471 = 2.3%.
( Ticked means it's either resolved on data.table side, or the maintainer has been been contacted. )
easycsv
-- see below; discussion issue filed https://github.com/bogind/easycsv/issues/1fst
-- Problems appear resolved in fst
-dev https://github.com/fstpackage/fstiml
-- filed https://github.com/christophM/iml/issues/11PhenotypeSimulator
-- filed https://github.com/HannahVMeyer/PhenotypeSimulator/issues/7popEpi
-- filed https://github.com/WetRobot/popEpi/issues/144 and https://github.com/WetRobot/popEpi/pull/145sdcMicro
-- filed https://github.com/sdcTools/sdcMicro/issues/253SIRItoGTFS
-- same as easycsv
socialmixr
-- fixed in data.table: PR #2805 SpaDES.core
-- submitted PR https://github.com/PredictiveEcology/SpaDES.core/pull/64splitstackshape
-- filed https://github.com/mrdwab/splitstackshape/issues/61 and https://github.com/mrdwab/splitstackshape/pull/62tstools
-- fixed in data.table: PR #2806 For both easycsv
and SIRItoGTFS
, the error comes from changing showProgress = getOption("datatable.showProgress")
to showProgress = interactive()
(more specifically from, deleting the default datatable.showProgress
argument) in fread
. I don't see this change mentioned in the NEWS
, perhaps this should be remedied before notifying easycsv
and SIRItoGTFS
Issue trackers (note: both are the same author).
I'm also not seeing anything in NEWS
that would suggest the change to sdcMicro
, which was relying on being able to assing to a data.table
with no columns. I recall this being discussed but don't remember the specific issue.
HannahVMeyer/PhenotypeSimulator#7 is due to a breaking change/regression in {data.table}: specifying skip='string'
would previously skip lines up to and including the line with the specified string. As of 1.10.5, that line itself isn’t skipped. I can file a new issue with an MWE, if desired.
@klmr thanks. Based on verbose
output it looks like fread
attempts to determine whether that row is a header
using the auto-detect rules that have been around for a while. It's hinted at in the NEWS
but I don't see the specific issue:
People appear to be using
skip="string"
orskip=nrow
to find the header row exactly, which is retained and simpler.
Filed #2786
fst
looks like a genuine change:
# fst
library(fst)
library(data.table)
# Convenience method to tests on different versions of the package
# Not used in the CRAN release
fstwriteproxy <- function(x, path, compress = 0, uniform.encoding = TRUE) {
write_fst(x, path, compress, uniform.encoding) # use current version of fst package
}
fstreadproxy <- function(path, columns = NULL, from = 1, to = NULL, as.data.table = FALSE) {
read_fst(path, columns, from, to, as.data.table)
}
fstmetaproxy <- function(path) {
metadata_fst(path)
}
roundtrip_vector <- function(x) {
df <- data.frame(x = x, stringsAsFactors = FALSE)
roundtrip(df)$x
}
roundtrip <- function(df) {
temp <- tempfile()
fstwriteproxy(df, temp)
on.exit(unlink(temp))
fstreadproxy(temp)
}
roundtrip_vector_dt <- function(x) {
dt <- data.table(x = x, stringsAsFactors = FALSE)
roundtrip(dt)$x
}
# ITime
test_that("preserves time of day", {
x <- as.ITime(c(1:10, NA), origin = "1970-01-01")
res <- roundtrip_vector_dt(x)
expect_identical(res, x)
mode(x) <- "double"
res <- roundtrip_vector_dt(x)
expect_identical(res, x)
})
# Error: Test failed: 'preserves time of day'
# * `res` not identical to `x`.
# Classes differ: integer is not ITime
# * `res` not identical to `x`.
# Classes differ: numeric is not ITime
@HughParsonage I get a different error?
Error: Test failed: 'preserves time of day'
* `res` not identical to `x`.
target is NULL, current is ITime
* `res` not identical to `x`.
target is NULL, current is ITime
Yes difference in versions. Looks okay now.
fst | data.table | result |
---|---|---|
cran | cran | X |
cran | dev | X |
dev | cran | X |
dev | dev | OK |
For SpaDES.core
I can't work out what the error is, I get a node overflow error
due to testthat::capture_warnings
, but I have no idea why.
@HughParsonage make extra check to confirm that error is not present on 1.10.4, if it happens there then it is likely not the issue on our side, but other package dependency.
Fixed for the iml
package: https://github.com/christophM/iml/issues/11
Will submit to CRAN the next days
Current status :
CRAN:
ERROR : 11 : easycsv expss fst mlr PhenotypeSimulator popEpi prediction sdcMicro SIRItoGTFS SpaDES.core splitstackshape
WARNING : 5 : emil optiSel r2glmm simPH xgboost
NOTE : 139
OK : 320
TOTAL : 475 / 475
RUNNING : 0 :
BIOC:
ERROR : 8 : COMPASS CytoML ImmuneSpaceR LowMACA maftools MinimumDistance PGA VanillaICE
WARNING : 16 : CEMiTool CONFESS crossmeta IrisSpatialFeatures MethylMix methylPipe MSstats openCyto QUALIFIER R3CPET RiboProfiling rTANDEM S4Vectors SISPA TCGAbiolinksGUI TitanCNA
NOTE : 74
OK : 13
TOTAL : 111 / 111
RUNNING : 0 :
All of the CRAN 11+5 are either fixed in data.table, or communicated. Yay! BIOC 8+16 are unrelated to data.table, other than the following 6.
CRAN rev dep checks match our results. There were two packages (batchtools and metaMix) that unexpectedly failed with the update, but those turned out to be unrelated to data.table. v1.11.0 is now on CRAN.
Setup
Runtime: about 7 hrs. Most of the warnings are due to me not updating R to 3.4.4, rather than significant warnings. May filter later, but wanted to save in case I lost.
Platform
|setting |value | |:--------|:----------------------------| |version |R version 3.4.3 (2017-11-30) | |system |x86_64, mingw32 | |ui |RStudio (1.1.423) | |language |(EN) | |collate |English_Australia.1252 | |tz |Australia/Sydney | |date |2018-04-21 |Packages
|package |* |version |date |source | |:----------|:--|:-------|:----------|:--------------------------------| |bit64 |* |0.9-7 |2017-05-08 |CRAN (R 3.4.1) | |chron |* |2.3-52 |2018-01-06 |CRAN (R 3.4.4) | |curl | |3.2 |2018-03-28 |CRAN (R 3.4.4) | |data.table |* |1.10.5 |2018-04-21 |local (Rdatatable/data.table@NA) | |knitr | |1.20 |2018-02-20 |CRAN (R 3.4.4) | |reshape2 | |1.4.3 |2017-12-11 |CRAN (R 3.4.3) | |testthat | |2.0.0 |2017-12-13 |CRAN (R 3.4.3) | |xts |* |0.10-2 |2018-03-14 |CRAN (R 3.4.4) | |zoo |* |1.8-1 |2018-01-08 |CRAN (R 3.4.4) |Check results
148 packages with problems |package |version | errors| warnings| notes| |:------------------|:----------|------:|--------:|-----:| |annovarR |1.0.0 | 1| 0| 1| |antaresProcessing |0.15.1 | 0| 1| 1| |BatchExperiments |1.4.2 | 1| 1| 0| |BatchJobs |1.7 | 0| 1| 0| |BGData |1.0.0 | 0| 1| 0| |biomartr |0.7.0 | 1| 0| 0| |brainGraph |2.0.2 | 1| 0| 0| |bupaR |0.4.0 | 0| 1| 0| |BuyseTest |1.0 | 0| 1| 1| |Causata |4.2-0 | 0| 1| 4| |cdparcoord |1.0.0 | 0| 1| 0| |cffdrs |1.7.6 | 0| 1| 0| |classifierplots |1.3.3 | 0| 1| 0| |clickstream |1.3.0 | 0| 1| 0| |corpustools |0.3.3 | 0| 1| 1| |corregp |2.0.2 | 0| 1| 0| |ctmm |0.4.1 | 0| 1| 0| |cvAUC |1.1.0 | 0| 1| 1| |dartR |1.0 | 1| 0| 0| |dat |0.4.0 | 0| 1| 0| |dataPreparation |0.3.5 | 0| 1| 0| |DCD |0.1.0 | 1| 0| 0| |dfmeta |1.0.0 | 0| 1| 0| |dgo |0.2.14 | 0| 1| 0| |DGVM3D |1.0.0 | 0| 1| 0| |Diderot |0.12 | 0| 1| 0| |difconet |1.0-4 | 0| 1| 0| |drgee |1.1.6 | 0| 1| 1| |dynamichazard |0.5.1 | 1| 0| 2| |easycsv |1.0.7 | 1| 0| 0| |eclust |0.1.0 | 1| 1| 2| |edgeRun |1.0.9 | 1| 0| 0| |EdSurvey |1.0.6 | 0| 1| 0| |eeptools |1.1.1 | 0| 1| 0| |eiCompare |2.1 | 0| 1| 0| |EthSEQ |2.1.2 | 1| 0| 0| |EurosarcBayes |1.1 | 0| 1| 0| |EventStudy |0.34 | 0| 1| 1| |expss |0.8.6 | 2| 1| 0| |fdq |0.2 | 0| 1| 0| |FedData |2.5.2 | 0| 1| 0| |Fgmutils |0.9.4 | 0| 1| 0| |flippant |1.1.0 | 0| 1| 0| |fst |0.8.4 | 1| 0| 1| |genBart |1.0.1 | 1| 0| 0| |GenomicTools |0.2.6 | 1| 0| 0| |GGIR |1.5-18 | 1| 0| 1| |ggswissmaps |0.1.1 | 0| 1| 0| |greport |0.7-1 | 0| 1| 1| |hisse |1.8.5 | 1| 0| 0| |Hmisc |4.1-1 | 0| 1| 1| |hoardeR |0.9.2 | 1| 0| 0| |hypervolume |2.0.8 | 0| 1| 1| |iml |0.3.0 | 2| 0| 0| |installr |0.19.0 | 0| 1| 0| |inTrees |1.2 | 0| 1| 0| |ITGM |0.6 | 0| 1| 0| |itsadug |2.3 | 0| 1| 0| |JWileymisc |0.2.1 | 0| 1| 0| |kangar00 |1.1 | 1| 0| 0| |KRIG |0.1.0 | 1| 0| 0| |LabourMarketAreas |3.2 | 0| 1| 0| |lava |1.6.1 | 1| 0| 1| |lavaSearch2 |1.3.3 | 0| 1| 0| |LSPFP |1.0.0 | 0| 1| 0| |MetamapsDB |0.0.2 | 1| 0| 0| |metaMix |0.2 | 1| 0| 0| |mlr |2.12.1 | 1| 1| 1| |mlrMBO |1.1.1 | 0| 1| 1| |MODIStsp |1.3.3.1 | 1| 0| 0| |MultiABEL |1.1-6 | 0| 1| 1| |NetLogoR |0.3.2 | 0| 1| 1| |networkR |0.1.0-0 | 0| 1| 1| |nlmixr |0.9.0-3 | 0| 1| 1| |NNS |0.3.8.6 | 0| 1| 0| |openEBGM |0.5.0 | 0| 1| 0| |OpenML |1.8 | 0| 1| 0| |openSTARS |1.0.0 | 0| 1| 0| |optiSel |2.0 | 0| 1| 1| |orgR |0.9.0 | 0| 1| 0| |padr |0.4.0 | 1| 0| 1| |partialCI |1.1.1 | 0| 1| 1| |partools |1.1.6 | 0| 1| 0| |pcrsim |1.0.2 | 1| 0| 0| |petro.One |0.1.3 | 1| 0| 0| |PhenotypeSimulator |0.2.0 | 1| 0| 0| |pkggraph |0.2.2 | 0| 1| 0| |plotly |4.7.1 | 0| 1| 0| |popEpi |0.4.4 | 2| 1| 0| |PopGenReport |3.0.0 | 0| 1| 0| |prioritizr |3.0.4 | 0| 1| 2| |ProFound |1.0.1 | 1| 1| 0| |Publish |2018.04.17 | 0| 1| 0| |quickPlot |0.1.3 | 1| 0| 1| |RAM |1.2.1.3 | 0| 1| 2| |rangeMapper |0.3-1 | 0| 1| 0| |RAPIDR |0.1.1 | 1| 0| 0| |Rblpapi |0.3.8 | 1| 0| 0| |readtext |0.50 | 1| 0| 0| |RecordLinkage |0.4-10 | 0| 1| 2| |REIDS |0.0.2 | 1| 0| 0| |reinsureR |0.1.0 | 0| 1| 0| |rGoodData |0.1.1 | 0| 1| 0| |riskRegression |2018.04.21 | 0| 1| 1| |RndTexExams |1.4.1 | 0| 1| 0| |Rnets |0.9.8 | 0| 1| 0| |RNewsflow |1.0.1 | 0| 1| 0| |robCompositions |2.0.7 | 0| 1| 1| |RSauceLabs |0.1.6 | 0| 1| 0| |rstanarm |2.17.4 | 2| 1| 3| |rtop |0.5-14 | 0| 1| 1| |SafeQuant |2.3.1 | 1| 0| 0| |sdcMicro |5.2.0 | 2| 0| 1| |sdcTable |0.22.8 | 1| 0| 0| |SDLfilter |1.0.0 | 0| 1| 0| |seleniumPipes |0.3.7 | 0| 1| 0| |sentometrics |0.3.5 | 0| 1| 1| |simPop |1.1.0 | 0| 1| 1| |SIRItoGTFS |0.2.2 | 1| 0| 0| |sitree |0.1-2 | 0| 1| 0| |socialmixr |0.1.2 | 1| 0| 0| |sommer |3.4 | 0| 1| 0| |SpaDES.core |0.1.1 | 1| 2| 0| |sparseFLMM |0.2.2 | 0| 1| 0| |speaq |2.3.1 | 1| 0| 0| |splitstackshape |1.4.4 | 1| 0| 0| |strataG |2.0.2 | 0| 1| 1| |strvalidator |2.0.0 | 1| 0| 0| |surveyplanning |2.9 | 0| 1| 0| |SWMPr |2.3.0 | 0| 1| 0| |synthACS |1.0.1 | 0| 1| 1| |tcR |2.2.1.11 | 0| 1| 4| |teachingApps |1.0.2 | 0| 1| 1| |textTinyR |1.1.0 | 0| 1| 1| |trelliscope |0.9.8 | 0| 1| 0| |tRophicPosition |0.7.5 | 1| 1| 0| |tstools |0.3.6 | 1| 1| 0| |validateRS |1.0.0 | 0| 1| 0| |vardpoor |0.9.11 | 0| 1| 0| |VIM |4.7.0 | 0| 1| 1| |wally |1.0.9 | 0| 1| 0| |winRatioAnalysis |0.1.0 | 0| 1| 0| |word.alignment |1.0.9 | 0| 1| 0| |xgboost |0.6.4.1 | 0| 1| 1| |yCrypticRNAs |0.99.2 | 1| 0| 0|Log
## BatchExperiments (1.4.2) Maintainer: Michel Lang