search

dpc10ster / RJafroc

Artificial Intelligence: Evaluating AI, optimizing AI
19 stars 8 forks source link

Test errors with R 3.6.1 #22

Closed pwep closed 5 years ago

pwep commented 5 years ago

Four structural errors in test-significance-testing.R for the following combinations:

Dataset FOM Method
dataset02 Wilcoxon DBMH
dataset02 Wilcoxon ORH
dataset05 HrAuc DBMH
dataset05 HrAuc ORH 
✖ |   7 4     | Significance testing excluding CAD [8.7 s]
────────────────────────────────────────────────────────────────────────────────
test-significance-testing.R:41: failure: SignificanceTestingAllCombinations
`ret1` not equal to `ret`.
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtRRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtRRFC": Component "StdErr": Attributes: < target is NULL, current is list >
...
Dataset = dataset02, FOM = Wilcoxon, method = DBMH

test-significance-testing.R:41: failure: SignificanceTestingAllCombinations
`ret1` not equal to `ret`.
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtRRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtRRFC": Component "StdErr": Attributes: < target is NULL, current is list >
...
Dataset = dataset02, FOM = Wilcoxon, method = ORH

test-significance-testing.R:41: failure: SignificanceTestingAllCombinations
`ret1` not equal to `ret`.
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtRRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtRRFC": Component "StdErr": Attributes: < target is NULL, current is list >
...
Dataset = dataset05, FOM = HrAuc, method = DBMH

test-significance-testing.R:41: failure: SignificanceTestingAllCombinations
`ret1` not equal to `ret`.
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtRRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtRRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrtFRRC": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrtFRRC": Component "DF": target is numeric, current is array
Component "ciAvgRdrEachTrtRRFC": Component "StdErr": Attributes: < target is NULL, current is list >
...
Dataset = dataset05, FOM = HrAuc, method = ORH
────────────────────────────────────────────────────────────────────────────────
pwep commented 5 years ago

...and three structural errors for test-StSignificanceTestingCad.R using:

Dataset FOM Method
datasetCadLroc Wilcoxon dualModality
datasetCadLroc PCL dualModality
datasetCadLroc ALROC dualModality 
✖ |  17 3     | Significance testing: StSignificanceTestingCadVsRadiologists [77.8 s]
────────────────────────────────────────────────────────────────────────────────
test-StSignificanceTestingCad.R:36: failure: StSignificanceTestingCadVsRadiologists
StSignificanceTestingCadVsRadiologists(...) not equal to `ret`.
Component "ciAvgRdrEachTrt": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrt": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrt": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrt": Component "DF": target is numeric, current is array
Dataset = datasetCadLroc, FOM = Wilcoxon, method = dualModality

test-StSignificanceTestingCad.R:36: failure: StSignificanceTestingCadVsRadiologists
StSignificanceTestingCadVsRadiologists(...) not equal to `ret`.
Component "ciAvgRdrEachTrt": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrt": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrt": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrt": Component "DF": target is numeric, current is array
Dataset = datasetCadLroc, FOM = PCL, method = dualModality

test-StSignificanceTestingCad.R:36: failure: StSignificanceTestingCadVsRadiologists
StSignificanceTestingCadVsRadiologists(...) not equal to `ret`.
Component "ciAvgRdrEachTrt": Component "StdErr": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrt": Component "StdErr": target is numeric, current is array
Component "ciAvgRdrEachTrt": Component "DF": Attributes: < target is NULL, current is list >
Component "ciAvgRdrEachTrt": Component "DF": target is numeric, current is array
Dataset = datasetCadLroc, FOM = ALROC, method = dualModality
────────────────────────────────────────────────────────────────────────────────
pwep commented 5 years ago

Push request #21 is required to identify the Dataset, FOM, Method which raised the errors.

dpc10ster commented 5 years ago

Please see latest commit; I could find no substantive errors in file StSignficanceTesting.R for the first failure; I cleaned up code some; I regenerated to goodValues folders and Travis passes on all 3 checks (including new version of R); details in commit message

pwep commented 5 years ago

More on the errors. Use str to show the structure of the data frame.

I have rebuild an old dataset02WilcoxonDBMH RDS file from R v3.6.0 and printed the structure using str. The results are shown below, for the ciAvgRdrEachTrtRRRC section, which is a source of one of the errors.

> oldData2WilcoxonDBMH <- readRDS("oldData2WilcoxonDBMH.rds")
> str(oldData2WilcoxonDBMH[9])
List of 1
 $ ciAvgRdrEachTrtRRRC:'data.frame':    2 obs. of  6 variables:
  ..$ Treatment: Factor w/ 2 levels "0","1": 1 2
  ..$ Area     : num [1:2] 0.897 0.941
  ..$ StdErr   : num [1:2(1d)] 0.0332 0.0216
  ..$ DF       : num [1:2(1d)] 12.7 12.7
  ..$ CILower  : num [1:2] 0.825 0.894
  ..$ CIUpper  : num [1:2] 0.969 0.988

Notice how StdErr and DF are listed as num [1:2(1d)]

In the v3.6.1 equivalent, the ciAvgRdrEachTrtRRRC section now reads...

> newData2WilcoxonDBMH <- readRDS("tests/testthat/goodValues/SigTest/dataset02WilcoxonDBMH")
> str(newData2WilcoxonDBMH[9])
List of 1
 $ ciAvgRdrEachTrtRRRC:'data.frame':    2 obs. of  6 variables:
  ..$ Treatment: Factor w/ 2 levels "0","1": 1 2
  ..$ Area     : num [1:2] 0.897 0.941
  ..$ StdErr   : num [1:2] 0.0332 0.0216
  ..$ DF       : num [1:2] 12.7 12.7
  ..$ CILower  : num [1:2] 0.825 0.894
  ..$ CIUpper  : num [1:2] 0.969 0.988

StdErr and DF now have a structure shown as num [1:2]

This change is related to the bug fix in R v3.6.1.

Pull request #23 will make all versions generate the same structure (the v3.6.1 structure), so one set of goodValues RDS files are required.

dpc10ster commented 5 years ago

Hi Peter,

Thanks as always for the careful explanations. I am on the laptop now, which has 3.6.0 installed, so I cannot generate the goodValues files. If you have access to 3.6.1, could you? Otherwise I will need to wait until the 18th, when I return home. [I would like to keep this machine on the old-release, for debugging purposes.]

The relevant code is in a self-explanatory file in testthat (cleanGenerateGoodValues.R), which needs to be uncommented and run once!

careful here

uncomment and run to clean out goodValues and generate new ones

#

fns <- list.dirs("tests/testthat/goodValues")

for (i in 2:length(fns)) cat(paste0(fns[i], "/*"),"\n") # don't want to

delete the directory goodValues

for (i in 2:length(fns)) unlink(paste0(fns[i], "/*"))

devtools::test()

Thanks,

Dev

On Sat, Jul 13, 2019 at 3:45 AM Peter Phillips notifications@github.com wrote:

More on the errors. Use str to show the structure of the data frame.

I have rebuild an old dataset02WilcoxonDBMH RDS file from R v3.6.0 and printed the structure using str. The results are shown below, for the ciAvgRdrEachTrtRRRC section, which is a source of one of the errors.

oldData2WilcoxonDBMH <- readRDS("oldData2WilcoxonDBMH.rds") str(oldData2WilcoxonDBMH[9]) List of 1 $ ciAvgRdrEachTrtRRRC:'data.frame': 2 obs. of 6 variables: ..$ Treatment: Factor w/ 2 levels "0","1": 1 2 ..$ Area : num [1:2] 0.897 0.941 ..$ StdErr : num [1:2(1d)] 0.0332 0.0216 ..$ DF : num [1:2(1d)] 12.7 12.7 ..$ CILower : num [1:2] 0.825 0.894 ..$ CIUpper : num [1:2] 0.969 0.988

Notice how StdErr and DF are listed as num [1:2(1d)]

In the v3.6.1 equivalent, the ciAvgRdrEachTrtRRRC section now reads...

newData2WilcoxonDBMH <- readRDS("tests/testthat/goodValues/SigTest/dataset02WilcoxonDBMH") str(newData2WilcoxonDBMH[9]) List of 1 $ ciAvgRdrEachTrtRRRC:'data.frame': 2 obs. of 6 variables: ..$ Treatment: Factor w/ 2 levels "0","1": 1 2 ..$ Area : num [1:2] 0.897 0.941 ..$ StdErr : num [1:2] 0.0332 0.0216 ..$ DF : num [1:2] 12.7 12.7 ..$ CILower : num [1:2] 0.825 0.894 ..$ CIUpper : num [1:2] 0.969 0.988

StdErr and DF now have a structure shown as num [1:2]

This change is related to the bug fix https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17570 in R v3.6.1.

Pull request #23 https://github.com/dpc10ster/RJafroc/pull/23 will make all versions generate the same structure (the v3.6.1 structure), so one set of goodValues RDS files are required.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dpc10ster/RJafroc/issues/22?email_source=notifications&email_token=AH4NJRCD722L6FYWZXIC76TP7GBZXA5CNFSM4H7RDE7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ3MB3I#issuecomment-511099117, or mute the thread https://github.com/notifications/unsubscribe-auth/AH4NJRDL4B65V4WKXBOJHNLP7GBZXANCNFSM4H7RDE7A .

dpc10ster commented 5 years ago

The R CMD check failures occur on both 3.6.0 and 3.6.1. However, there are no failures on using devtools::check(). In any case the relevant testing code has a skip_on_cran() statement. I am not going to worry about this issue. I think the issue belongs with R CMD check or RStudio.