dpc10ster / RJafroc

Artificial Intelligence: Evaluating AI, optimizing AI
19 stars 8 forks source link

test-StSignificanceTestingCad.R has thousands of warnings #8

Closed pwep closed 5 years ago

pwep commented 5 years ago

After applying #7 (the fix for issue #6), tests can progress. I have not got all the way to the end of the test files yet, but test-StSignificanceTestingCad.R gnerates thousands of warnings.

With the RJafroc package loaded I run the test file using:

testthat::test_file("tests/testthat/test-StSignificanceTestingCad.R")

The first few test pass with warnings (coming from expect_known_output), but then these lines generate around 2000 warnings.

https://github.com/dpc10ster/RJafroc/blob/3358e97bf5cfeb3e3e643e2ca41346496036d3f7/tests/testthat/test-StSignificanceTestingCad.R#L36-L39

These warnings are all the same:

Warning in regularize.values(x, y, ties, missing(ties)) :
  collapsing to unique 'x' values

Are these warnings OK to expect for this dataset? If so, the test could be done in two steps:

  1. by adding an expect_warning() test for StSignificanceTestingCadVsRadiologists and saving the result to a variable
  2. Using the variable in expect_known_output()

So something like this:

expect_warning(pcl_singular <- StSignificanceTestingCadVsRadiologists (datasetCadLroc, FOM = "PCL", method = "singleModality"),
   "collapsing to unique 'x' values")
expect_known_output(pcl_singular,tmp, print = TRUE, update = TRUE)

All n warnings from regularize.value, occurring within StSignificanceTestingCadVsRadiologists, can be accounted for, and the only warning created is the usual Creating reference output from the expect_known_output() function.

dpc10ster commented 5 years ago

I am going to look into this tomorrow; I don't expect the warnings

dpc10ster commented 5 years ago

The problem was traced to NOT specifying how to handle ties when interpolating between two operating points. For our case the ties = min option is appropriate.

PCL <- (approx(lroc$FPF, lroc$PCL, xout = FPFValue))$y # computes PCL @ FPFValue

StSignificanceTestingCadVsRadiologists (datasetCadLroc, FOM = "PCL", method = "singleModality")

Browse[2]> PCL
[1] 0.5908333
Warning message:
In regularize.values(x, y, ties, missing(ties)) :
  collapsing to unique 'x' values
Browse[2]> ALroc
[1] 0.1003894

PCL <- (approx(lroc$FPF, lroc$PCL, xout = FPFValue, ties = min))$y # computes PCL @ FPFValue StSignificanceTestingCadVsRadiologists (datasetCadLroc, FOM = "PCL", method = "singleModality")

Browse[2]> PCL
[1] 0.5908333
Browse[2]> ALroc
[1] 0.1003894

Inserted 'ties = min' option in 7 lines. One in 'gpfMyFOM.R' and 6 in 'PlotEmpiricalOperatingCharacteristics.R'

dpc10ster commented 5 years ago

I think it is appropriate to close this issue; my latest upload to the development branch uses the ties = min argument to function approx to fix these warnings.