search

isetbio / UnitTestToolbox

A toolbox for validating the data and execution of scripts for any project.
1 stars 1 forks source link

Fast validations and custom tolerances #14

Closed DavidBrainard closed 7 years ago

DavidBrainard commented 8 years ago

When we use a custom tolerance, the Fast validation hash should respect it, right?

This does not seem to be happening for a validation in our temporalFittingEngine code, see output below. The Full passes (with custom tolerance) but the Fast fails. The validation files appear to have been generated together.

[ 1] 'v_crossValidateFits' Internal validation : PASSED Run-time status : no exception raised Ground truth url : http://52.32.77.154/repository/temporalFittingEngine/validation/fast/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat Ground truth localP : /Users/aguirre/.gradle/caches/modules-2/files-2.1/validation.fast.CrossValidationRoutines/crossValidateFits/run00001/75e5df9a577878df758636c08ed1fec0b4548125/crossValidateFits-run00001-mat.mat Fast validation : FAILED against ground truth data of 12-Nov-2016 12:49:40.

Ground truth info : eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb' Local host info : GKA-MacBook.local / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'aguirre' DataHash-this run : 9f6e4cbd61f2690c18e95222874028fd93508bb41d06c88819e55877b39282e2 DataHash-groundTruth : 2d716f3b535f969099d256975002de9fe73e9d904ed77fcae646dc0fbe00918b

Ground truth url : http://52.32.77.154/repository/temporalFittingEngine/validation/full/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat Ground truth localP : /Users/aguirre/.gradle/caches/modules-2/files-2.1/validation.full.CrossValidationRoutines/crossValidateFits/run00001/28e619906dbd3b5c51f5d43b9e4090f0d36aed91/crossValidateFits-run00001-mat.mat Validating using custom tolerance for field validationData1(1).paramMainMatrix: 0.0005 Validating using custom tolerance for field validationData1(1).testfVals: 1e-05 Validating using custom tolerance for field validationData1(1).trainfVals: 1e-05 Full validation : PASSED using custom tolerance for fields: {'validationData1'} against ground truth data of 12-Nov-2016 12:49:52. Ground truth info : eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb' Local host info : GKA-MacBook.local / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'aguirre' Note (*) : script does not store any extra data. Validation report : v_crossValidateFits

npcottaris commented 8 years ago

Hmmm... The above passes both FAST and FULL on my iMac. I think that this is probably a precision issue with GKA-MacBook. We can try lowering the custom tolerance until it passes validation on the GKA, but this will be a trial an error approach.

Output from my iMac shown below:

------------------------------------------------------------------------------------------------------------
Running in 'FULL' mode with 'catchExceptionAndContinue' runtime hehavior and verbosity level = 'high'.
Will validate 1 scripts.
------------------------------------------------------------------------------------------------------------

[  1] 'v_crossValidateFits'
    Internal validation  : PASSED
    Run-time status      : no exception raised
    Ground truth  url    : http://52.32.77.154/repository/temporalFittingEngine/validation/fast/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat
    Ground truth  localP : /Users/nicolas/.gradle/caches/modules-2/files-2.1/validation.fast.CrossValidationRoutines/crossValidateFits/run00001/75e5df9a577878df758636c08ed1fec0b4548125/crossValidateFits-run00001-mat.mat
    Fast validation      : PASSED against ground truth data of 12-Nov-2016 12:49:40.
     > Ground truth info :       eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb'
     > Local host info   :          manta.PSYCH.UPENN.EDU / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'nicolas'
    Data hash key        : 2d716f3b535f969099d256975002de9fe73e9d904ed77fcae646dc0fbe00918b
    ---------------------------------------------------------------------------------------------------------------------------------
    Ground truth  url    : http://52.32.77.154/repository/temporalFittingEngine/validation/full/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat
    Ground truth  localP : /Users/nicolas/.gradle/caches/modules-2/files-2.1/validation.full.CrossValidationRoutines/crossValidateFits/run00001/28e619906dbd3b5c51f5d43b9e4090f0d36aed91/crossValidateFits-run00001-mat.mat
    Validating using custom tolerance for field validationData1(1).paramMainMatrix: 0.0005
    Validating using custom tolerance for field validationData1(1).testfVals: 1e-05
    Validating using custom tolerance for field validationData1(1).trainfVals: 1e-05
    Full validation      : PASSED using custom tolerance for fields: {'validationData1'}  against ground truth data of 12-Nov-2016 12:49:52.
     > Ground truth info :       eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb'
     > Local host info   :          manta.PSYCH.UPENN.EDU / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'nicolas'
    Note (*)             : script does not store any extra data.
    Validation report    : ***** v_crossValidateFits *****

 SUMMARY REPORT 
[  1] 'v_crossValidateFits': ......................................... Internal validation: PASSED ; Runtime status: NO EXCEPTION RAISED ; Fast validation: PASSED ; Full validation: PASSED ; (using custom tolerances for fields: {'validationData1'});
npcottaris commented 7 years ago

Can you please update your UnitTestToolbox, rerun the validation for 'v_crossValidateFits', and send me the the output in the command window from GKA-MacBook?

I think the reason the FULL validation is passing whereas FAST is failing is because in the FULL validation we compare the abs(diff(groundTruth-validation)) to the set tolerance, whereas in the FAST validation we truncate both groundTruth and validation depending on the setTolerance, and then taking their hash. There can be cases where the diff value < tolerance (therefore, making the FULL validation pass), whereas the truncated (to the same tolerance) values differ from each other, therefore making the FAST validation fail.

npcottaris commented 7 years ago

We have now concluded the a float number can have different representations in different machines due to differences in the the numerical precision of these machines. This can happen even when rounding the number to low precision, such as 6 significant digits, and thus creates a big problem for validating using hashes, because the hash will be different even at low precisions. We will therefore move to a FULL-only scheme where we validate by comparing validation vs. ground truth data within a tolerance window.

DavidBrainard commented 7 years ago

Giving up on FAST. Closing.