When using recalibration, there is an error in showing external validation results (ROC plot) in the shiny app.

[su-boussard-lab]

Describe the bug Running external validation with recalibration in PLP version 6.0.4 adds additional evaluation results to calibration_summary and other tables per recalibration method. However, in the validation section of the shiny app, the calibration plot always displays non-recalibrated results, and the ROC plot displays error. A dropdown list for selecting the recalibration method can be added, and results can be displayed based on it.

Thanks, Behzad

[jreps]

Yeah, I see the issue now. Thanks for letting me know. When extracting the data for the table to select from the SQL only extracts 'Test' or 'Validation' which would ignore the recalibration. I think we could easily modify this to extract anything that is not 'Train' or like you suggest, let the user pick depending on what type of evaluations are available.

This code is actually now in OhdsiShinyModules: https://github.com/OHDSI/OhdsiShinyModules/blob/main/R/prediction-validation.R so I will add an issue in that package with a reference to this.

I've actually been thinking, in general the shiny viewer is not great at displaying validation results (currently you need the internal validation to get to the validation). I'm planning on adding a new button next to the 'View protocol', 'View results', 'view diagnostics' buttons for validation that would let you explore validation even if you don't have the original model. This would let us add more stuff for exploring validation results - so if there is any validation specific stuff missing from the app you (or anyone else reading this) would like, post it here.

[nbehzad]

Thank you for providing this information. I fixed it in my local repository, but there are three points to consider in this issue:

• Using not "Train" will also add the CV. Both "Train" and "CV" may be included.
• When inserting the validation results into the database, recalibrated results are included with the same performance_id as validation, causing the results to be repeated. You used the same performance_id for Train, Test, and CV as well, but this is not a problem because the Shiny app does not show all train, test, and CV results at once. As a result, when validation results are inserted into the database, we must create a new performance_id for each recalibration method. This worked for me when I tested it.
• If you add a new performance_id per recalibrated validation (for each modelDesign and validation database), a column called 'evaluation' should be added to the result table of the shiny app as well as plot legend to separate validation and recalibrated validation results.

The last point, AUPRC, can also be added to the result table in the Shiny app.

Thanks, Behzad