Presubmission inquiry - predictNMB: Evaluate Clinical Prediction Models by Net Monetary Benefit

[RWParsons]

Submitting Author Name: Rex Parsons Submitting Author Github Handle: !--author1-->@RWParsons<!--end-author1-- Other Package Authors Github handles: @robinblythe, @agbarnett Repository: https://github.com/RWParsons/predictNMB Submission type: Pre-submission Language: en

---

• Paste the full DESCRIPTION file inside a code block below:

Package: predictNMB
Type: Package
Title: Evaluate Clinical Prediction Models by Net Monetary Benefit
Version: 0.0.0.9000
Authors@R: c(
    person("Rex", "Parsons", role = c("aut", "cre"),
           email = "rex.parsons94@gmail.com",
           comment = c(ORCID = "0000-0002-6053-8174")),
    person("Robin", "Blythe", role = "aut",
           comment = c(ORCID = "0000-0002-3643-4332")),
    person("Adrian", "Barnett", role = "aut",
           comment = c(ORCID = "0000-0001-6339-0374")),
    person("Susanna", "Cramb", role = "ctb",
           comment = c(ORCID = "0000-0001-9041-9531")),
    person("Steve", "McPhail", role = "ctb",
           comment = c(ORCID = "0000-0002-1463-662X"))
  )
Description: Estimates when and where a model-guided treatment strategy may outperform 
    a treat-all or treat-none approach by Monte Carlo simulation and evaluation of the Net Monetary Benefit (NMB).
License: GPL (>= 3)
Encoding: UTF-8
LazyData: true
RoxygenNote: 7.2.1
Imports: 
    assertthat,
    cutpointr,
    dplyr,
    ggplot2,
    magrittr,
    pmsampsize,
    rlang,
    stats,
    tibble,
    tidyr
Suggests: 
    flextable,
    knitr,
    parallel,
    rmarkdown,
    testthat (>= 3.0.0)
Config/testthat/edition: 3
VignetteBuilder: knitr
URL: https://rwparsons.github.io/predictNMB/
BugReports: https://github.com/rwparsons/predictNMB/issues

Scope

• Please indicate which category or categories from our package fit policies or statistical package categories this package falls under. (Please check an appropriate box below):

  Data Lifecycle Packages

  • [ ] data retrieval

  • [ ] data extraction

  • [ ] data munging

  • [ ] data deposition

  • [ ] data validation and testing

  • [ ] workflow automation

  • [ ] version control

  • [ ] citation management and bibliometrics

  • [ ] scientific software wrappers

  • [ ] field and lab reproducibility tools

  • [ ] database software bindings

  • [ ] geospatial data

  • [ ] text analysis

    Statistical Packages

  • [ ] Bayesian and Monte Carlo Routines

  • [ ] Dimensionality Reduction, Clustering, and Unsupervised Learning

  • [ ] Machine Learning

  • [ ] Regression and Supervised Learning

  • [x] Exploratory Data Analysis (EDA) and Summary Statistics

  • [ ] Spatial Analyses

  • [ ] Time Series Analyses

• Explain how and why the package falls under these categories (briefly, 1-2 sentences). Please note any areas you are unsure of: The package helps the user perform Monte Carlo simulations to evaluate the performance of a (simulated) clinical prediction model to assign (simulated) patients to treatment options with user-defined levels of performance/costs to evaluate the costs/benefits of the use of that model-decision process. I could also consider this a wrapper package as it really just abstracts away the complexity of performing this Monte Carlo simulation study from the user rather than adding anything particularly fancy regarding the Monte Carlo procedure.

• If submitting a statistical package, have you already incorporated documentation of standards into your code via the srr package? Yes.

• Who is the target audience and what are scientific applications of this package?
  People considering the development of a clinical decision support system used to guide patient care between two possible interventions (or non-treatment) and wanting to know how likely it is that their model will be clinically useful compared to a treat-all or treat-none strategy. Performing simulations using hypothetical clinical prediction models of likely estimated performance may show that value is unlikely and deter researchers from model development efforts. These models are rarely used in clinical practice due to them often failing to improve patient outcomes compared to standard care. All evaluation is done in health economic terms, to orient the results towards value-based care.

• Are there other R packages that accomplish the same thing? If so, how does yours differ or meet our criteria for best-in-category? To the best of my knowledge, there are no other packages to simulate prediction models and streamline their evaluation of them in monetary terms.

• (If applicable) Does your package comply with our guidance around Ethics, Data Privacy and Human Subjects Research? Yes.

• Any other questions or issues we should be aware of?: Much of the standards from srr are irrelevant for this package since all data analysed are simulated (I checked the general and bayesian standards). In terms of G1.0 (Statistical Software should list at least one primary reference from published academic literature.), I currently have a journal article under review which uses the simulation process here for another purpose but there are no other existing literature which describe this process alongside this form of evaluation (as far as I know). The literature that we cite in the paper to create our simulated models is cited in the introduction-to-predictNMB vignette.

[annakrystalli]

@ropensci-review-bot check srr

[ropensci-review-bot]

This is not an 'srr' package

[annakrystalli]

Hello @RWParsons 👋

Many thanks for your pre-submission enquiry!

The editorial team has concluded that the package definitely fits in our "stats" scope.

To confirm everything is okay however, before moving to full submission, we do suggest going through the formal process of documenting compliance with the stats standards. Although you've stated that standards have been documented, the srr check indicates that they have not. It would be great if you could complete them to ensure submission would be accepted.

You can call @ropensci-review-bot check srr yourself in this issue at any time to confirm documentation has been completed successfully. You can find more details in our documentation.

Any questions, just let me know!

[RWParsons]

Hi @annakrystalli! Thanks for the quick response and sorry for the rookie mistake. I've updated the documentation for srr but currently only for the general stats requirements. I know you said it fits within the stats scope but I'm just wondering whether that'd be within the "Bayesian and Monte Carlo Routines" subsection or whether it's better suited elsewhere (maybe "EDA" instead just based on the standards included?).

[annakrystalli]

Hello @RWParsons, good question.

This is where the act of documenting against standards can be helpful in narrowing down the category. The stats-devguide states categories are appropriate where at least half of all standards can be applied. Given that and your efforts to add the standard documentation, have they made you lean more towards one or the other categories? Would atleast half the Bayesian & Monte Carlo routines standards apply?

[RWParsons]

Thanks for the quick response and advice @annakrystalli ! I've just tallied up how many would be NA for the Bayesian and the EDA standards and I don't make the cut for the Bayesian standards (only get about 1/3 to be relevant) but I do for the EDA (29/34 standards are relevant).

I'll make the changes to the package and document the EDA standards throughout before calling the srr checker bot.

Thanks!

[RWParsons]

@ropensci-review-bot check srr

[ropensci-review-bot]

I'm sorry @RWParsons, I'm afraid I can't do that. That's something only editors are allowed to do.

[annakrystalli]

Ooops! Sorry about that @RWParsons . I think you should be able to run that command so will double check with our both developers. In any case, I'll run it for you. :)

[annakrystalli]

@ropensci-review-bot check srr

[ropensci-review-bot]

'srr' standards compliance:

• Complied with: 69 / 102 = 67.6% (eda: 30 / 34; general: 39 / 68)
• Not complied with: 33 / 102 = 32.4% (eda: 4 / 34; general: 29 / 68)

:heavy_check_mark: This package complies with > 50% of all standads and may be submitted.

[annakrystalli]

🎉 Nice work @RWParsons ! Your package is effectively ready for submission whenever you are.