[REVIEW]: statConfR: An R Package for Static Models of Decision Confidence and Metacognition

editorialbot commented 2 months ago

Submitting author: !--author-handle-->@ManuelRausch@samhforbes<!--end-editor-- Reviewers: @haoxue-fan, @christinamaher Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/dbac9fa6ff476bf8eda77f12aa077192"><img src="https://joss.theoj.org/papers/dbac9fa6ff476bf8eda77f12aa077192/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/dbac9fa6ff476bf8eda77f12aa077192/status.svg)](https://joss.theoj.org/papers/dbac9fa6ff476bf8eda77f12aa077192)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@haoxue-fan & @christinamaher, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @samhforbes know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @haoxue-fan

📝 Checklist for @christinamaher

editorialbot commented 2 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 2 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.02 s (1776.2 files/s, 241132.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               26            410            710           2513
TeX                              3             54             10            552
Markdown                         2             44              0            156
YAML                             1              1              4             18
Rmd                              1              2              6              0
-------------------------------------------------------------------------------
SUM:                            33            511            730           3239
-------------------------------------------------------------------------------

Commit count by author:

    67  Manuel Rausch
     8  Hellmann

editorialbot commented 2 months ago

Paper file info:

📄 Wordcount for paper.md is 794

✅ The paper includes a Statement of need section

editorialbot commented 2 months ago

License info:

🟡 License found: GNU General Public License v3.0 (Check here for OSI approval)

editorialbot commented 2 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.concog.2017.02.007 is OK
- 10.1038/s41467-021-23540-y is OK
- 10.3758/s13414-017-1431-5 is OK
- 10.1038/s41562-019-0813-1 is OK
- 10.1016/j.concog.2011.09.021 is OK
- 10.3758/s13414-021-02284-3 is OK
- 10.1037/a0019737 is OK
- 10.1093/nc/nix007 is OK
- 10.1037/rev0000249 is OK
- 10.7554/eLife.75420 is OK
- 10.1007/978-3-642-45190-4_3 is OK
- 10.1093/nc/niw002 is OK
- 10.1038/s41467-022-31727-0 is OK
- 10.1016/j.cognition.2020.104522 is OK
- 10.1016/j.neuroimage.2020.116963 is OK
- 10.1038/s41562-022-01464-x is OK
- 10.1037/rev0000411 is OK
- 10.1177/17456916221075615 is OK
- 10.1037/xge0001524 is OK
- 10.1121/1.1907783 is OK
- 10.1037/met0000634 is OK
- 10.31234/osf.io/5ze8t is OK

MISSING DOIs

- No DOI given, and none found for title: Signal detection theory and psychophysics
- No DOI given, and none found for title: Detection theory: A user’s guide

INVALID DOIs

- None

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

samhforbes commented 2 months ago

Hi @haoxue-fan, @christinamaher this is our review thread. Feel free to raise any issues that need addressing according to the reviewer checklist in individual issues in the software repos and link back here. If there's any larger comments you need to make or anything you want me to look at, you can of course post here, or ping me for questions. Thanks again for agreeing to review.

samhforbes commented 1 month ago

Hi @haoxue-fan, @christinamaher I just thought I'd check in and see how things were coming along. Please ping me if there's anything you need.

christinamaher commented 1 month ago

Hi @samhforbes ! Thanks for checking in, apologies for the delay. I am returning today from conference travel. I aim to complete this by the end of the week.

haoxue-fan commented 1 month ago

Review checklist for @haoxue-fan

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/ManuelRausch/StatConfR?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@ManuelRausch) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[ ] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

christinamaher commented 1 month ago

Review checklist for @christinamaher

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/ManuelRausch/StatConfR?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@ManuelRausch) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[ ] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

christinamaher commented 1 month ago

Hi @samhforbes ! I believe I've gone through the relevant points in the checklist above. The issues (related to points I left unchecked above) are as follows - General Checks - Human Research Approvals: The dataset provided is from a human experiment. Although it appears to have been previously published, the documentation does not provide details regarding informed consent for data sharing or ethical approvals.

Documentation - Functionality Documentation: The code would benefit significantly from improved documentation. Function Documentation: Each function is lacking explanation of its purpose, parameters, and outputs. README File: The README file should provide a clearer guide to the package's components, including how to use the functions effectively.

Software Paper - Summary: The paper lacks a high-level summary of the package’s primary use-case. Including this summary would help readers understand the main objectives and applications of the package. Model Descriptions: Authors should consider providing a detailed description of the models included in the package and an explain for the rationale behind the choice of models and the fitting procedures used. Visual Aids: Incorporate figures or schematics that visualize the models and the results would greatly enhance the paper's readability and package's usability. Plotting Utilities: Consider adding plotting utilities to the package to facilitate visualization of results.

samhforbes commented 1 month ago

Great, thanks @christinamaher @ManuelRausch while we wait for feedback from @haoxue-fan there's some really useful feedback to work on.

haoxue-fan commented 1 month ago

Thanks for the opportunity to review the package and sorry for the delayed response! I first want to applaud for the authors @ManuelRausch for their effort putting together this package - it is super useful for researchers in relevant fields both in terms of encouraging them to try out different models as well as lower the coding barrier. I was able to load the R package and run the code as stated in README without difficulty. However, I have a couple of comments listed below most related to the writing and the documentation aspects of the package that I think worth improving:

Documentation: I share the same feeling as @christinamaher that the package can benefit greatly from more documentation. The way that the paper is currently written and the README file makes me feel that the author assumes a relatively high degree of expertise in the related research domain in the audience, which may or may not be true. Therefore, I think it would be great to provide a step-by-step walkthrough example in the paper. This can be the current example use described in README, but probably with more text descriptions e.g., what the output means; what are some functions that can be used; what are the orders that the functions should be used.
Relatedly, I notice that on the function help page there are descriptions related to the input data format as well as detailed description of each model. It would be great to make it explicit (e.g., mention that the users can refer to XXX page, or paraphrase them concisely in the paper/README).
Is it possible for the audience to implement their own models? Is there any guideline on how to do that, and is there any platform for people to share the model with each other? I understand that this maybe too much to ask for, but either way (allowing community contribution or not), it may worth spoiling some ink describing whether the users can implement their customized models, whether this is recommended, and what are some tips that can be shared.
When I was running the example code, the function fitConfModels took a while to run. I am not sure whether this is normal/my computer is too old. Either way, it may worth adding some description on README for the running speed so that the users can calibrate their expectation (I was anxiously checking whether my R is still working since there was no output for a while).

ManuelRausch commented 3 weeks ago

Thank you very much @christinamaher and @haoxue-fan for your feedback. I am sorry that I have not been responsive; I was distracted. I will work on a revision as soon as I can.

samhforbes commented 3 weeks ago

Oh how lovely @ManuelRausch. Enjoy this wonderful period!

openjournals / joss-reviews