openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
722 stars 38 forks source link

[REVIEW]: `SuperNOVA`: Semi-Parametric Identification and Estimation of Interaction and Effect Modification in Mixed Exposures using Stochastic Interventions in `R` #5422

Closed editorialbot closed 1 year ago

editorialbot commented 1 year ago

Submitting author: !--author-handle-->@blind-contours<!--end-author-handle-- (David McCoy) Repository: https://github.com/blind-contours/SuperNOVA Branch with paper.md (empty if default branch): main Version: v1.0.0-JOSS Editor: !--editor-->@csoneson<!--end-editor-- Reviewers: @mhu48, @lucasmiranda42 Archive: 10.5281/zenodo.10038794

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/70888201f8453113ab2a8d287b78c0f6"><img src="https://joss.theoj.org/papers/70888201f8453113ab2a8d287b78c0f6/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/70888201f8453113ab2a8d287b78c0f6/status.svg)](https://joss.theoj.org/papers/70888201f8453113ab2a8d287b78c0f6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mhu48 & @lucasmiranda42, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @csoneson know.

Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest

Checklists

📝 Checklist for @mhu48

📝 Checklist for @lucasmiranda42

editorialbot commented 1 year ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 1 year ago
Software report:

github.com/AlDanial/cloc v 1.88  T=0.29 s (450.2 files/s, 57502.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               92           2012           2045           8221
TeX                              3            170              8           1274
Markdown                         4            134              0           1051
Rmd                              4            296            325            363
Bourne Shell                    25             74            468             76
YAML                             2             12              4             71
-------------------------------------------------------------------------------
SUM:                           130           2698           2850          11056
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 1 year ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1093/biostatistics/kxu058 is OK
- 10.1097/01.ee9.0000606120.58494.9d is OK
- 10.1073/pnas.1510489113 is OK
- 10.1515/ijb-2015-0013 is OK
- 10.2202/1557-4679.1217 is OK
- 10.1111/j.1541-0420.2011.01685.x is OK
- 10.1002/sim.5907 is OK
- 10.1007/978-3-319-65304-4_14 is OK
- 10.1007/978-1-4419-9782-1 is OK
- 10.1007/978-3-319-65304-4 is OK
- 10.2202/1557-4679.1356 is OK
- 10.1111/rssb.12362 is OK
- 10.1111/biom.13375 is OK
- 10.21105/joss.02526 is OK
- 10.5281/zenodo.1342293 is OK
- 10.5281/zenodo.3558313 is OK
- 10.5281/zenodo.3698329 is OK

MISSING DOIs

- None

INVALID DOIs

- None
editorialbot commented 1 year ago

Wordcount for paper.md is 1295

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

csoneson commented 1 year ago

👋🏼 @blind-contours, @mhu48, @lucasmiranda42 - this is the review thread for the submission. All of our communications will happen here from now on.

Please check the post at the top of the issue for instructions on how to generate your own review checklist. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues directly in the software repository. If you do so, please mention this thread so that a link is created (and I can keep an eye on what is happening). Please also feel free to comment and ask questions in this thread. It is often easier to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

Please feel free to ping me (@csoneson) if you have any questions or concerns. Thanks!

lucasmiranda42 commented 1 year ago

Review checklist for @lucasmiranda42

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

lucasmiranda42 commented 1 year ago

Hi! First impressions on my side.

1) The paper is well written: it clearly states how the package fits in the field, and the problems it intends to solve. My only concern is regarding the references: in the README file of the main repo there are several other references that are not mentioned in the manuscript. I would suggest either adding them or justifying why this is not the case :)

2) The package installs without problems. I filed an issue with a minor concern regarding how the instructions are documented, but overall everything seems to work properly. The example provided in the README file does however not work. It seems to be due to an undocumented change in the API; I filed an issue accordingly.

3) While automatic tests are provided, the CI pipeline seems to be failing at the moment. I would suggest inspecting why and fixing the failing tests.

4) Lastly, while the provided example seems well documented (I haven't gone fully through it yet due to the issue mentioned above) I couldn't find a link to more extensive documentation of the package capabilities and functions. In case it indeed does not exist, I would suggest expanding here!

I will for now wait until (2) is clarified and/or fixed, and fully test functionality then.

Best, and looking forward! Lucas

mhu48 commented 1 year ago

Review checklist for @mhu48

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

mhu48 commented 1 year ago

Hi,

I have reviewed your paper titled "Package X for Statistical Analysis" submitted to the Journal of Open Statistical Software. Overall, the paper is well written and clearly articulates the purpose and scope of the proposed package. However, I do have some concerns and suggestions that I would like to share with you.

Firstly, I recommend that you either include some more references in the manuscript to support this research project, or justify why the current two listed are sufficient for the paper.

Secondly, I successfully installed the package without any issues. However, I encountered similar problems with the example provided in the README file and the CI pipeline. It would be a good idea if you could investigate and address the issues causing the failures to ensure that the package is reliable and robust. I look forward to reviewing the full functionality and performance of the package once this is resolved. I also suggest adding one or two more alternative examples so the new user could more easily understand the different ways to apply this package.

Lastly, I would appreciate more details discussing the potential scope and usage of the package in the second paragraph under the secion, "SuperNOVA ’s Scope".

Overall, I believe that the package has the potential to be a valuable contribution to the field.

Best regards,

Ming

csoneson commented 1 year ago

👋🏻 Just wanted to check in here. @blind-contours - please have a look at the comments from the reviewers above, and let us know if you have any questions or when you are ready for the reviewers to take another look. Thanks!

csoneson commented 1 year ago

👋🏻 @blind-contours - could you give us an update on how things are going here? Don't hesitate to let us know if you have any questions. Thanks!

blind-contours commented 1 year ago

@csoneson, thank you for your patience. I have re-written the paper based on the comments from @mhu48. I need to run some additional checks to ensure functionality as I have done some considerable updates and also included data-adaptive mediation analysis to the package. Let me run my checks and get back to you in a couple days. Thank you for your review.

csoneson commented 1 year ago

👋🏻 @blind-contours - could you provide an update on how things are going here? Thanks!

blind-contours commented 1 year ago

Almost done! At JSM now presenting the method. I'll reach out shortly when I'm finished.

blind-contours commented 1 year ago

@csoneson, @mhu48, @lucasmiranda42 -- I've fixed the example in the readme introduction and re-written the readme to be clearer and include mediation. I've also updated the paper. The current CI is still broken because SL3 has an imputation dependency that isn't on CRAN anymore so the install fails at the moment. I'm trying to figure out how to fix this in the YML script. But for now, everything in the package should work. Please let me know what other comments/suggestions you have and I'll update the package.

blind-contours commented 1 year ago

The package passes all green on my end with no notes or anything.

csoneson commented 1 year ago

Thanks @blind-contours - @mhu48, @lucasmiranda42, when you have a chance, could you take a look at the modified submission and let us know what you think. Thanks!

csoneson commented 1 year ago

👋🏻 @mhu48, @lucasmiranda42 - just wanted to check in to see if you have an estimate of when you might be able to take another look at this submission. Thanks a lot in advance!

mhu48 commented 1 year ago

Thanks for the reminder. I will try to find some time over this weekend to review this revision.

On Sat, Sep 16, 2023 at 8:02 AM Charlotte Soneson @.***> wrote:

👋🏻 @mhu48 https://github.com/mhu48, @lucasmiranda42 https://github.com/lucasmiranda42 - just wanted to check in to see if you have an estimate of when you might be able to take another look at this submission. Thanks a lot in advance!

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/5422#issuecomment-1722225856, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKGGVY4IKSUN7QENPKJYQL3X2WPP5ANCNFSM6AAAAAAXQBEYBI . You are receiving this because you were mentioned.Message ID: @.***>

csoneson commented 1 year ago

Thanks @mhu48 - @lucasmiranda42, do you have an estimate of when you think you will be able to take another look? Thanks.

mhu48 commented 1 year ago

Hi Charlotte,

Thanks for the reminder, I forgot to click send on the email last night.

I reviewed the revised submission this weekend and feel that it is ready for publication.

Thank you for the opportunity to contribute.

Best,

Ming

On Mon, Sep 25, 2023 at 11:10 AM Charlotte Soneson @.***> wrote:

Thanks @mhu48 https://github.com/mhu48 - @lucasmiranda42 https://github.com/lucasmiranda42, do you have an estimate of when you think you will be able to take another look? Thanks.

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/5422#issuecomment-1734059852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKGGVY36LPSOFHH5VOQSGBTX4GUJDANCNFSM6AAAAAAXQBEYBI . You are receiving this because you were mentioned.Message ID: @.***>

csoneson commented 1 year ago

Thanks @mhu48 - could you please also tick the remaining boxes in your review checklist above?

mhu48 commented 1 year ago

Review checklist for @mhu48

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

csoneson commented 1 year ago

👋🏻 @lucasmiranda42 - could you let us know when you think you would be able to take a look at the revised version of this submission? Thanks in advance!

lucasmiranda42 commented 1 year ago

Dear all, the package now runs smoothly on my end and fulfils all functionality claims. My only concern is regarding documentation, which I couldn't find (other than the README). @blind-contours could you elaborate on whether this is intended and why API docs may not be needed? With this, the paper would have green light on my side as well. Best! Lucas

blind-contours commented 1 year ago

@lucasmiranda42 there should be a detailed vignette that accompanies the package. Let me ensure that it is compiling properly. Was it not showing up on your version when you did testing?

blind-contours commented 1 year ago

I've added a new directory /doc that should have an html file that brings up the vignette. I'm going to make some edits to it now but it should be available for users.

csoneson commented 1 year ago

@lucasmiranda42 - just wanted to check whether the additional information provided by @blind-contours allowed you to access the vignette?

lucasmiranda42 commented 1 year ago

Perfect! All set on my side, then :)

csoneson commented 1 year ago

Thank you @lucasmiranda42! @blind-contours - I will also take a quick look through the submission, and will get back to you shortly with the next steps.

csoneson commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

csoneson commented 1 year ago

@editorialbot check references

editorialbot commented 1 year ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1515/ijb-2015-0013 is OK
- 10.2202/1557-4679.1217 is OK
- 10.1111/j.1541-0420.2011.01685.x is OK
- 10.1002/sim.5907 is OK
- 10.1007/978-3-319-65304-4_14 is OK
- 10.1007/978-1-4419-9782-1 is OK
- 10.1007/978-3-319-65304-4 is OK
- 10.2202/1557-4679.1356 is OK
- 10.1111/rssb.12362 is OK
- 10.1111/biom.13375 is OK
- 10.21105/joss.02526 is OK
- 10.5281/zenodo.1342293 is OK
- 10.5281/zenodo.3558313 is OK

MISSING DOIs

- None

INVALID DOIs

- None
csoneson commented 1 year ago

@blind-contours I have gone through the submission and overall it looks good. Just a couple of points, mostly related to the bibliography - could you please:

In addition, the sentence starting on line 51 and ending on line 57 is very long, and I would suggest splitting it up to increase readability.

You can generate a new proof with @editorialbot generate pdf

Once this is done, please have a look at the next steps in the post below.

csoneson commented 1 year ago

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

Editor Tasks Prior to Acceptance

blind-contours commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:warning: An error happened when generating the pdf.

blind-contours commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

blind-contours commented 1 year ago

Zenodo DOI: 10.5281/zenodo.10038794

blind-contours commented 1 year ago

Link here: https://zenodo.org/records/10038794

blind-contours commented 1 year ago

I think I've made the suggested changes and also created the release version and upload to Zenodo correctly - if there are issues do please let me know.

csoneson commented 1 year ago

@editorialbot check references

editorialbot commented 1 year ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1111/rssb.12362 is OK
- 10.5281/zenodo.1342293 is OK
- 10.1515/ijb-2015-0013 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.48550/arXiv.2305.01849 is INVALID because of 'https://doi.org/' prefix
- 10.1111/j.1541-0420 is INVALID
csoneson commented 1 year ago

Thanks @blind-contours - a couple of things remain:

Thanks!

blind-contours commented 1 year ago

Sorry about that. Everything should be fixed now.

blind-contours commented 1 year ago

@editorialbot generate pdf

blind-contours commented 1 year ago

@editorialbot check references