[REVIEW]: A Python module to combine p values arising from discrete tests

editorialbot commented 9 months ago

Submitting author: !--author-handle-->@Wrzlprmft@vissarion<!--end-editor-- Reviewers: @steppi, @mdhaber Archive: 10.5281/zenodo.8338798

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f"><img src="https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f/status.svg)](https://joss.theoj.org/papers/ec8a185cc7c9c7880ee77bba4078e01f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@steppi & @mdhaber, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @vissarion know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @mdhaber

📝 Checklist for @steppi

editorialbot commented 9 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 9 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1080/01621459.1962.10482147 is OK
- 10.1093/biomet/asx076 is OK
- 10.18637/jss.v101.i01 is OK
- 10.2466/pr0.95.2.449-458 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 9 months ago

Software report:

github.com/AlDanial/cloc v 1.88  T=0.21 s (119.7 files/s, 13481.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          14            398            393           1375
reStructuredText                 2             83             15            164
make                             1             28              6            143
YAML                             4             10              0             80
TeX                              1              5              0             44
Markdown                         1             13              0             43
JSON                             1              0              0             11
INI                              1              0              0              4
-------------------------------------------------------------------------------
SUM:                            25            537            414           1864
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

editorialbot commented 9 months ago

Wordcount for paper.md is 638

editorialbot commented 9 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

mdhaber commented 9 months ago

Review checklist for @mdhaber

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/BPSB/combine-p-values-discrete?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license? Yes, here.
[x] Contribution and authorship: Has the submitting author (@Wrzlprmft) made major contributions to the software? Does the full list of paper authors seem appropriate and complete? Yes. @Wrzlprmft appears to be the sole author.
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines It's challenging to assess the contribution according to whether it constitutes "three months of work for an indivual". There was a two-month period of work in the beginning, but I can't tell whether it was a full time effort. There also appears to be a recent burst of activity in September, presumably to prepare the package for review.
- The commit history is nearly two years old, but most activity happened in the first two months or very recently.
- There are 184 commits. Over the course of three months, that's an average of three commits per day, which may or may not be full-time equivalent.
- There is one author.
- There are approximately 3000 lines total, the result of 5000 additions and 2000 deletions. 800+ lines are implementation and 900+ lines of tests, and these lines are reasonably compact (not padded with tons of whitespace).
- It doesn't look like the work has already been cited.
- I think it is likely to be used in scientific work. Whether it is cited depends on the disposition of the users to cite open source software!
- The package is "feature-complete" and useable by others, not "half-baked" or "thin" Based on discussions with the author, I'm not sure whether this meets JOSS's "three months of work for an individual" for this individual to the letter, but I don't think it would make sense to reject the work on this basis. Had the same work - or work of lower quality - been completed by someone with less expertise over a greater period of time, the letter of this criterion would have been satisfied, but I don't think the work would be any more worthy of publication.
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item. I do not see any claims about original data.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item. I do not see any claims about original results.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item. I do not see any claims about real original human or animal research. The "extended example" does not appear to contain real animal data; it uses fictitious data to illustrate the analysis procedure.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation? Yes. I confirmed that pip install combine_pvalues_discrete works on Colab.
[x] Functionality: Have the functional claims of the software been confirmed? I've begun to confirm some of the claims indirectly. For instance, when the null hypothesis of each individual test is true, the distribution of combined p-values produced by the software appears uniform. Also, when combining p-values from continuous tests, the results produced using the software appear to match those produced by scipy.stats.combine_pvalues. Followup: I've verified some of the claims more directly. Using scipy.stats.monte_carlo_test, I wrote my own version of Fisher's approach to combining p-values. This is similar to what the author shows in Checking the result. So I haven't done a line-by-line review of all functionality, but the basic functionality is definitely there.
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)
- [x] "The presented module combine_pvalues_discrete implements this approach in a:
- [x] fast, yes, it is much faster to use the population of p-values rather than simulate them from random data
- [x] thorough,
- [x] and tested manner
- [x] taking care of pitfalls such as correctly handling
- [x] complements,
- [x] sidedness, and
- [x] empirical 𝑝 values
- [x] as well as handling tedious and error-prone tasks..." I have confirmed that the software correctly counts p-values considering numerical tolerances (atol, rtol are important and often neglected) and computes the p-value from counts (see "Permutation P-values should never be zero").
- [x] "This approach is considerably faster than a permutation test starting at the level of individual datasets."
- [x] "Thanks to modern computing and NumPy, it is easy to make the number of samples very high and the result very accurate." (in the documentation)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is? Yes, here.
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution. Yes. The requirements are included in a requirements.txt file.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems). Yes. The "extensive example" seems very practical, even if it is not truly real-world data.
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)? Yes. I was able to explore features not covered in the tutorials using the rest of the documentation.
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified? Yes, automated tests are written using the Pytest framework. I have not verified that the tests are sufficient, but it looks like reasonable care has been taken.
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support ~~I see guidelines for reporting issues, but I don't see guidelines for contributing or seeking support.~~ All guidelines added here.

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided? Yes, I think the description of the purpose is very clear for anyone familiar with null hypothesis significance testing, which is a rather non-specialist audience.
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work? Yes. The paper is clear that the problem is that there are not other options in the Python ecosystem for combining p-values that arise from discrete tests (i.e. tests with discrete null distributions).
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages? Somewhat. The most obvious choice for combining p-values in Python, scipy.stats.combine_pvalues, is only valid for p-values from continuous tests, and the paper cites this. It also cites the R poolr package. However, to more fully satisfy this criterion, the paper could mention other meta-analysis packages in Python (e.g. PythonMeta, PyMARE, statsmodels.stats.meta_analysis) and/or other languages (e.g. R's metap). Yes, it does so more completely now.
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)? Yes. I have a few specific questions, but I don't have any substantial suggestions for most of it.
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax? References seem appropriate, but I need to go over them in more detail (e.g. syntax).

Wrzlprmft commented 9 months ago

@editorialbot generate pdf

editorialbot commented 9 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Wrzlprmft commented 9 months ago

@editorialbot generate pdf

editorialbot commented 9 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

steppi commented 8 months ago

Review checklist for @steppi

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/BPSB/combine-p-values-discrete?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@Wrzlprmft) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[X] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines I saw your response here, https://github.com/BPSB/combine-p-values-discrete/issues/1. I agree that the level of effort seems appropriate, even if the time spent was not quite 3 months. I don't think it would be unreasonable for this to have taken 3 months.
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed? I've created a simulation which shows the software works beautifully exactly as advertised, here https://gist.github.com/steppi/77357e4456d9f4b92af2c614ec665c99
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)? I think the package is well documented. I left issue https://github.com/BPSB/combine-p-values-discrete/issues/3 suggesting to add examples to the docstrings as suggested in the JOSS review guidelines
```
Good: All functions/methods are documented including example inputs and outputs
OK: Core API functionality is documented
Bad (not acceptable): API is undocumented
```
because I agree this is good practice. I also asked if lines could be wrapped in the docstrings to make them easier to read in a terminal. Right now there lines up to over 400 characters wide.
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified? There is a thorough test suite and automated CI. I think it would be nice to have documentation for how to run the test suite and have submitted a relevant issue here https://github.com/BPSB/combine-p-values-discrete/issues/2.
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

vissarion commented 7 months ago

Hi, @Wrzlprmft could you please update us on your status in addressing issues raised by @mdhaber?

Hi, @steppi could you please update us on the progress of your review? I see there are some checkboxies unchecked. Is it because there are still in progress or there is some issue with them?

Wrzlprmft commented 7 months ago

Hi, @Wrzlprmft could you please update us on your status in addressing issues raised by @mdhaber?

In my understanding, there is nothing for me to address at the moment. I addressed @mdhaber’s last comments here (and the commits referenced therein). Since my last reply was mostly asserting his understandings, I didn’t expect a response and assumed no further clarification or changes were needed.

mdhaber commented 7 months ago

Yes, my comments were addressed, and I only planned to do a more detailed check of the reference section.

steppi commented 7 months ago

Hi, @steppi could you please update us on the progress of your review? I see there are some checkboxies unchecked. Is it because there are still in progress or there is some issue with them?

Sorry @vissarion, since agreeing to review this I ended up being busier than expected but things are settling down. I will be able to complete my review within two weeks. The unchecked boxes are for things that are still in progress.

mdhaber commented 7 months ago

I completed my checklist. References look good except for "metap" probably isn't intended to be capitalized in the rendered document.

Wrzlprmft commented 7 months ago

References look good except for "metap" probably isn't intended to be capitalized in the rendered document.

Fixed. While I disagree with lowercasing proper names, this is reproducing the lack of capitalisation of the cited source.

vissarion commented 7 months ago

@mdhaber do you recommend to accept this paper?

vissarion commented 7 months ago

Hi @steppi, do you have any news from this review?

steppi commented 7 months ago

Sorry for the delay. I’ll get it done this week.

mdhaber commented 7 months ago

@mdhaber do you recommend to accept this paper?

Yes

steppi commented 6 months ago

I've completed my review and recommend this paper for publication. I would like to see the updates I suggested made to the documentation but don't think this is a blocker.

vissarion commented 6 months ago

@Wrzlprmft it seems that both reviewers recommend acceptance! Could you please address the issue of the documentation raised by @steppi ?

When a submission is ready to be accepted, we ask that the authors issue a new tagged release of the software (if changed), and archive it (see this guide). Please do this and post the version number and archive DOI here.

steppi commented 6 months ago

@vissarion, @Wrzlprmft has addressed the issues, or in one case provided his rationale for deviating from standard practice.

Wrzlprmft commented 6 months ago

@Wrzlprmft it seems that both reviewers recommend acceptance!

So it does.

Thanks to the reviewers (@mdhaber and @steppi) for their diligent criticism and constructive comments.

When a submission is ready to be accepted, we ask that the authors issue a new tagged release of the software (if changed), and archive it (see this guide). Please do this and post the version number and archive DOI here.

Version: 1.2.2
DOI: 10.5281/zenodo.10684537

editorialbot commented 6 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Wrzlprmft commented 6 months ago

@Wrzlprmft please edit the zenodo record so that the title, author names and affiliation are the same as in paper.

Hopefully done. I added the paper title as an alternative title. Please tell me if it should be the main title.

vissarion commented 6 months ago

@Wrzlprmft please change the main title of zenodo to "A Python module to combine 𝑝 values arising from1 discrete tests." i.e. as in JOSS paper so that the DOI looks correct and has the same title as in your paper.

https://zenodo.org/records/10684537

Wrzlprmft commented 6 months ago

@Wrzlprmft please change the main title of zenodo to "A Python module to combine 𝑝 values arising from1 discrete tests." i.e. as in JOSS paper so that the DOI looks correct and has the same title as in your paper.

https://zenodo.org/records/10684537

Done (except for the stray “1” which I presume to be an error).

vissarion commented 6 months ago

@editorialbot set 10.5281/zenodo.8338798 as archive

editorialbot commented 6 months ago

Done! archive is now 10.5281/zenodo.8338798

vissarion commented 6 months ago

@editorialbot set 1.2.2 as version

editorialbot commented 6 months ago

Done! version is now 1.2.2

vissarion commented 6 months ago

@editorialbot check references

editorialbot commented 6 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1080/01621459.1962.10482147 is OK
- 10.1093/biomet/asx076 is OK
- 10.18637/jss.v101.i01 is OK
- 10.2466/pr0.95.2.449-458 is OK
- 10.2202/1544-6115.1585 is OK
- 10.5281/zenodo.4266738 is OK

MISSING DOIs

- None

INVALID DOIs

- None

vissarion commented 6 months ago

@editorialbot generate pdf

editorialbot commented 6 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

vissarion commented 6 months ago

@editorialbot recommend-accept

editorialbot commented 6 months ago

Attempting dry run of processing paper acceptance...

editorialbot commented 6 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1080/01621459.1962.10482147 is OK
- 10.1093/biomet/asx076 is OK
- 10.18637/jss.v101.i01 is OK
- 10.2466/pr0.95.2.449-458 is OK
- 10.2202/1544-6115.1585 is OK
- 10.5281/zenodo.4266738 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 6 months ago

:wave: @openjournals/dsais-eics, this paper is ready to be accepted and published.

Check final proof :point_right::page_facing_up: Download article

If the paper PDF and the deposit XML files look good in https://github.com/openjournals/joss-papers/pull/5038, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

arfon commented 6 months ago

@editorialbot accept

editorialbot commented 6 months ago

Doing it live! Attempting automated processing of paper acceptance...

editorialbot commented 6 months ago

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

``` cff-version: "1.2.0" authors: - family-names: Ansmann given-names: Gerrit orcid: "https://orcid.org/0000-0002-5472-7067" doi: 10.5281/zenodo.8338798 message: If you use this software, please cite our article in the Journal of Open Source Software. preferred-citation: authors: - family-names: Ansmann given-names: Gerrit orcid: "https://orcid.org/0000-0002-5472-7067" date-published: 2024-02-29 doi: 10.21105/joss.06096 issn: 2475-9066 issue: 94 journal: Journal of Open Source Software publisher: name: Open Journals start: 6096 title: A Python module to combine p values arising from discrete tests. type: article url: "https://joss.theoj.org/papers/10.21105/joss.06096" volume: 9 title: A Python module to combine $p$ values arising from discrete tests. ```

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

editorialbot commented 6 months ago

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

editorialbot commented 6 months ago

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/5071
Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.06096
If everything looks good, then close this review issue.
Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

Wrzlprmft commented 6 months ago

🎉 Everything looks fine to me. Thanks to everybody involved.

However, I cannot close this issue (or find the editorial technical team to contact about this) as instructed. It seems like @vissarion or @arfon has to do this.

arfon commented 6 months ago

@steppi, @mdhaber – many thanks for your reviews here and to @vissarion for editing this submission! JOSS relies upon the volunteer effort of people like you and we simply wouldn't be able to do this without you ✨

@Wrzlprmft – your paper is now accepted and published in JOSS :zap::rocket::boom:

editorialbot commented 6 months ago

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.06096/status.svg)](https://doi.org/10.21105/joss.06096)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.06096">
  <img src="https://joss.theoj.org/papers/10.21105/joss.06096/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.06096/status.svg
   :target: https://doi.org/10.21105/joss.06096

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

openjournals / joss-reviews