[REVIEW]: PySDM v1: particle-based cloud modelling package for warm-rain microphysics and aqueous chemistry

whedon commented 3 years ago

Submitting author: !--author-handle-->@piotrbartman@dhhagan<!--end-editor-- Reviewers: @darothen, @josephhardinee Archive: 10.5281/zenodo.6321270

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/62cad07440b941f73f57d187df1aa6e9"><img src="https://joss.theoj.org/papers/62cad07440b941f73f57d187df1aa6e9/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/62cad07440b941f73f57d187df1aa6e9/status.svg)](https://joss.theoj.org/papers/62cad07440b941f73f57d187df1aa6e9)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@josephhardinee & @darothen, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @dhhagan know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @josephhardinee

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@piotrbartman) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @darothen

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@piotrbartman) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

dhhagan commented 2 years ago

Hey @josephhardinee any update on your review? If you don't have the time, please let me know so I can find an alternate reviewer. Thanks!

josephhardinee commented 2 years ago

I've completed the review (Sorry it took so long, the package was quite comprehensive). Everything looks good with possibly one exception. The API level documentation is lacking detail once you get below the top level with most methods having no documentation, especially on inputs/outputs (See image below for instance). However, the authors have a fairly robust set of example notebooks that I would argue serves as decent documentation(https://github.com/atmos-cloud-sim-uj/PySDM-examples). I do think we should leave a note encouraging authors to expand API level docs. With that note I would be okay approving this if that meets JOSS standards as the authors appear to be very active in development. @dhhagan would this be acceptable from an editorial viewpoint?

slayoo commented 2 years ago

@josephhardinee, thank you! Challenge accepted - please stay tuned for a report of a docstring sprint (November).

dhhagan commented 2 years ago

Hey @josephhardinee - thanks so much for the update and your effort/work! It is much appreciated. While we don't have strict documentation guidelines, I do think that having docstring-based docs is favorable as it's easier to keep up-to-date moving forward. It seems like the authors will be adding those, so it may make sense to wait until those are finished up so that the published release has those included. Is that okay with you, @slayoo?

slayoo commented 2 years ago

Yes! We have already set up a pylint workflow for the project which gives us indications where docstrings (and other things) are missing or need improvement. Will now focus on improving it and report back here within a few weeks. Thanks @dhhagan, @josephhardinee & @darothen!

darothen commented 2 years ago

@dhhagan Any immediate action items here that you need from the reviewers?

slayoo commented 2 years ago

@darothen @dhhagan Let me just confirm, we are actively cleaning up the code towards 100% pylint-clean (incl. docstrings):

Still, we need some days to finish it up.

BTW, we have recently rolled out a proof-of-concept coupler for ClimateMachine.jl and PySDM (with 2D prescribed-flow test case so far featuring PySDM particle-based microphyics and ClimateMachine's fluid flow): https://github.com/CliMA/ClimateMachine.jl/pull/2244

dhhagan commented 2 years ago

Sounds good @slayoo - thanks for the update!

slayoo commented 2 years ago

@whedon check references from branch JOSS

whedon commented 2 years ago

Attempting to check references... from custom branch JOSS

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1002/qj.441 is OK
- 10.1145/2833157.2833162 is OK
- 10.5194/npg-24-535-2017 is OK
- 10.5194/gmd-8-1677-2015 is OK
- 10.5194/acp-18-7313-2018 is OK
- 10.5194/gmd-10-1817-2017 is OK
- 10.21105/joss.00755 is OK
- 10.1029/2018MS001285 is OK
- 10.5194/gmd-13-5119-2020 is OK
- 10.5194/gmd-13-1335-2020 is OK
- 10.1002/2017MS000930 is OK
- 10.5194/gmd-11-3623-2018 is OK
- 10.1002/qj.1913 is OK
- 10.1029/2002JD002673 is OK
- 10.1007/s10546-020-00595-w is OK
- doi:10.1002/fld.1071 is OK
- 10.1029/2019MS001689 is OK
- 10.21105/joss.02807 is OK
- 10.1038/s41467-019-12982-0 is OK

MISSING DOIs

- 10.1007/978-3-030-77964-1_2 may be a valid DOI for title: On the design of Monte-Carlo particle coagulation solver interface: a CPU/GPU Super-Droplet Method case study with PySDM

INVALID DOIs

- 1520-0469(1967)024<0688:CDGBC>2.0.CO;2 is INVALID

slayoo commented 2 years ago

@whedon check references from branch JOSS

whedon commented 2 years ago

Attempting to check references... from custom branch JOSS

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1002/qj.441 is OK
- 10.1145/2833157.2833162 is OK
- 10.5194/npg-24-535-2017 is OK
- 10.5194/gmd-8-1677-2015 is OK
- 10.1175/1520-0469(1967)024<0688:CDGBC>2.0.CO;2 is OK
- 10.5194/acp-18-7313-2018 is OK
- 10.5194/gmd-10-1817-2017 is OK
- 10.21105/joss.00755 is OK
- 10.1029/2018MS001285 is OK
- 10.5194/gmd-13-5119-2020 is OK
- 10.5194/gmd-13-1335-2020 is OK
- 10.1002/2017MS000930 is OK
- 10.5194/gmd-11-3623-2018 is OK
- 10.1007/978-3-030-77964-1_2 is OK
- 10.1002/qj.1913 is OK
- 10.1029/2002JD002673 is OK
- 10.1007/s10546-020-00595-w is OK
- doi:10.1002/fld.1071 is OK
- 10.1029/2019MS001689 is OK
- 10.21105/joss.02807 is OK
- 10.1038/s41467-019-12982-0 is OK

MISSING DOIs

- None

INVALID DOIs

- None

slayoo commented 2 years ago

@whedon generate pdf from branch JOSS

whedon commented 2 years ago

Attempting PDF compilation from custom branch JOSS. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

slayoo commented 2 years ago

We have already set up a pylint workflow for the project which gives us indications where docstrings (and other things) are missing or need improvement. Will now focus on improving it and report back here within a few weeks.

Dear @dhhagan, @josephhardinee & @darothen,

As of PySDM v1.24 (released just now) 100% of PySDM Python modules have docstrings included. These are top-file module docstrings, mostly 1-2 lines, not yet a full class/method coverage, but given that most of PySDM source files are single-class modules, this should give the users/contributors a meaningful overview of the project when browsing the online docs at https://atmos-cloud-sim-uj.github.io/PySDM.

Moreover, the presence of the module-level docstrings is now ensured at CI level for all PRs using pylint, so new code will follow suit. The documentation is generated also during CI builds (and deployed when merging to master), with python -We -m pdoc, causing any code reference mismatches within the docstrings to be detected and flagged by triggering CI failure. Thus, refactors missing docstring updates will not pass unnoticed.

The documentation sprint has been an occasion for significant cleanups and some refactors aimed at simplifying directory structure and API (all reflected in tests, examples and README Python/Julia/Matlab snippets - as all are checked as a part of CI builds). The code snippets in the paper have also been updated (public API changes were minor, mainly requiring updates of import lines).

We are planning further steady buildup of the docstrings coverage as well as rollouts of other user-oriented resources.

We hope that the reached interim milestone of 100% module docstring coverage and the introduced CI documentation-related automation can serve as a solid indication of our commitment to maintain and enhance PySDM documentation. Looking forward to your comments and acceptance of the paper for publication in JOSS.

Thank you for your consideration, Sylwester

josephhardinee commented 2 years ago

I'd like to commend the authors for seeing through their commitment to improve documentation as well as the accompanying code cleanups. This looks good to me and I think this greatly deserves publication in JOSS.

slayoo commented 2 years ago

thank you, @josephhardinee

dhhagan commented 2 years ago

@slayoo Great to see all of the improvements. @darothen how does it look? Are you ready to check off the last few boxes?

slayoo commented 2 years ago

For the record, from now on (https://github.com/atmos-cloud-sim-uj/PySDM/pull/717) the code snippets included in the JOSS paper will be executed as a part of PySDM CI build to ensure the example code works well with future updates to PySDM.

slayoo commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

slayoo commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

PDF failed to compile for issue #3219 with the following error:

 Error reading bibliography file paper.bib:
(line 162, column 3):
unexpected "u"
expecting space, ",", white space or "}"
Looks like we failed to compile the PDF

slayoo commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

slayoo commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

dhhagan commented 2 years ago

Hey @darothen - just wanted to ping you one more time now that the holidays are over. Is there anything else you need from the authors to finish up this submission?

dhhagan commented 2 years ago

@whedon remind @darothen in 1 day

whedon commented 2 years ago

Reminder set for @darothen in 1 day

whedon commented 2 years ago

:wave: @darothen, please update us on how your review is going (this is an automated reminder).

dhhagan commented 2 years ago

Hey @slayoo I am going to try to reach Dan R. offline to see if he can finish this up before needing to find another reviewer. Sorry for the delay here!

darothen commented 2 years ago

Sorry for the delay folks; between job change + holidays + AMS it took far longer than I had hoped to check in here.

I reviewed all of the changes and the discourse from the past few months. I think all of my comments/recommendations have either been directly addressed or addressed in the spirit of the substantial improvements and additions made to the library since the first version I reviewed last year. In my opinion, the work here is excellent and is ready for publication in JOSS without any additional modifications.

slayoo commented 2 years ago

@whedon generate pdf from branch master

whedon commented 2 years ago

Attempting PDF compilation from custom branch master. Reticulating splines etc...

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

slayoo commented 2 years ago

@dhhagan hope all is OK. If possible, please update us on the workflow and further requirements given the above messages from the reviewers. Thank you

dhhagan commented 2 years ago

Hey @slayoo I should have an update for you soon.

dhhagan commented 2 years ago

@slayoo Can you please do the following:

[ ] Make a tagged release and archive
[ ] Report the version number and archive DOI in this thread

slayoo commented 2 years ago

We have moved ahead of the "v1" branch some time ago for this project, so here are DOIs for Zenodo-archived last release from the v1 branch described in this paper:

PySDM v1.27: https://doi.org/10.5281/zenodo.6321270
PySDM-examples v1.27: https://doi.org/10.5281/zenodo.6321279

Thank you

dhhagan commented 2 years ago

@slayoo Can you update the title and author list in zenodo to make it match the paper?

dhhagan commented 2 years ago

@editorialbot set v1.27 as version

editorialbot commented 2 years ago

Done! version is now v1.27

openjournals / joss-reviews