[REVIEW]: PyToughReact – A Python Package for automating reactive transport and biodegradation simulations.

editorialbot commented 7 months ago

Submitting author: !--author-handle-->@temmy222@matthewfeickert<!--end-editor-- Reviewers: @blakeaw, @frank1010111, @dkedar7 Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/b241a294dfbb4d0a3184a2240c59d67d"><img src="https://joss.theoj.org/papers/b241a294dfbb4d0a3184a2240c59d67d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/b241a294dfbb4d0a3184a2240c59d67d/status.svg)](https://joss.theoj.org/papers/b241a294dfbb4d0a3184a2240c59d67d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@blakeaw & @frank1010111, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @matthewfeickert know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @blakeaw

📝 Checklist for @frank1010111

📝 Checklist for @dkedar7

editorialbot commented 7 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 7 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.cageo.2012.04.008 is OK
- 10.1016/j.cageo.2016.06.014 is OK
- 10.21105/joss.02412 is OK
- 10.1111/j.1745-6584.2008.00462.x is OK

MISSING DOIs

- No DOI given, and none found for title: PyTOUGH: a Python scripting library for automating...
- No DOI given, and none found for title: Tecplot360 User’s Manual

INVALID DOIs

- None

editorialbot commented 7 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.23 s (820.6 files/s, 248785.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                            67           3627            201          31647
Python                          63           2160           4446           7813
SVG                              1              0              0           2671
JavaScript                      10            124            194            789
CSS                              4            190             35            780
reStructuredText                34            402            754            616
Markdown                         2            128              0            415
TOML                             1              9              3             66
YAML                             3             10             18             54
TeX                              1              0              0             46
INI                              1              7              9             33
DOS Batch                        1              8              1             26
make                             1              4              7              9
-------------------------------------------------------------------------------
SUM:                           189           6669           5668          44965
-------------------------------------------------------------------------------

Commit count by author:

   304  temmy222
     1  dependabot[bot]

editorialbot commented 7 months ago

Paper file info:

📄 Wordcount for paper.md is 2011

✅ The paper includes a Statement of need section

editorialbot commented 7 months ago

License info:

✅ License found: MIT License (Valid open source OSI approved license)

editorialbot commented 7 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

matthewfeickert commented 7 months ago

@editorialbot set v1.0.3 as version

editorialbot commented 7 months ago

Done! version is now v1.0.3

matthewfeickert commented 7 months ago

@blakeaw, @frank1010111 Thanks for agreeing to review this submission! This is the review thread for the paper. All of our communications will happen here from now on. :+1:

As you can see above, you each should use the command @editorialbot generate my checklist to create your review checklist. @editorialbot commands need to be the first thing in a new comment.

As you go over the submission, please check any items that you feel have been satisfied (and if you leave notes on each item that's even better). There are also links to the JOSS reviewer guidelines. I find it particularly helpful to also use the JOSS review criteria and review checklist docs as supplement/guides to the reviewer checklist @editorialbot will make for you.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#6486 so that a link is created to this Issue thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 4 weeks. Please let me know if either of you require some more time (that's perfectly okay). We can also use @editorialbot to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@matthewfeickert) if you have any questions/concerns.

frank1010111 commented 7 months ago

Review checklist for @frank1010111

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/temmy222/PyTOUGHREACT?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@temmy222) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[ ] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[ ] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

frank1010111 commented 7 months ago

Hi @temmy222

I have a question on authorship: I see that you're responsible for all the code commits and are listed as the sole author of the python package, but you're sharing authorship with Ipsita Gupta on the JOSS paper. I know that the official JOSS guidelines on authorship allow for non-code contributions to count, but could you confirm Dr. Gupta's contribution to this publication?

Thank you

matthewfeickert commented 7 months ago

Thank you for opening up Issues on https://github.com/temmy222/PyTOUGHREACT as you move through the review, @frank1010111! This is the ideal reviewer workflow, so this is great to see. :rocket:

blakeaw commented 7 months ago

Review checklist for @blakeaw

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/temmy222/PyTOUGHREACT?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@temmy222) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

matthewfeickert commented 7 months ago

As all reviewers have their checklist generated now I'll have @editorialbot give us reminders in 3 weeks to follow up on the initial state of the review.

matthewfeickert commented 7 months ago

@editorialbot remind @blakeaw in 3 weeks

editorialbot commented 7 months ago

Reminder set for @blakeaw in 3 weeks

matthewfeickert commented 7 months ago

@editorialbot remind @frank1010111 in 3 weeks

editorialbot commented 7 months ago

Reminder set for @frank1010111 in 3 weeks

matthewfeickert commented 7 months ago

@editorialbot add @dkedar7 as reviewer

editorialbot commented 7 months ago

@dkedar7 added to the reviewers list!

matthewfeickert commented 7 months ago

@temmy222 @dkedar7 has responded to an earlier email request for reviewers that I had made before the review started, and as we require two reviewers, but it is nice to have three, I've added them as well.

@dkedar7 thanks for agreeing to review! Please check out https://github.com/openjournals/joss-reviews/issues/6486#issuecomment-1996385081 for some reviewer instructions and tips from me and let me know here if you have any questions. :+1:

dkedar7 commented 6 months ago

Review checklist for @dkedar7

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/temmy222/PyTOUGHREACT?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@temmy222) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[ ] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[ ] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[ ] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[ ] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

editorialbot commented 6 months ago

:wave: @blakeaw, please update us on how your review is going (this is an automated reminder).

editorialbot commented 6 months ago

:wave: @frank1010111, please update us on how your review is going (this is an automated reminder).

frank1010111 commented 6 months ago

So far my review is going well, but one hang-up is testing the functionality of the software. At the moment, I have no way to verify that PyTOUGHREACT produces valid TOUGH scripts when given valid inputs. The automated tests do not check outputs to ensure they conform to any particular grammar. That makes it really hard to check that particular box.

temmy222 commented 5 months ago

Hi @frank1010111 @matthewfeickert @blakeaw. I believe I resolved most of the issues raised. Please let me know if any are still left. Thank you

matthewfeickert commented 5 months ago

The automated tests do not check outputs to ensure they conform to any particular grammar. That makes it really hard to check that particular box.

@frank1010111 Can you check if https://github.com/temmy222/PyTOUGHREACT/issues/8 has been properly resolved to address this concern? It appears the issue was closed without addressing adding functionality testing. If that's correct it should be reopened.

matthewfeickert commented 5 months ago

@blakeaw, @frank1010111, @dkedar7 :wave: Can you all please provide updates on the status of your reviews? If there are things in the submission code that are blocking your reviews from progressing please open Issues on https://github.com/temmy222/PyTOUGHREACT/issues/ that summarize them and link them back here.

blakeaw commented 5 months ago

Apologies for my slow completion of this review. Things have been a little busier than I expected. I will try to finish reviewing everything this week and/or open any additional issues as needed.

frank1010111 commented 5 months ago

Hi @matthewfeickert :wave: A few issues are still outstanding

General checks

Reproducibility: I don't have a copy of TOUGHREACT, so I have reached out to the authors of the software, given them the files generated by the tutorials, and am waiting to hear back from them on whether the results are valid inputs for TOUGHREACT

Functionality

Functionality: Same as reproducibility

Documentation

Functionality documentation: A few of the sub-modules are not documented, but the documentation is in a much better state than at the beginning of the review Automated testing: The tests that are included don't prove that the software is functional yet

Software paper

This needs a lot of work, and I created an issue tracking the acceptability criteria here: https://github.com/temmy222/PyTOUGHREACT/issues/18

After these issues are resolved, I will be ready to conclude my review.

temmy222 commented 5 months ago

Hi @frank1010111

I added additional testing to test different pieces of the software independently. I will continue to add more tests. The coverage for the tests is now about 70%.

The documentation has been further improved.

The paper has been modified based on the checklist you specified in the issue.

matthewfeickert commented 5 months ago

@frank1010111 @blakeaw Thank you for your continued reviews, and thank you all for your patience with my lack of responsiveness (I have been traveling for work internationally for the last weeks and my email is unfortunately quite a mess at the moment).

Reproducibility: I don't have a copy of TOUGHREACT, so I have reached out to the authors of the software, given them the files generated by the tutorials, and am waiting to hear back from them on whether the results are valid inputs for TOUGHREACT

@frank1010111 Thank you for raising this point. Until now it has not been clear to me that LBNL's TOUGHREACT is not open source. This information is missing from the paper draft and should be disclosed. I will consult with the other JOSS editors on how to proceed, as we note in the review criteria docs

What about submissions that rely upon proprietary languages/development environments?

As outlined in our author guidelines, submissions that rely upon a proprietary/closed source language or development environment are acceptable provided that they meet the other submission requirements and that you, the reviewer, are able to install the software & verify the functionality of the submission as required by our reviewer guidelines.

If an open source or free variant of the programming language exists, feel free to encourage the submitting author to consider making their software compatible with the open source/free variant.

The reviewers are not able to install the software, which unless this can be overcome seems like a blocker moving forward.

@temmy222 I would recommend that you make PRs that target the Issues that the reviewers open up (like https://github.com/temmy222/PyTOUGHREACT/issues/18, https://github.com/temmy222/PyTOUGHREACT/issues/22) and ask them for clarification or review in their Issues before you close them. This will make the reviewer work much easier and more efficient.

matthewfeickert commented 5 months ago

@temmy222 The dependency on TOUGHREACT is currently a problem, as it is closed source and proprietary and the reviewers are required to be able to test the functionality of the software.

As noted in the Submitting a paper to JOSS docs

Submissions using proprietary languages/development environments

We strongly prefer software that doesn’t rely upon proprietary (paid for) development environments/programming languages. However, provided your submission meets our requirements (including having a valid open source license) then we will consider your submission for review. Should your submission be accepted for review, we may ask you, the submitting author, to help us find reviewers who already have the required development environment installed.

All of the reviewers are going to need to be able to have at least runtime access to TOUGHREACT for the review to proceed. Please let us know what your plan is here. I would suggest starting by contacting LBL RE: seeing if there is any special consideration for licensing for academic journal reviews.

Please also reopen https://github.com/temmy222/PyTOUGHREACT/issues/8 as this can not be closed until this is resolved.

matthewfeickert commented 4 months ago

@editorialbot remind @temmy222 in 1 week

editorialbot commented 4 months ago

Reminder set for @temmy222 in 1 week

temmy222 commented 4 months ago

Hi @matthewfeickert. Apologies for delayed response. I have contacted the authors of TOUGHREACT to see if they can provide an executable. I will let you know their response. Thank you

editorialbot commented 4 months ago

:wave: @temmy222, please update us on how things are progressing here (this is an automated reminder).

matthewfeickert commented 3 months ago

@temmy222 Can you please give us an update on the interactions with the TOUGHREACT authors?

matthewfeickert commented 1 month ago

@temmy222 I've been away for a few weeks for work related travel and am coming back to follow up on reviews. Can you please give an update on the status of things by the end of this week?

temmy222 commented 1 month ago

Hi @matthewfeickert. I still havent heard back from the authors of TOUGHREACT. Since this is a requirement for publishing in JOSS, I will like to withdraw the paper for publication. Thank you to everyone who has spent time reviewing the paper. The quality of the software has improved as a result of the reviews

matthewfeickert commented 1 month ago

@temmy222 Sorry for the slow reply here — I've been traveling internationally for work again. Thanks also for letting us know on the state of things and we understand and respect your decision to withdraw for the time being. If in the future you are able to broker a situation with the TOUGHREACT authors such that reviewers are able to get a licence for the software please feel free to resubmit to JOSS and reference this review. Please also feel free to follow up with me either on this issue or on email if you have questions or feedback.

@blakeaw, @frank1010111, @dkedar7 Thank you all for your work on reviewing this submission. We all appreciate it.

matthewfeickert commented 1 month ago

@editorialbot withdraw

editorialbot commented 1 month ago

I'm sorry @matthewfeickert, I'm afraid I can't do that. That's something only eics are allowed to do.

matthewfeickert commented 1 month ago

@openjournals/pe-eics Can you please withdraw the paper given https://github.com/openjournals/joss-reviews/issues/6486#issuecomment-2316623003?

kyleniemeyer commented 1 month ago

@editorialbot withdraw

editorialbot commented 1 month ago

Paper withdrawn.

kyleniemeyer commented 1 month ago

@temmy222 although publication in JOSS isn't possible for your package at this time, you can still make your software citeable by following GitHub's guide to creating a permanent archive via Zenodo.

openjournals / joss-reviews