[REVIEW]: pyrexMD: Workflow-Orientated Python Package for Replica Exchange Molecular Dynamics

[whedon]

Submitting author: @ArtVoro (Arthur Voronin) Repository: https://github.com/KIT-MBS/pyREX Version: v1.0 Editor: @jgostick Reviewers: @janash, @rosecers, @jarvist Archive: 10.5281/zenodo.5744760

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/957b0672d03306d75cc12fada3a6b6da"><img src="https://joss.theoj.org/papers/957b0672d03306d75cc12fada3a6b6da/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/957b0672d03306d75cc12fada3a6b6da/status.svg)](https://joss.theoj.org/papers/957b0672d03306d75cc12fada3a6b6da)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@janash & @rosecers, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @jgostick know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @janash

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@ArtVoro) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

I believe that this check mark is what has been causing the delay for both me and @rosecers. Unfortunately, when this paper was first submitted, the package was not well-documented and was hard to understand. We have asked for some clarification and code clean up. Unfortunately, I feel that even with the improvements this work does not meet the requirements of JOSS for substantial scholarly effort and I can't recommend this package for publication in JOSS at this time.

JOSS defines the following for substantial scholarly effort:

As a rule of thumb, JOSS’ minimum allowable contribution should represent not less than three months of work for an individual. Some factors that may be considered by editors and reviewers when judging effort include:

• Age of software (is this a well-established software project) / length of commit history.

• Number of commits.

• Number of authors.

  This package has only one author.

• Total lines of code (LOC). Submissions under 1000 LOC will usually be flagged, those under 300 LOC will be desk rejected.

  While this submission appears to fit this criteria, upon closer inspection this is questionable. Many of the functions in misc.py, for example do not add any functionality that are not in Python already (for example

This package has many more lines of code than are needed to achieve its objective. Many of the functions are what I would call light wrappers around other libraries and do not add much functionality themselves. Many functions (particularly in misc.py reimplement functionality that are already available in Python or have statements that are unnecessary.

• Whether the software has already been cited in academic papers.

  The software has not been cited in academic papers

• Whether the software is sufficiently useful that it is likely to be cited by your peer group.

  I am not convinced of this point because the software is mostly thin wrappers around existing code.

I unfortunately cannot recommend this software package for publication in JOSS at this time. I think it would be more appropriate for the author to obtain a DOI through something like Zenodo and resubmit for publication if the package proves sufficiently useful.

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @rosecers

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@ArtVoro) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @janash, @rosecers it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/S0009-2614(99)01123-9 is OK
- 10.1063/1.2056540 is OK
- 10.1002/jcc.20291 is OK
- 10.1073/pnas.1111471108 is OK
- 10.1021/acs.jpcb.6b13105 is OK
-  10.25080/Majora-629e541a-00e  is OK
- 10.1002/jcc.21787 is OK
- 10.1093/bioinformatics/btx789 is OK
- 10.5281/zenodo.2654393 is OK
- 10.1371/journal.pone.0242072 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.87 s (96.6 files/s, 92505.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                            22           4640             66          20004
JavaScript                      14           2404           2467           9203
Python                          21           1955           4952           5588
SVG                              1              0              0           2671
CSS                              5            208             45            809
Jupyter Notebook                10              0          24370            551
Markdown                         2             45              0            135
TeX                              1             12              0            122
reStructuredText                 6             38             66             43
DOS Batch                        1              8              1             26
make                             1              4              7              9
-------------------------------------------------------------------------------
SUM:                            84           9314          31974          39161
-------------------------------------------------------------------------------

Statistical information for the repository 'ceb0d5cffa2b47c6e28644d8' was
gathered on 2021/06/01.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Arthur                          50         20471           4475           63.93
Arthur Voronin                  64         12204           1870           36.07

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Arthur                    26569          129.8          1.5               14.26

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[whedon]

:wave: @janash, please update us on how your review is going (this is an automated reminder).

[whedon]

:wave: @rosecers, please update us on how your review is going (this is an automated reminder).

[janash]

Hi @whedon - I've gone over the COI and general checks and installed the dependencies. Planning to closely review the software later this week and this weekend!

[rosecers]

Conflict of interest

• [X] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [X] I confirm that I read and will adhere to the JOSS code of conduct. General checks
• [X] Repository: Is the source code for this software available at the repository url?
• [X] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [X] Contribution and authorship: Has the submitting author (@ArtVoro) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

My largest concern comes from the citability and need within the community -- there are no publications that mention "pyrexMD", and the publications of the two authors using REMD (3 as of 2020) have not yet been cited. To this end, the authors need to comment on two things: the breadth of scope of this software (it appears quite narrow), and whether similar workflows would be supported by pre-existing software (particularly SSAGES). All citation metrics checked via searching WebOfKnowledge and GoogleScholar.

Functionality

• [X] Installation: Does installation proceed as outlined in the documentation?
• [X] Functionality: Have the functional claims of the software been confirmed?

The results contained in the paper summary are reproducible in the examples. Beyond that, no functional claims have been made.

• [X] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

The authors have an explicit statement of need in their write-up; however, mentioned above, they have not addressed the breadth of the scope of this package and the currently available software packages intended to this end.

• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.

The requirements for pyrexMD need to be more prominently featured. For example, some packages require specific python versions, which may not be clear upon installation. pytest and autopep8should be added as a requirement for testing and examples purposes.

• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

The authors include examples, however, these are not well-documented and require more explanation / commenting, without which I can't know what the examples are intending to demonstrate or if the results are as expected.

Additionally, the authors should embed some examples in the documentation -- if there is a user considering using this package, they may want to see a demonstration of the package before downloading and without searching through the API.

• [X] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

There is API documentation -- it is sufficient, but could greatly benefit from further formatting and editing.

• [X] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

The tests are sufficient, however, they currently raise several warnings, which should be remedied. As for coverage, no coverage test or CI is implemented to check on the functionality.

• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

No.

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

The summary is minimal -- the authors should expand considerably on the REX methodology and the workflows supported by pyrexMD.

• [ ] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?

There is an explicit statement of need that needs to be amended (described in preceding comments).

• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?

No.

• [ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

The authors are fairly terse in their writing, and need to expand in several places (e.g. the figure explanations and applications).

• [X] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Overall, the examples, use cases, and need for this software need to be expanded upon. The examples should include commentary as to the expected behavior and "behind the scenes" operation of the software, including an introductory paragraph at the top of the example to explain what is being demonstrated. The authors have not identified use cases for this software beyond their own publications -- they should detail in their write-up examples of the sorts of analyses possible with this software. Finally, there needs to be some statement as to the novelty of this software -- is it fulfilling a need within the community which is not currently being met?

[ArtVoro]

thank you for the feedback. i will try to solve most of the things mentioned over this weekend.

[janash]

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@ArtVoro) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

This software mainly seems to be a wrapper around three libraries - gromacs, nglview, and mdanalysis. The software has not yet been cited in academic papers. In its current state, the paper does not appear to meet the definition of substantial scholarly effort.

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?

I had difficulty installing the software due to a missing dependency with mpi4py on my mac. I used brew install mpich which then allowed my installation to proceed. I recommend adding a CI action (GitHub Actions or other of your choice) to test installation on supported operating systems. You should also run your tests using CI and add code coverage.

• [] Functionality: Have the functional claims of the software been confirmed?

I feel that I need expanded example usage documentation to determine this. The main functional claims of the software are in the last paragraph of the statement of need. It might be beneficial to number the examples as tutorials and reference the tutorial numbers in the paper.

• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

No performance claims made.

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.

There is a list of installation instructions. I did encounter an error with dependencies as described above. I have a few comments about the installation instructions:

• there is no need to do an "editable' -e install for most users, unless you expect them to be pulling from the repo for updates.
• I recommend putting the package on the Python Package Index (PyPI) so that users can install with pip.

• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

There is example usage, but the documentation associated with the example usage needs to be expanded. Please use a combination of markdown and code cells in your example notebook to give an explanation of tutorials.

• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

Please move documentation link on the readme to top. There is API documentation that is adequate for using the package. However, it would benefit from additional documentation or better organization.

• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

There is a set of tests which can be run with pytest (manual steps), but there are no automated tests.

• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

No.

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?

No. Please expand this portion.

• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Summary The author needs to give more justification for the scholarly effort of this package as suggested by @rosecers . The documentation should be improved and expanded, with more explanation given for the examples. A tutorial describing a full workflow (set-up, running simulation, and analysis) should be provided as well.

I also suggest automated building and testing of your package using GitHub Actions, and putting your package on PyPI for ease of installation.

[rosecers]

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?

I had difficulty installing the software due to a missing dependency with mpi4py on my mac. I used brew install mpich which then allowed my installation to proceed. I recommend adding a CI action (GitHub Actions or other of your choice) to test installation on supported operating systems. You should also run your tests using CI and add code coverage.

I'd like to echo @janash's difficulty with installation, as I also had issue with mpi4py on my mac; I concur that the authors should test installation on supported operating systems. Ultimately, I needed to run:

cd $INSTALL_DIR
git clone https://github.com/KIT-MBS/pyrexMD.git
cd pyrexMD
conda install clangxx_osx-64
pip install mpi4py
python -m pip install -e .

[jgostick]

@janash and @rosecers, thanks for the thorough review so far! @ArtVoro, you have a clear list of things to work on, so I'll check back next week.

[ArtVoro]

sorry for the late response.

I did some CI with building and testing the software (took me longer than expected) and reworked the tutorials (juypter notebooks). Now ill expand the online documentation and then revisit the paper.

[ArtVoro]

Update about the tasks:

• reworked the installation instructions
• added code coverage
• added CI via github actions: build gromacs -> build python package -> run tests -> update documentation (build and tests are performed on ubuntu & mac)
• reworked example jupyter notebooks
• added a quick guide to online documentation (https://kit-mbs.github.io/pyrexMD/quick_guide.html)

Todo:

• tasks related to the paper itself

[ArtVoro]

A few notes:

• The package was private until the JOSS submission so obviously there are no citations about the software. However I am submitting a paper within the next month where I explicitly call the name of the software.
• I intend to upload the package to PyPI but will wait until the end of this review because (as far as I know) it is not possible to reuse a package version number

[ArtVoro]

@jgostick we pushed a new version of paper.md and extended the citations accordingly. I think all points mentioned from the reviewers should be covered now.

[ArtVoro]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[janash]

Thank you @ArtVoro, the documentation is much improved. I would still like to see some additional comments/explanation in the example notebook tutorials.

I have concerns about code quality and maintainability. Particularly, a large amount of code is repeated when it need not be. Please see the issue I have filed on your repository (https://github.com/KIT-MBS/pyrexMD/issues/1). I would suggest some sort of decorator for the gromacs functions instead of the current implementation.

This is relevant for a few reasons: 1 - Repeating code in this way is a bad practice. Since the functionality is the same for all these parts, it will be harder to maintain or change in the future (think having to change several places vs one place if you want to change behavior). 2. JOSS lists lines of code as a metric in scholarly effort.

[ArtVoro]

thanks for the feedback. the repeated code on gmx.py was actually on my todo list but had low priority. however since its not much effort i can fix it over this weekend.

about the notebooks, i can take another look at it and add more comments/explanation as you suggested but I dont know when exactly. Im working on 2 papers right now and my schedule is already packed. due to the delays in reviews I want to send out the drafts first.

[ArtVoro]

i fixed the repeated code by calling help functions now.

[rosecers]

Hey @ArtVoro, I've finally been able to get another look at this and was once again in a tornado of trying to install the package and run the tests.

Comments on installation and testing:

• You have included both a requirements.txt and requirements_python3.6.txt. You should instead use environment markers to automatically detect the two and avoid confusion.
• In general the installation requirements are quite stringent -- are all of these exact versions necessary?
• Issues with requirements_python3.6.txt:
  • heat=1.0.0 requires python >=3.7
  • ipywidgets 7.4.2 depends on widgetsnbextension~=3.4.0, and the requested version is 3.5.1
• Running pyrexMD/tests/test_gmx.py currently results in an error due to a segfault. I am not a regular GROMACS user, so this may be due to my install, but should be addressed nonetheless in case it is due to memory concerns: gmx convert-tpr -s pyrexMD/examples/files/protein//../traj/traj_protein.tpr -o /Users/rosecers/work_folders/cosmo/tests/pyrexmd/pyrexMD/traj_protein_protein.tpr

Comments on what I've been able to get through in the code:

• gmx.py: I am concerned with how much of gmx.py is modified versions of GromacsWrapper or effectively switch statements. For the functions which are primarily purposed in GromacsWrapper, I would suggest you create a fork of GromacsWrapper to modify the code or propose the modified functions directly to that repository if you think these functions will be appreciated by the community at large. I don't think that they necessitate inclusion into a new package.

• misc.py: How much of this is original code, and how much are wrappers to slim down code or add a print statement? This is a rather bloated file, most of which should be deleted.

  • Please delete everything that is effectively

        import module 
        def func(*args, **kwargs):
            return module.func(*args, **kwargs)

    and just refer to the original module functions or a set of module functions instead. There is also code for which the functionality already exists in the current dependencies. Examples: cwd, pwd, isdir, isfile, pathexists, realpath, dirpath, get_filedir, mkdir, and many more.

  • apply_matplotlib_rc_settings can just be a matplotlib style sheet.
  • is update_alias_docstring just for your own use to create docstrings? Where else is it used?
  • What are the functional purposes of CONFIG? With the exception deepcopy_without and update_by_alias, it appears to contain functions that are just renamings, not even wrappers, of built-in functions.

  All in all the documentation, paper, and code have been improved, but still need to be streamlined. I'll try to give another look-over next week.

[ArtVoro]

Hi @rosecers. Ill adress your points one by one

Comments on installation and testing:

• i did merge both requirements files into one and added the environment markers as you suggested. i expect people to use the setup.py however and just refer to requirements.txt to look up which versions were used as a stable version (but maybe thats just my own thinking).
• i catched some versions which were defined as == but loosened to >=. The packages which are left to == right now should stay this way as the did break my code at some point (especially nglview)
• about the segfault with gromacs, it must be something with your installation. the tests pass on my local PC and laptop and also during the github CI test. you can maybe refer to https://github.com/KIT-MBS/pyrexMD/blob/master/.github/workflows/gh-actions-ci.yml -> ## job 1 ## (build-gromacs) -> follow the segments with the run: ... commands for your OS
• small additional info about segfault errors etc....there are very specific errors with certain CPU chips and gromacs versions (e.g. on AMD Ryzen 3000 series CPUs, the hardware random number generator (RDRAND) can behave incorrectly, always returning -1 (0xFFFFFFFF) -> but this is a compatilibity issue related to gromacs.

Comments on code:

• while i see your point with gmx.py I have to disagree with your proposal. The major point is that the current gromacswrapper does exactly what it is supposed to do, i.e. it allows the user to execute gromacs commands via python with the same behavior as it is expected via commandline. The functions which are used in gmx.py are just a very small portion (like 10 out of 105 gromacs functions or so) which have some added functionality because I have to streamline how I handle data in my workflows and are basically just some QoL improvements in my case. However since every gromacs user handles their work slightly different, e.g. by naming their files, data structure etc. forking and then doing pull request seems not reasonable. Besides this way I have a clear overview which functions were used and extended without having 1:1 copy of unchanged code and making it hard to find the differences.
• regarding misc.py however I agree that removing some functions (mainly the os package stuff) was necessary. misc.py is a collection of miscellaneous functions which i use somewhat regular. I wanted to do a cleanup at some point but just forgot about it. I went through the file now and removed all functions which are clearly unnecessary for the package. A few functions which you maybe adressed (mkdir, cd, joinpath) i will keep however since I use them ALOT and they have certain behavior added which I expect and dont want to repeat over and over.
• matplotlib style sheet -> thanks for suggestion. I was not aware this existed.
• update_alias_docstring is just used to by me. I can remove it and just do "alias = function" in the code but I personally dislike that the function name does not get updated there when using "help(function)" or "help(alias)". if you want it removed just tell me.
• CONFIG creates an object which behaves very similar to a dictionary. I use almost everywhere to store and modify config variables in functions or pass some config variables between function calls. I think you refer to methods such as setitem, delitem etc but if I remember correctly such methods are not set by default for an object and i coded them explicitly so that CONFIG object behaves in a certain way. Besides it helps me to get a quick overview of the allowed kwargs of each function.

[jgostick]

Hi @ArtVoro, it looks like the reviewers have given you a long list of to-dos. Can you try to address them all in such a way that they are able to finalize their review checklists?

[ArtVoro]

Hi @jgostick, as far as I'm aware I have addressed all tasks which were mentioned following all posts above. Maybe you can give me an example how I could make it more obvious to finalize the review checklists? The only open tasks might be:

@janash

Thank you @ArtVoro, the documentation is much improved. I would still like to see some additional comments/explanation in the example notebook tutorials.

Are the comments/explanations of the notebooks sufficient to be accepted? If not can you give me more directions where exactly you want more comments/explanation so i can fix this precisely?

@rosecers

All in all the documentation, paper, and code have been improved, but still need to be streamlined. I'll try to give another look-over next week.

I'm not quire sure what you mean by streamline so if you can give some clear instructions, I can address this.

[jgostick]

Hi All, there is a lot going on in this review...I would appreciate it if the reviewers (@rosecers and @janash) could use the checklist at the very top of this thread to track which issues have been addressed. This way I can have a single source of truth for the status of the reviews.

[ArtVoro]

@jgostick @rosecers @janash would be nice if we could finalize the review :) there was no response for quite some time so i hope the review didnt get burried under all other email notifications.

[janash]

@ArtVoro - Thank you for your efforts. I do think the package is much improved from when it was first submitted. However, I can't recommend it for publication at this time.

@ArtVoro @jgostick , please see my comments under "Substantial Scholarly Effort", above.

[rosecers]

@ArtVoro - Thank you for your efforts. I do think the package is much improved from when it was first submitted. However, I can't recommend it for publication at this time.

@ArtVoro @jgostick , please see my comments under "Substantial Scholarly Effort", above.

@janash @jgostick I agree on both points -- the considerable improvement as well as the remaining concerns about scope and applicability. As such, I cannot recommend it for publication either.

[jgostick]

@whedon check repository

[whedon]

Wordcount for paper.md is 1258

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.30 s (337.4 files/s, 210645.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                            24           4781             72          19894
JavaScript                      14           2405           2473           9211
Python                          28           2562           5606           7582
SVG                              1              0              0           2671
CSS                              5            208             45            810
Markdown                         3            176              0            499
Jupyter Notebook                 9              0           2934            344
YAML                             4             18             43            275
TeX                              1             24              0            199
reStructuredText                 8             41             71             43
DOS Batch                        1              8              1             26
make                             1              4              7              9
INI                              1              1              0              8
Bourne Shell                     1              1              0              4
-------------------------------------------------------------------------------
SUM:                           101          10229          11252          41575
-------------------------------------------------------------------------------

Statistical information for the repository 'd52ad5522d6ab3862603f4fc' was
gathered on 2021/08/30.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
ArtVoro                          1            72             12            0.10
Arthur                          79         28523           9930           44.68
Arthur Voronin                  85         28922          18599           55.22

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
ArtVoro                      63           87.5          1.3                4.76
Arthur                    13517           47.4          3.6                9.42
Arthur Voronin            16259           56.2          0.0               16.64

[jgostick]

Hi @ArtVoro, at JOSS we do not reject papers, but prefer to allow the authors to work on them until their acceptable. The two reviewers did a very thorough job, but they are out of time to stick around for more rounds of review and review. Instead, one of JOSS's associate editors (@jarvist) will chime in on some issues that need to be addressed. I think this is largely documentation and explanation of why this code is useful.

[jarvist]

Dear all, I'm writing this as an informal editorial assessment. I've read the paper, looked at some of the code, and read the two reviewers comments.

The codes are relatively lightweight wrappers around the GROMACS python bindings, and some analysis tools. There is not much algorithmic complexity, instead these are mostly glue codes. However, what they seem to enable is a Jupyter-notebook based system to construct replica-exchange MD runs, and then analyse the results, all on the computational cluster. This seems quite useful. I don't have a problem with the coding style, but do point out that thin-wrappers are by their nature a leaky abstraction. Combined with the inter-dependency of the packages used, I expect these codes to be quite brittle to any upstream updates.

I think the paper could be made more clear. I do not find the central statements of "'all-purpose' toolkit" or "workflow-orientated functions" to be specific or descriptive. I would like the PAPER.md to be rewritten to state the functionality that pyrexMD offers more directly. The figures in the paper are pretty, but they mean little to someone who is not an expert in this particular MD method.

I share some of the concerns of the reviewers about how likely this package will be taken up as a community code. MD research groups generally have internal scripts. I would suggest that simplifying the repository README, and providing teaching examples might help.

The documentation is quite vague: I think it would be useful for everyone if the very first paragraph stated that this package wrapped GROMACS and MDAnalysis python bindings to provide a Jupyter-notebook based environment to design, run and analyse replica-exchange MD.

If the paper and documentation could be made more clear and specific, I would recommend publication.

I am happy to offer more feedback and specifics in this thread, if that would be useful.

Best wishes, Jarv

[ArtVoro]

Hi everyone, first of all: thank you all for the additional feedback.

@jarvist i have a couple of comments and questions regarding your post.

1) did you check the quick guide? i made it to give an initial overview what to expect from the package. Would be nice if you can give feedback regarding this: https://kit-mbs.github.io/pyrexMD/quick_guide.html.

2) you mention that you want the documentation to be more clear and specific. I asked a few colleagues (all doing either coarse-grained or full atom MD simulations) some time ago to check the API docs and tell me if they find it sufficient to work with. They all told me that given the name of the function, and as I do the naming of the return values and the description of the return values it is enough for them to fully understand what to expect from the functions. Generally speaking, documentation can obviously always be improved but requires lots of work. At the moment I try aim to hit the "documentation is sufficient" mark.

As it is right now, each module of the API docs states

• what is expected in the module (e.g. HINT: This module contains functions related to (contact-guided) Replica Exchange Molecular Dynamics. It contains mainly functions to automate and speed-up the process of setting up REX simulations.)
• a small 'module example'
• then followed by all functions and their respective doc string (which also sometimes contain a small example)

Therefore I want to know what exactly you expect if you say you want it clearer and more specific. Can you maybe give one or two examples what to improve?

3) regarding the figures: I can obviously only display structure analyses figures and explicitly selected three figures for my target audience (= people doing MD/REMD & Structure Analyses) to display some 'highlights'

• something clear to everyone doing MD (here: RMSD curve with trajectory viewer). I dont think this figure can be improved.
• something related to contact-guided REMD (field-specific, here: TPR curve for ranked bias contacts). The figure shows more than just TPR and therefore requires some additional reading to fully understand but I tryed to explain the additional lines etc.. Should I expand on the figure description or what would you suggest to improve this?
• something related to structure comparison (here: LocalAccuracy figure with GlobalDistanceTest scores). I guess the only way to improve this figure is to explain shortly what GlobalDistanceTest means in case somebody does not know it yet and intoduce the two score variants and abbreviations TS (total score) and HA (high accuracy).

I hope with your answers I can understand better what you expect so that I can improve those aspects.

[ArtVoro]

quick note on current todos:

• edit first paragraph of documentation as suggested

quote of jarvist: I think it would be useful for everyone if the very first paragraph stated that this package wrapped GROMACS and MDAnalysis python bindings to provide a Jupyter-notebook based environment to design, run and analyse replica-exchange MD.

• I will sit down and rework PAPER.md and try to explain it clearer what exactly pyrexMD package does and offers, as suggested by @jarvist

[jarvist]

Dear ArtVoro,

I think the detailed documentation (APIs & docstrings) is of a fine standard for publication.

What I feel is either missing or too vague are the higher level descriptions, to understand what the package does, what it enables, why should someone use it.

I did see your quickstart guide, but again I didn't really understand the motivation for each section. The 'normal MD' looked like a standard command-line GROMACS workflow. But I look at those commands and wonder: do I have to supply the mdp files? What are the inputs and outputs? Do the functions generate files as side effects, and is this how the whole process hangs together? Also, ideally those examples should be runnable. (So provide a demo .pdb file etc.) Does your first example contain a typo? It looks like a mix of python and command line. Having them as runnable iPython notebooks helps avoid such errors.

gmx.grompp(-f"md.mdp", o="traj.tpr", c="npt.gro", t="npt.cpt")

A showing a side-by-side comparison between your package and doing replica-exchange 'manually' much also be illuminating for any potential users.

Similarly with the figures - I can see them, I can see they look quite nice, but I don't understand what the motivation for them is. Does the package provide a particularly easy way of generating such figures? Is doing this analysis in pyREX / Jupyter notebooks so much more easy?

I'm certainly not saying that the documentation needs totally rewriting, nor massively expanding - but more that it needs more sign posting.

Best wishes,

Jarv

[ArtVoro]

understood. i will try to improve those aspects today and tell you when both documentation and paper are reworked.

Also, ideally those examples should be runnable. (So provide a demo .pdb file etc.)

there is already an example folder with jupyter notebooks in the package included (with all necessary files). the readme page also states how to run them.

gmx.grompp(-f"md.mdp", o="traj.tpr", c="npt.gro", t="npt.cpt")

yes it is a typo. should be f="md.mdp". thanks for pointing that out.

[ArtVoro]

Hello @jarvist,

I collected additional feedback and updated the paper, readme and quick guide sections of the github documentation.

• git repo: https://github.com/KIT-MBS/pyrexMD
• readme: https://kit-mbs.github.io/pyrexMD/about_pyrexMD.html
• quick guide: https://kit-mbs.github.io/pyrexMD/quick_guide.html

The biggest changes are usually at the very beginning

• paper and readme: explicitly state who should be interested in this package and what its main goals are
• quick guide: reworked top section, module overview section added

Additionally, I made many small changes in the paper and quick guide with the focus of being more specific. I also noted that the code snippets are taken from the example folder and can be run locally to test the behavior.

Best regards Arthur

[ArtVoro]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[ArtVoro]

@jarvist one short comment on your previous statement:

I don't have a problem with the coding style, but do point out that thin-wrappers are by their nature a leaky abstraction. Combined with the inter-dependency of the packages used, I expect these codes to be quite brittle to any upstream updates.

I personally dont expect the code to be brittle. My main dependencies for wrapping are given by Gromacs and MDAnalysis. Gromacs is very stable and didnt have any big changes since their command-line syntax change in (i believe) version 2016. Besides I only use a small portion of very basic gromacs commands to streamline the setups.

Regarding MDAnalysis, I expect problems only during major version updates if they change some data structures. In this case they just recently had a major version update from 1.1.1 to 2.0.0 in August. 2 functions of my code got affected but I already addressed all changes in August.

[ArtVoro]

Hi @jarvist ,

I just wanted to check on the current status of the review. Seems like you didn't find the time yet to check the changes. I hope that we can finalize the review soon because I'm about to submit another paper where i want to reference the pyrexMD paper.

Best regards, Arthur

[jarvist]

Dear Arthur, My apologies - I was on parental leave until the 4th October. I will assess your corrections & provide feedback in the next week. Best wishes.

[ArtVoro]

Hi @jarvist, no worries and I hope you and your family are doing well.

Best regards, Arthur

[ArtVoro]

Hi @jarvist, do you have any feedback regarding the corrections? almost 2 months have passed since the last update. I'm sure I adressed all your points but am waiting for response now.

Best regards, Arthur

[jgostick]

I have also dropped the ball here. I will ping @jarvist on Slack to see how he's doing.

[ArtVoro]

@jgostick @jarvist Ok, I'm hoping for the best and that we can finalize the review. It's been quite some time and since I adressed all points I assume it just needs a quick check.

Best regards, Arthur

[jarvist]

Dear @ArtVoro , Thank you for your patience and apologies for my very late 2nd review! Teaching commitments & sick children have made for a very busy Autumn so far.

I've had a read through your paper, as well as the README.md and landing page of the documentation. I think the text is much improved, and it clearly states what the package does and why someone should be interested in it. As well as making a better publication, I'm think this will increase the likelihood of someone using your code.

I think the captions for the figures are useful for a peer in a nearby field to understand more of the capabilities. The text around the 'examples' section flows much better, and similarly motivates and explains what is going on. I think abstracting some of this text for the paper is a good idea.

Overall the documentation and paper have been made more clear and more specific, and I think they are now of sufficient quality to recommend publication.

Very minor typo - GROMACS is usually in all caps (as its an acronym), two instances were written as 'Gromacs' in your paper. There may be other instances in your main documentation.

[ArtVoro]

@jarvist thank you very much for your kind words and your help with the review.

You are right with the all caps of GROMACS. I checked all files and fixed all instances in the paper as well as some within the documentation. The updated version is already pushed to the repo.

@jgostick what are the next steps?