LiveCoMS Review

[ramess101]

@ejmaginn @drroe @jrelliottoh @dcarls0n

The reviewer comments are in.

Here is the email LiveCoMS sent me (and maybe they sent it to some of you as well):

Dear Dr. Messerly, Thank you for submitting your manuscript "Best Practices for Computing Transport Properties 1. Self-Diffusivity and Viscosity from Equilibrium Molecular Dynamics v1" to the Living Journal of Computational Molecular Science. The editorial team and a group of expert reviewers, including one student or postdoc reviewer, have assessed your submission and feel that it has potential for publication. We invite you to revise the paper and resubmit for further review. Please see the attached reviewer comments for further details about necessary revisions. Some particular points to examine: • Some questions were raised about rationales for some recommendations by reviewers. Please make sure all recommendations have specific rationales clearly listed. • When possible, all references should have literature citations. One rationale discussed the GROMACS manual; ideally, a more direct primary citation should be used. • The check lists for computing self-diffusivity/shear viscosity with the Einstein and Green-Kubo equilibrium approaches are identical, with the exception of some varying section numbers. It would almost certainly be clearer to consolidate the checklists, one per property, with alternate options for Einsten and Green-Kubo approaches. • It would be extremely helpful to put somewhat of a more quantitative requirement on what "middle" of the simulation is, and how to recognize it. Even if the subject is an open area of research, at least having quantitative measures is important for implementability. • Comments on the reviewer's need to use double precision for NVE vs. NVT (we can facilitate more information from the reviewer as necessary) Several other recommendations in addition to what is in the reviews: • It would be better to make recommendations in active voice "we recommend" versus passive voice "it is recommended" in order to clarify which recommendations are coming directly from the authors of this article, rather than the existing literature. • D.M.Z. lists funding acknowledgements, but is not an author. • For the list of extra sources near the beginning, the authors and titles should be provided in the lists, not just in the references (full details can be left in the references). • Is it possible to include records (text + headers of emails, perhaps?) of the permission on use of figures from other publishers, and the condidions, in the GitHub repository? This could avoid later questions about what the permissions were if the article were further shared. Your revised manuscript should be accompanied by detailed responses to the reviewers' comments and a summary of changes made to the manuscript. You should also include as an additional file a version that highlights changes in the manuscript. Suggestions for how to create such a version in LaTeX can be found in the author’s information on the LiveCoMS website. If there are formatting issues, we suggest downloading the newest template files, as they have recently been improved, which may fix some formatting concerns. We ask that you submit the revised version of your manuscript within 4 weeks. After that point in time we will presume that you have withdrawn your submission. Thanks again for your submission to LiveCoMS! Please feel free to contact me with any questions. Sincerely, Michael Shirts Editor, Living Journal of Computational Molecular Science

[ramess101]

Here are the Reviewer 1 comments:

Do you have additional comments for the author?

NA

How does the article address the specific reviewer criteria for the article type (Perpetual Reviews, Tutorials, Comparisons of Computational Software, and Lessons Learned), as described at https://livecomsjournal.github.io/policies/reviewer_information/?

NA

In what ways should the paper be improved to be easy to read, free from grammatical errors, have a professional presentation, and meet the article formatting guidelines laid out in the author policies (at https://livecomsjournal.github.io/authors/policies/)?

The article is written well.

To what extent does the article engage with current understanding in the scholarly community? If this is a revision, to what extent do the authors engage with the community participating on their GitHub version?

The article will hopefully contribute to good simulation practice in the field.

What is the level of significance of this work and its suitability for the journal? Is it likely to have a strong positive impact on the targeted set of readers? If this is a revision, what is the significance of the updated material?

The significance is high.

[ramess101]

Here are the Reviewer 2 comments:

Do you have additional comments for the author? I used this document while calculating viscosity for myself and it got me pretty much all the way there, so thanks!

The only difficulty I had was conserving energy in the NVE ensemble (which you cover in common pitfalls) using GROMACS 2018. I investigated the hints given in common pitfalls, but it turned out that I needed to use double precision. Rather than deal with slower double precision, I ended up using NVT, as shown to be acceptable by Basconi and Shirts, with a long time constant.

Great work!

How does the article address the specific reviewer criteria for the article type (Perpetual Reviews, Tutorials, Comparisons of Computational Software, and Lessons Learned), as described at https://livecomsjournal.github.io/policies/reviewer_information/?

There are short checklists (4 total depending on which property and which method you are using) Each point in the checklist references a section with more details on how to fulfill the checkbox. The checklist ensures a properly set up system and the generation of good statistics. I think it's worth considering moving "common pitfalls" to the second major bullet point (before "Post-simulation data analysis"). It's likely you can take steps to avoid them before running production simulations of many replicates.

Assertions are well-sourced from publications and using data created for paper (Figures 9 and 13) Explanations are accessible to people with moderate simulation training

In what ways should the paper be improved to be easy to read, free from grammatical errors, have a professional presentation, and meet the article formatting guidelines laid out in the author policies (at https://livecomsjournal.github.io/authors/policies/)?

The style is neat and clean. The figures are high enough quality and presented close to where they are talked about. It is generally easy to read and I believe the language used is accessible to non-experts.

I think it can be improved by another thorough read-through. There are some very long sentences and a few minor typos.

To what extent does the article engage with current understanding in the scholarly community? If this is a revision, to what extent do the authors engage with the community participating on their GitHub version?

The authors rely heavily on previous work and use real examples from highly cited articles. Also, at the beginning of the article, they provide a list of relevant sources for further reading.

What is the level of significance of this work and its suitability for the journal? Is it likely to have a strong positive impact on the targeted set of readers? If this is a revision, what is the significance of the updated material?

This work outlines, in detail, the recommended approach for the calculation of two common physical properties: diffusivity and viscosity. Since they are so common, it is a logical topic for inclusion in the journal. I believe this will have the greatest positive impact on people new to the field in addition to helping align the methodology of experienced users. It is to find the equations necessary to calculate viscosity and diffusivity. However, it is not obvious, especially for those new to the field, how to properly generate the data necessary for a theoretically sound calculation. The authors comprehensively explore the requirements necessary to achieve this and provide many good references for those who require more detail. The emphasis on system setup and running replicate simulations of appropriate length, backed up by literature examples, is extremely valuable.

[ramess101]

Here are the Reviewer 3 comments:

Do you have additional comments for the author?

The article itself is a great example for others in the community to follow when writing best practice guides for other topics. Look forward to the forthcoming NEMD article.

How does the article address the specific reviewer criteria for the article type (Perpetual Reviews, Tutorials, Comparisons of Computational Software, and Lessons Learned), as described at https://livecomsjournal.github.io/policies/reviewer_information/?

The checklists are easy to follow and should help readers avoid pitfalls.

For novice readers, there is an opportunity to make a small tweak to the checklists to improve their understanding. On page 5, experienced practitioners will understand why the checklist for calculating viscosity with the Green-Kubo equilibrium approach appears before the Einstein equilibrium approach. However, a person new to simulations might not catch the significance, and if not careful, not read the explanation in Section 6.3. It seems like a short note next to the checklists for viscosity, alerting readers to the intention behind switching the order of appearance of the methods (and directing them to the explanation in Section 6) would be helpful for novice readers.

In what ways should the paper be improved to be easy to read, free from grammatical errors, have a professional presentation, and meet the article formatting guidelines laid out in the author policies (at https://livecomsjournal.github.io/authors/policies/)?

The article is well-written and presented. I will be making this article required reading for trainees at all levels (including undergraduate students) in my research lab.

To what extent does the article engage with current understanding in the scholarly community? If this is a revision, to what extent do the authors engage with the community participating on their GitHub version?

This article aligns very well with the current literature and available supplementary resources. Well done!

What is the level of significance of this work and its suitability for the journal? Is it likely to have a strong positive impact on the targeted set of readers? If this is a revision, what is the significance of the updated material?

This article will have a very positive impact on the readers in the molecular simulation community. The key idea in this article is that it addresses the practical considerations the one must confront to compute self-diffusivity and viscosity from molecular dynamics simulations.

[ramess101]

@ejmaginn @drroe @jrelliottoh @dcarls0n

I think we should all read over these comments. If there is a comment/recommendation that you feel like you have a good revision/response for, please add it to this issue. We will then compile/combine the responses and resolve any disputes that might arise. Once you have finished with your responses (even if you don't provide one for each comment) please make it clear that you are finished so that we can finalize our revisions/responses.

I would like to resubmit in a week or two. Let me know if that is going to be a problem.

[drroe]

D.M.Z. lists funding acknowledgements, but is not an author.

Did they get an old copy of the PDF? I don't see it in the current repo.

[drroe]

Comments on the reviewer's need to use double precision for NVE vs. NVT (we can facilitate more information from the reviewer as necessary)

The only difficulty I had was conserving energy in the NVE ensemble (which you cover in common pitfalls) using GROMACS 2018. I investigated the hints given in common pitfalls, but it turned out that I needed to use double precision. Rather than deal with slower double precision, I ended up using NVT, as shown to be acceptable by Basconi and Shirts, with a long time constant.

My feeling is that we should recommend against using single precision. It's great that the calculation worked out for the reviewer, but the accumulation of round-off error inherent in single precision calculations will eventually cause major issues with the forces, especially on the time scales required for simulations that are long enough for "good" determination of diffusion constants. So I would recommend here full double precision or a double precision/fixed precision model, not single precision. Other thoughts?

[ramess101]

@drroe

Did they get an old copy of the PDF? I don't see it in the current repo.

Yes. We fixed this a few days after submitting.

[drroe]

Yes. We fixed this a few days after submitting.

Got it - I thought it sounded familiar, just couldn't remember when the fix was implemented.

[ramess101]

@drroe

My feeling is that we should recommend against using single precision. It's great that the calculation worked out for the reviewer, but the accumulation of round-off error inherent in single precision calculations will eventually cause major issues with the forces, especially on the time scales required for simulations that are long enough for "good" determination of diffusion constants. So I would recommend here full double precision or a double precision/fixed precision model, not single precision. Other thoughts?

I think that is a good recommendation to include in Section 4.3 "General Transport: Common pitfalls" where we discuss conserving energy. Probably something like (new text in bold):

"When simulating in the NVE ensemble, it is imperative that the integrator conserve energy. Performing simulations with single precision is one of the main causes for poor conservation of energy. For this reason, we recommend the use of full double precision or double/fixed precision."

[ramess101]

• Some questions were raised about rationales for some recommendations by reviewers. Please make sure all recommendations have specific rationales clearly listed.

We will need to read through this one more time to look for any recommendations that are missing a rationale.

• When possible, all references should have literature citations. One rationale discussed the GROMACS manual; ideally, a more direct primary citation should be used.

Caption to Figure 9 uses GROMACS manual to reference LINCS. I will include the actual LINCS articles ([49] and [50] in the GROMACS manual).

First paragraph of page 14 references the manual for the fit of the autocorrelation function. I will include the actual reference ([172] in the GROMACS manual).

Section 6.4 quotes the manual, references the manual for particle mesh Ewald, and references it for tail modifications. I think the quote should stay as it is. I will include the actual Ewald article ([14] in GROMACS manual) and some references for tail modifications.

• The check lists for computing self-diffusivity/shear viscosity with the Einstein and Green-Kubo equilibrium approaches are identical, with the exception of some varying section numbers. It would almost certainly be clearer to consolidate the checklists, one per property, with alternate options for Einsten and Green-Kubo approaches.

I will take a stab at this. But I personally like having them separated. I think it is less confusing because you can go straight to your checklist and ignore all sections that refer to the other method.

• It would be extremely helpful to put somewhat of a more quantitative requirement on what "middle" of the simulation is, and how to recognize it. Even if the subject is an open area of research, at least having quantitative measures is important for implementability.

I think we already address this pretty well. We say, "Currently, we are unaware of an objective approach for defining the "middle" region. Until such an approach exists, we recommend that the author reports how the region was selected..." However, I am somewhat confused myself as to why we say that we should use the middle region but then in Figure 2 we show the "long time fit." I assume Figure 2 just doesn't go to long enough times to observe the noise, but this is confusing regardless.

• Comments on the reviewer's need to use double precision for NVE vs. NVT (we can facilitate more information from the reviewer as necessary)

Addressed previously.

Several other recommendations in addition to what is in the reviews:

• It would be better to make recommendations in active voice "we recommend" versus passive voice "it is recommended" in order to clarify which recommendations are coming directly from the authors of this article, rather than the existing literature.

I think we have done this extensively. But I will read it again to see if we missed some... yes the only two instances where we say "it is recommended" were in the final two sentences of the conclusions. I have made this change.

• D.M.Z. lists funding acknowledgements, but is not an author.

Already removed.

• For the list of extra sources near the beginning, the authors and titles should be provided in the lists, not just in the references (full details can be left in the references).

OK, I will do this.

• Is it possible to include records (text + headers of emails, perhaps?) of the permission on use of figures from other publishers, and the conditions, in the GitHub repository? This could avoid later questions about what the permissions were if the article were further shared.

We do include the records for permissions in the "permissions" folder. I assume that is what is being asked.

[ramess101]

I think it's worth considering moving "common pitfalls" to the second major bullet point (before "Post-simulation data analysis"). It's likely you can take steps to avoid them before running production simulations of many replicates.

I think this makes sense. Does anyone object to moving common pitfalls up in the checklist?

I think it can be improved by another thorough read-through. There are some very long sentences and a few minor typos.

Again, we need to reread the manuscript before re-submission.

For novice readers, there is an opportunity to make a small tweak to the checklists to improve their understanding. On page 5, experienced practitioners will understand why the checklist for calculating viscosity with the Green-Kubo equilibrium approach appears before the Einstein equilibrium approach. However, a person new to simulations might not catch the significance, and if not careful, not read the explanation in Section 6.3. It seems like a short note next to the checklists for viscosity, alerting readers to the intention behind switching the order of appearance of the methods (and directing them to the explanation in Section 6) would be helpful for novice readers.

This is a valid point. If we end up combining the Green-Kubo and Einstein checklists it will be a moot point though.

[ramess101]

@drroe @ejmaginn @jrelliottoh @dcarls0n

As suggested by the reviewers, I have modified the checklists so that:

1) Common pitfalls is the second bullet point 2) Green-Kubo and Einstein checklists are combined for a given property.

In the end, I think combining the checklists is a good idea. It makes it clear what is the same and what is different for the two methods, whereas before the reader would have to mentally compare the two checklists. Please look over the checklist to make sure the distinction between Green-Kubo and Einstein is clear and that no information was lost by combining the checklists.

You will note that I had to reduce the font size so that the checklists would fit on a single page. The livecoms template was somewhat finicky in which fonts were changed with different latex commands. This was the best I could make it look.

[ramess101]

@drroe @ejmaginn @jrelliottoh @dcarls0n

So at this point we have gone through all of the easy/clear recommendations made by the reviewers. All that remains is for each of us to read through the document looking for:

1) Recommendations that do not have a clear rationale 2) Typos and long sentences 3) Passive voice (I removed all that I could find with a simple Ctrl+F search) 4) New errors that have been introduced by our edits

Also, I should mention that a included three sentences regarding a recent publication discussing finite size effects for viscosity (first paragraph of Section 6.1.3). Although the figures presented in Reference 45 (Kim et al.) would be useful to include in a future version of our manuscript, for the time being I thought it was better to just discuss their findings. Essentially, they demonstrate that finite size effects do exist but that they manifest themselves as fluctuations with respect to system size rather than a simple scaling behavior, i.e. they are not systematic like they are for self-diffusivity. Due to the oscillatory nature, it is still not feasible to correct for finite size effects, and so our recommendation did not change. If anyone objects to this insertion, please let me know.

[ramess101]

@drroe @dcarls0n @ejmaginn @jrelliottoh

I am working on drafting our preliminary responses to the reviewers. I will upload a .docx and .pdf file by the end of the day.

[drroe]

"When simulating in the NVE ensemble, it is imperative that the integrator conserve energy. Performing simulations with single precision is one of the main causes for poor conservation of energy. For this reason, we recommend the use of full double precision or double/fixed precision."

@ramess101 This sounds good. I think we should just add a few words on why single precision can lead to poor conservation of energy, something like (new text in bold):

"Performing simulations with single precision is one of the main causes for poor conservation of energy due to accumulation of round-off errors"

What do you think?

[ramess101]

@drroe @ejmaginn @jrelliottoh @dcarls0n

We still need to think of how to clarify this issue:

Reviewer: It would be extremely helpful to put somewhat of a more quantitative requirement on what "middle" of the simulation is, and how to recognize it. Even if the subject is an open area of research, at least having quantitative measures is important for implementability.

Me: I think we already address this pretty well. We say, "Currently, we are unaware of an objective approach for defining the "middle" region. Until such an approach exists, we recommend that the author reports how the region was selected..." However, I am somewhat confused myself as to why we say that we should use the middle region but then in Figure 2 we show the "long time fit." I assume Figure 2 just doesn't go to long enough times to observe the noise, but this is confusing regardless.

This is the relevant section:

I think we should clarify that the fit in Figure 2 is to the "middle region" even though it appears to be the "long time fit." The simulation is only 1 ns, so even the simulation really went to "long times" we would start to see the increased noise. The term "long-time asymptote" could be confusing because we are not fitting the long-time data, but we do assume that the middle region provides an estimate for the asymptotic behavior.

Any suggestions?

I was thinking something like the following:

A typical log-log plot, borrowed from Ref. [35], is provided in Figure 2, where the linear regression to the "middle" region is included. From visual inspection, the "ballistic" short time interval ranges from the beginning of the simulation to approximately 100 ps. The "middle" region is identified by the linear regime with a slope of 1 (for a log-log plot) spanning from approximately 100 ps to 1000 ps. Note that the noisy "long time" simulation data (beyond 1000 ps) are not depicted in Figure 2 and are excluded from the linear fit.

Caption: The gray dashed lines are linear fits to the "middle" region of the MSD, as determined by the authors.

Any objections or additions?

[ramess101]

@drroe @ejmaginn @jrelliottoh @dcarls0n

OK, I have uploaded "Response_LiveCoMS_Reviewers_1" with a draft of our responses. Note that the Response 1 to the editor still needs some recommendation rationales. It would also be nice to have some specific examples of typos and wordsmithing for Response 3 to Reviewer 2.

I plan on re-reading the manuscript this weekend. Can we all finalize our comments by next week?

[ramess101]

@drroe @ejmaginn @jrelliottoh @dcarls0n

I used latexdiff to generate a PDF with all the changes that have been made after we originally submitted the manuscript. This file can be found in the LiveCoMS_review_process\ folder as "diff_postreview_1.pdf". There are a few instances where the changes were too complicated for the automated difference tracker. But for the most part this file shows all the changes that have been made in the last few months. As requested by LiveCoMS, this file will be provided when we resubmit the manuscript.

[ramess101]

@drroe @ejmaginn @jrelliottoh @dcarls0n

Thank you everyone for your feedback. The revised manuscript has been re-submitted.