Round 2, Editor's comments

Received: Sept. 21, 2021

Your manuscript "High-productivity, high-performance workflow for virus-scale electrostatic simulations with Bempp-Exafmm" was seen by the original 4 referees, and as I mentioned, 2 extra referees, whose comments are appended below. These new referees are experts in Poisson-Boltzmann and numerical analysis.

The new referees (particularly Referee #6) raised important points that perhaps will help better clarify the main goal of the paper.

The main goal of the paper is to provide a usable and reusable workflow for virus-scale electrostatic simulations; I think the goal of the paper is clear, but as Referee #6 mentions, it's not entirely clear what the workflow features are and how this even compares with other PB solvers. We would recommend having a subsection under Results section (perhaps the first subsection of this section) clearly explaining all of these features, and how this basically makes your workflow different than standard PB solvers. It is important to make it very clear (in this subsection and in the paper in general) that you are not trying to propose a new PB solver, but rather, an interactive, reusable workflow for ease prototyping, etc.
You can move main methodological information to the Methods section, and in the Result section, just provide a brief overview of the methods.
While you are not proposing a new PB solver, it is important to demonstrate what are the main benefits of using your resource rather than a standard PB solver. Therefore, we think comparisons in terms of performance/convergence against other PB solvers are fair as requested by Reviewer #3 -- since a resource must also show that it has a comparable performance. However, we also agree with Reviewer #6 that it should not be the center of attention. Our suggestion is then to still present the comparison, but perhaps discuss these results in more detail in a Supplementary Information, and summarize the discussion in the main text, in the Results section. We would also recommend comparing with at least one more PB solver as well, if possible.
Please also address the other comments made by Referees #5 and #6, and provide a reply to the latest review by Referee #3.

Reviewer `#1`

The authors have done an adequate job addressing my concerns and the additions in response to the other reviewers' comments are also adequate. It still seems like the paper falls in the limbo between a computational biophysics and scientific computing audience. This is more of a concern for the editor than this reviewer: I am unsure whether to focus my comments on a computational biophysics reader's concerns (MM-PBSA should not be used to compute solvation energies of viruses) or scientific computing reader's concerns (more detailed analysis of scaling, convergence, etc.).

In summary, this paper is publishable as-is although I remain confused about the audience.

Reviewer `#2`

The authors have addressed my concerns.

Reviewer `#4`

My comments have been addressed satisfactorily.

Reviewers 3, 5, and 6 on separate issue threads.

We would recommend having a subsection under Results section (perhaps the first subsection of this section) clearly explaining all of these features, and how this basically makes your workflow different than standard PB solvers

In the first sub-section of the Results section, which describes a study of the conditioning of two formulations, we have added a description fo the Bempp features that are unique to it, complemented with three code samples (listings) to illustrate. We also added a description of the typical workflow for solving PB equations with BEM, highlighting what sets Bempp-Exafmm apart. The new content is about one column worth of text.

comparisons in terms of performance/convergence against other PB solvers are fair as requested by Reviewer #3 -- since a resource must also show that it has a comparable performance. However, we also agree with Reviewer #6 that it should not be the center of attention.

We have a new sub-section under Results reporting performance of our solver and APBS using a molecule of medium size. For small molecules, runtimes are in the order of seconds, regardless of the solver used. For the molecule we chose (1RCX) runtimes are between 100 and 10,000 seconds (depending on accuracy), and the user will notice the differences in this problem scale. For larger molecules, ABPS crashed. Please see the response to Reviewer #6, where we explain why comparing performance of two solvers non-trivial: it is necessary to find matching accuracies or the comparison is not "fair."

The new results are in Tables 4 and 5, and Figure 4. As shown in the Figure, at any accuracy better than about 3% error in the solvation energy, Bempp-Exafmm is faster. Moreover, Bempp+Exafmm scales better (runtime increases more slowly) with grid size, which allows solving larger problems at reasonable time.

We made the following changes in response to Reviewer #6's Point 4:

Section “Comparison with trusted community software”[…] stay focused on the main point ...

moved the results with APBS and our software on many molecules to appear as Supplementary Information, adding the requested information of grid spacing in each direction
removed the results with MIBPB because iterative grid refinement does not show monotonically changing solvation energy (in other words, convergence towards a value). These results do not add anything to our main point.

Please see the response to Reviewer #6 on this point. Particularly, we tried our best to compare our results with MIBPB because Reviewer #3 insisted. But the computed solvation energies are not converging to a value as you refine the mesh, which is not acceptable. However, we assert that it is not our job to troubleshoot a third-party software that is not behaving as advertised.

Importantly, we wish to add, MIBPB is not open source software. Our research practice is fully committed to open source, so we would not use nor recommend a black-box closed-source code. It is impossible to inspect it and attempt to make any improvements.

Email reply to EiC on Oct. 15, 2021

Dear Fernando,

Before we embark on the second revision, we would like to reach some agreements with you regarding the path to follow. My co-authors and I have discussed this with great care.

First of all, it looks like you have a recalcitrant Reviewer 3. We see no constructive path to agreement with this reviewer, and believe you will have to take executive decision to override him or her. The reviewer’s stance is clear by the fact that in this round, they gave no acknowledgement of our responses accompanying the first revision. The reviewer instead comes up with new comments, not mentioned in their original review report. I don’t know what your journal policy is on this, but it seems improper to ignore the authors’ responses and come up with new criticism on the second round.

Furthermore, the new comments of Reviewer 3 are in several places incorrect, which I hope is clear from the attached responses that we prepared as justification for our request to override him/her.

Reviewers 1, 2, and 4 are satisfied. We will respond to Reviewers 5 and 6 and are confident they will be satisfied. The reason for bringing two additional reviewers was to settle the differences with Reviewer 3, and it is clear on these new reviewers’ comments that they side with us.

Please give me your initial reaction to this communication, and in the next email I will detail our plan for the second revision, to address the new round of reviews.

I would also be glad to discuss by phone/web call. Let me know.

Kind regards, Lorena Round 2, Reviewer 3 comments and responses.pdf

From the EiC, Oct. 19, 2022

Thanks for getting in touch about your revision. The reason we enlisted two new reviewers (#5 and #6) was to indeed help us make a more informed decision about the paper. In the original pool of reviewers, Reviewer #3 was the only one with a direct expertise in Poisson-Boltzmann, and given that none of the editors in our team has this specific expertise, we then decided to get two other experts to review the paper (and the reviews, by the way).

We can definitely override a reviewer’s stance on a paper (and we’ve done that before), but we felt that we could not do that during the first revision, given that we needed at least a second opinion on the paper with a similar expertise as Reviewer #3. Now we have enough expertise onboard, and more information as well, and rest assured that we will do what is needed to reach a fair decision.

I apologize again for the time taken (looking for new reviewers definitely delayed the peer review process of your paper), but I hope you understand that it is of our best interest to reach a fair, well-informed decision.

Hope this clarifies our process, but please let me know if you still have questions and concerns. I’m certainly looking forward to receiving your second revision.

Revision submitted Dec. 7, 2022

From the EiC, Dec. 17, 2022

I'm looking at your revision right now to send it back to our reviewers. I had a few questions and feedback regarding your response to referees' document:

In page 9, you use the word “nonsensical”; however, please keep in mind that we avoid using language that may be construed as judgmental or demeaning. If you can review the document to replace these words and any similar words that I may have missed, that would be great.
Under Responses for Reviewer 5 (page 12), you started by saying that “the Reviewer suggested to use a better analytical solution…” but where is this comment? I got confused, since I don’t know to which comment from Reviewer 5 you are referring to. Did I miss anything?
On page 15, you started by addressing point #2 from Reviewer 6, but I don’t know if the response to point #1 got cut off? I do see that you responded to point #1 at the end of this page, but because of the inverted order (first point #2, then point #1), I just wanted to make sure that you did not accidentally remove anything.
Overall, would you be able to merge the “Comments” with the “Responses” section for each reviewer? I just found it a bit confusing to have to sometimes go back to the “Comments” section to better understand the responses. Instead of copying the entire review first (“Comments”), and then copying some parts of that review again to respond to them (“Responses”), I would recommend to simply having the entire review and then adding the responses right below each paragraph or a set of paragraphs. This would also make it easier for reviewers to understand the flow of the comments/responses. Does this make sense?

Let me know if you have any questions! Once you send me a new version, I’ll send it back to referees right away.

barbagroup / bempp_exafmm_paper