barbagroup / bempp_exafmm_paper

Manuscript repository for our research paper, including reproducibility packages for all results, and latex source files.
6 stars 2 forks source link

Reviewer 1 comments #9

Open labarba opened 3 years ago

labarba commented 3 years ago

This is an interesting article that describes a well-engineered, robust/reproducible, and very accessible (Jupyter notebooks) Poisson-Boltzmann solver. The authors have done an excellent job describing the rationale for the software, making it straightforward to install via Conda, and ensuring that the results of the paper are reproducible. Overall, the paper is sound and should be of interest to the journal audience. However, there are several issues that detract from the overall accessibility/impact of the article:

labarba commented 3 years ago

item 1

The article focuses on a use case of questionable value: the solvation energies of very large macromolecules. While this quantity makes sense as a metric for evaluating accuracy/convergence with respect to mesh discretization, the notion of a "[polar] solvation energy" for a virus is not particularly meaningful. The authors should clarify why this quantity is being used and provide readers with a discussion of how the code is meant to be used for physically meaningful quantities of interest.

The showcase calculation in the paper is meant to illustrate the capacity of our software. We focus on a virus for the case study because of the scale. While structure-based computations, like solvation energy, are more meaningful for small rigid molecules, in the case of large molecules, ensemble computations are valuable: many conformations can be run to obtain an energy landscape. With our workflow running from Jupyter notebook, a researcher may run sets of experiments where short dynamics simulations are coupled with solvation energy calculations, quantifying such variations. This type of approach is widely used, e.g., MMPBSA [1], even for very large molecular structures [2,3]. Another setting where the computational workflow would be useful is computing binding energies of large molecules [4], as long as the binding process does not involve large conformational changes. These are well-known limitations of the PB model. Regardless, the electrostatic (polar) free energy is a useful quantity to analyze virion-scale systems. For example, it has been used to study the structure [5] and stability of the capsid with varying pH [6].

We have added a discussion on the use of our code as part of a computational workflow, and the relevance in large-scale problems, such as viruses, with the corresponding references.

[1] Wang, Changhao, et al. "Recent developments and applications of the MMPBSA method." Frontiers in molecular biosciences 4 (2018): 87. [2] Karim, Hana Atiqah Abdul, et al. "Molecular dynamics simulation on designed antibodies of HIV-1 capsid protein (p24)." 3rd International Conference on Computation for Science and Technology (ICCST-3). Atlantis Press, 2015. [3] Li, Chuan, et al. "Highly efficient and exact method for parallelization of grid‐based algorithms and its implementation in DelPhi." Journal of computational chemistry 33.24 (2012): 1960-1966. [4] Wang, Ercheng, et al. "End-point binding free energy calculation with MM/PBSA and MM/GBSA: strategies and applications in drug design." Chemical reviews 119.16 (2019): 9478-9508. [5] Šiber, Antonio, Anže Lošdorfer Božič, and Rudolf Podgornik. "Energies and pressures in viruses: contribution of nonspecific electrostatic interactions." Physical chemistry chemical physics 14.11 (2012): 3746-3765. [6] Roshal, D., et al. "pH-induced morphological changes of proteinaceous viral shells." Scientific reports 9.1 (2019): 1-9.

labarba commented 3 years ago

item 2

It is unclear who the target audience is for either the paper or the software. The paper spends a significant amount of space discussing issues such as interior/exterior versions of the derivative formulation and eigen-spectra of the operators. This could be interesting to developers of BEM solvers but is unlikely to be meaningful to many consumers of biomolecular electrostatics software.

The discussion of different versions of the BEM formulation is meant to showcase how easy it is to investigate such a question with a platform that gives the researcher access to interactive computing (via, e.g., Jupyter notebooks), while offering highly performant computation. At the same time, the question is of interest to consumers of bimolecular electrostatics software because this allows them to assess the performance of different packages, and make an informed choice.

Choosing to submit to Nature Comp. Sci., we aim to reach the interdisciplinary researchers that both apply and develop research software. Our workflow shows that the paradigm of combining high researcher productivity with high performance has the potential to enable more discovery with less effort. Our target audience is people like ourselves, who work at the interface of software development and research application (in collaborations), rather than pure domain scientists.

The purpose of presenting the details of the interior/exterior versions and the issue of Eigen-spectra is to show the usefulness of our software and the power of interactive computing with a high-performance backend.

labarba commented 3 years ago

item 3

The discussion of mass matrix preconditioning was difficult to follow. In particular, it was unclear whether inversion of the mass matrix was a necessary step for solution of the problem or a way to improve conditioning. Therefore, it was unclear whether approximation of the matrix led to any issues with accuracy of the solution.

Preconditioning is necessary to solve this system with iterative solvers. Mass matrix inversion is a very efficient preconditioned for this type of problem (used widely in finite element methods, for example). The preconditioned is the inverse of the mass matrix, but since inverting is expensive, we use mass lumping as an approximation. This does not impact accuracy of the solution, because the approximation is applied on the preconditioner. We will clarify the paper.

Fixed in b58a0ba and 342efc6966.

labarba commented 3 years ago

item 4

Was Nanoshaper used for all meshes in the paper? This was unclear and is important for generalizing the results of the paper to other molecules.

Nanoshaper was used for all biomolecules (the sphere was meshed with bempp itself). Nanoshaper can mesh a wide spectrum of molecules, and we have used it successfully for meshes up to 100 million triangles. In the case of the virus mesh, we also use a short Python script that calls Trimesh to clean the mesh from Nanoshaper, for example to remove cavities. (We are sharing the final meshes for the purpose of reproducibility.)

Fixed in 621a547.

labarba commented 3 years ago

item 5

/usr/bin/time -v is not a standard option for all flavors of Linux; the authors should clarify.

We are using GNU time, which is indeed not available in all distributions. We will clarify in the paper.

labarba commented 3 years ago

item 6

The "Laplace" and "Helmholtz" FMM variants were introduced without sufficient description. Do these refer to the Poisson and linearized Poisson-Boltzmann formulations, respectively? More generally, the paper does a poor job of describing which version of the Poisson-Boltzmann equation it is solving (this isn't introduced until Section 4).

The format of Nature Comput. Sci. has the methods section at the end. We can add a sentence to clarify the terminology the first time that they appear in the Results section, and explain in the captions where needed: "Laplace" FMM kernel does indeed apply to the Poisson equations, while "Helmholtz" FMM applies to the linearized PB equation. We can also replace the term "Helmholtz" by "Yukawa," to be more precise (in the figure legends).

labarba commented 3 years ago

item 7

There appears to be a typographical error on page 8: "Richard extrapolation" instead of "Richardson extrapolation".

Typo. Will fix.