mesnardo
commented
5 years ago Comment 1
- [x] Whilst the manuscript is well written the narrative is somewhat muddled. It starts with an overview of reproducible computational research, then pivots to how containers can be employed within the context of cloud computing, then jumping to benchmarks of one particular cloud providers' hardware, and then -- finally -- pivots a final time to a cost analysis of the public cloud for simulations. These are all interesting topics. Indeed, many a conference proceeding has been dedicated to each of these areas. However, by combining them into a single 10 page document this present manuscript fails to do justice to any of these areas. Further, the high degree of specificity will likely have a negative impact the relevance of the manuscript going forwards; a consequence of it being a review of the current state of affairs of a rapidly evolving field. It would therefore be better if it focused on just one or two areas such as "benchmarking cloud computing platforms" or "reproducibility through containers."
Reply
- The main goal of this manuscript is to report how to conduct CFD research in a reproducible way on a public cloud, laying out all the tools we used and the steps we undertook. We intend this paper to also serve as a mini-tutorial for researchers wanting to adopt these reproducible practices (we have observed when giving seminars or domain-conference talks that containers haven't disseminated to the CFD community). We use Docker containers to capture the computational environment and Microsoft Azure tools to capture the run-time environment. The degree of specificity is, in part, the point: precisely reproducing our work presumes deploying the same computational environment.
- Right now, researchers using scientific computing are still concerned about cost and performances on public clouds, and this is why we provide benchmark results and are report how much it costs to run our CFD application. Freniere et al. (2016) reported performance degradation on AWS (compared to their local cluster). But in the last three years, cloud providers have improved their offerings for HPC solutions. Thus, another objective is to update the record on the observations made by Freniere et al. (in the same journal) to the new cloud services introduced in the last two years.
- Although cloud computing is a rapidly evolving field, we believe it is important to report on the current state of public cloud offerings for academic applications.
- The reviewer is a top expert in HPC, and we understand from this point of view it seems the paper does not do justice to the various topics covered. But the intended audience of this paper is not, in fact, either of our reviewers (their expertise was needed, of course, to provide a quality review of the manuscript!). The audience is intended to be researchers using scientific computing (possibly on old and over-subscribed university clusters), who are curious about using cloud resources, and interested in reproducible practices, but need guidelines to adopt these.
- Neither benchmarking cloud computing, nor reproducibility with containers, were a central focus for our paper. We include some benchmark results to update the record with respect to the article by Freniere et al., appearing in the same journal. We use container technology for our reproducible workflow (as others have), but extend the reproducibility ideas to automatic configuration of cloud resources.
Changes
- We made several changes, responding to both reviewers, which make the aims of the paper more clear. In particular, the Abstract and Introduction prepare the reader better for the rest of the paper.
Citation
Portions of the review report quoted here by permissions granted on the original under a CC-BY license.