Review 1, P. O'Leary

[labarba]

Citation:

Patrick O'Leary. Review: "Reproducible Workflow on a Public Cloud for Computational Fluid Dynamics". Authorea. July 26, 2019. DOI: https://doi.org/10.22541/au.156415458.85375575

Portions of the review report quoted here by permissions granted on the original under a CC-BY license.

[labarba]

Comment 1

• [x] Neither the introduction or the paper highlight the thesis or novel contributions. The reader is left to decide for themselves what aspect they believe are important. This needs to be improved. I found the following list to be reasonable possibilities: 1) Use of cloud computing as an alternative to onsite high-performance computing (HPC). 2) Improvement in public cloud offerings in last 2 to 4 years. 3) A proposed reproducible workflow. 4) The author's experiences.

---

Reply

• The main thesis of the paper is the idea of extending the set of open research objects created and shared for reproducibility (what we call "reproducibility packages"), to the automatic configuration of the computing environment on a cloud service. We explain our contribution in text added at the end of paragraph 3 of the Introduction. We also added a new paragraph at the end of the Introduction to emphasize the point that our paper updates the record with respect to the paper by Freniere and co-authors.
• Because many researchers in "traditional" computational fields (compared to data science) are still highly skeptical of using cloud for their work, we feel it's necessary to provide evidence that it's feasible today to complete CFD studies with cloud computing. Specifically, we need to show that cloud offerings today are practical for parallel computing with GPUs. We added a few sentences to the Abstract and to the last paragraph of the Introduction to emphasize this point.
• Another principal concern of researchers considering cloud resources is the cost. We can't address the issue in this paper in every detail. But we do want to communicate the cost of running the experiments we showcase, and share our ideas about how this concern may be addressed in the future (cost should come down, universities should tackle how to charge indirect costs, and maybe funding agencies could make bulk contracts with providers).

Changes

• Improve abstract and introduction to highlight present contributions (8d998af).

[labarba]

Comment 2

• [x] the code snippets breakup the prose with generic, easy to find information such as how to instantiate or run a docker image

---

Reply

• We intend this paper to also serve as a mini-tutorial for researchers wanting to adopt more reproducible practices. (This is mentioned in the beginning of Section 2.) We understand that for this reviewer (an expert in cloud computing, and HPC) these things are obvious. We find among our colleagues and at CFD conferences that containers are still an esoteric technology for most applied computational researchers.

Changes

• No changes needed.

[labarba]

Comment 3

• [x] Section One - Defines why reproducibility is essential for science and describes the related work but fails to differentiate their work from past contributions. The reader is left to speculate that it is merely an update to M. Schwab et al., C. Boettiger, C. Freniere et al. works.

---

Reply

• We are citing all of those works in the Introduction. With the new language introduced addressing comment 1, it should be clear now how we extend the work of the previous authors: we propose capturing the cloud environment programmatically and to share with readers the configuration files; we update the record on the observations made by Freniere et al. to the new cloud services introduced in the last two years.

Changes

• Improve abstract and introduction to highlight present contributions (8d998af).

[labarba]

Comment 4

Section Two - Walks the reader through the authors' reproducibility workflow outlined in figure 1.

Notes:

• [x] The delete jobs and pool step is missing in figure 1, but presented (including a listing) in the text.
• [x] Do we need to see the content of listing 1 - 4? Why not add a script to the repository and point to it like the YAML config files in examples/snake2d2k35/config_shipyard?
• [x] The docker build fails on the COPY ../ssh_config stage. Moreover, do the authors want someone to push their image to the authors' docker hub repository? Besides, the repository in footnote 1 contains a typo which highlights the far more significant issues in reproducibility of well-written error-free publications. This paper is a well-written paper, but errors happen.
• [x] The authors generalize that "same container ... can now run at scale." What about the MPI drivers? Does this effect where it can run and what hardware it can use?

---

Reply

• We added the step "Delete jobs and pool" to the workflow in Fig. 1.
• Do we need the listings? — From the reviewer's point of view, these may seem unnecessary, because the reviewer is already an expert in using containers and cloud resources. But we are aiming our paper for readers who are not experts, may have heard of containers as a new and interesting technology, but have never used them. These readers will see from these listings that adopting containers to make their computations more reproducible is not very complicated or time consuming. We want to include these listings as part of the narrative to encourage more researchers to adopt these tools.
• We updated the Dockerfiles to avoid the error message raised during the COPY instruction. We also fixed the typo in the URL of the GitHub repository.
• We do not expect the readers to build the Docker images locally. Indeed, we store the images on DockerHub, so that they can immediately use them. We provide the command-line instructions we used to build the images (as this is part of the workflow). If the readers want to re-build/modify the image locally, we do not expect them to push it to our DockerHub repository. We modified the last paragraph of sub-section 2.1 to reflect our intentions.
• The Docker images were specifically built (on our local workstation) to be able to run MPI applications on RDMA-capable VMs (such as the NC24r instance with a CentOS-based 7.3 HPC image) using multiple-instance tasks on Azure Batch service. The RDMA network on Azure supports MPI traffic for applications running with Intel MPI 5.x (or a later version). (In other words, to take advantage of RDMA on Linux compute nodes, we must use Intel MPI on the nodes.) We added a sentence over the end of paragraph 4 of the section "Cost Analysis and User Experience" mentioning the Docker images was specifically crafted to be used with Azure Batch service. Thus, we also removed the sentence "same container ... run at scale."

Changes

• Fix typo in URL of footnote (aac96cb).
• Change directory of the build context so that Docker build does not fail (67634d9).
• Add step "Delete pool/jobs" to workflow diagram in Fig. 1 (d612512).
• Remove sentence "same container ... run at scale" (758e485).
• Mention Docker images were specifically built to run with Azure Batch service (c908f53).
• Change "Anyone" to "The reader interested..." (a84844f).

[labarba]

Comment 5

• [x] At its' core, this paper is describing the use of cloud computing as an alternative to onsite high-performance computing (HPC). At 'scale' is a loaded term in HPC. This paper examines performance improvements in public cloud offerings and compares small runs in the cloud versus a six-year-old onsite HPC system. However, an MPI performance test up to 8 nodes, a mini-application performance tests up to 8 nodes, and a computational fluid dynamics domain science examination of 2 nodes does not adequately explore scaling.

---

Reply

• The language "at scale" indeed can evoke a "leadership computing" scenario (large DOE machines, etc.), and the reviewer is right to protest its use. We rephrased, because we're in the "long tail" setting here, of researchers running in parallel, but moderate size simulations. The runs are still production size for our application, with mesh sizes in the order of 50 million cells, allowing for three-dimensional fluid dynamics studies in the regimes of interest for our research problem.

Changes

• Remove sentence "same container ... run at scale" (758e485).

[labarba]

Comment 6

Section Three - Provides the results in the form of an MPI benchmark and a Poisson mini-application performance analysis, and analysis associated with a standard set of computational science experiments involving flow around a flying snake.

Notes:

• [x] Both the cloud and the six-year old HPC system use FDR Infiniband. Is Azure superior performance a result of tuning, configuration, or age? The analysis provided by this section is that Azure interconnect is improving. Cloud offerings moving from 10 Gbps Ethernet to FDR Infiniband in the past four years makes this result visible without testing. Did we need the OSU benchmarks for verification?
• [x] For the Poisson mini-application, the results again verify Azure has newer hardware than the authors onsite HPC system. However, what was the cause of the variance? Job/Node placement?
• [x] The detailed description of the flying snake workflow is appreciated.

---

Reply

• Is Azure superior performance a result of tuning, configuration, or age? —We didn't apply any manual tuning: both machines are running the same code, but of course the compiler versions are different, and the hardware is different. We are not emphasizing in this paper that performance is "superior" on Azure: just that it is a viable alternative to on-premises HPC clusters (because many researchers continue to be skeptical of cloud). We noticed that the reviewer twice mentions that we're using a "six-year old HPC system." Even if this is an old machine, it is what we have at our university and it is probably the situation for the majority of university researchers that they don't have much choice in regards to the local machines they run on. We certainly are not attempting to do an "apples to oranges" comparison of performance. The point is to dispel the misconception that cloud is (still) inadequate.
• Do we need the OSU benchmark results? We want to showcase that public cloud offerings have improved in the last few years. Our benchmark results show comparable networking performances between our university-managed HPC cluster and Microsoft Azure. We believe it is important to highlight these results as it is still common wisdom that public clouds are inadequate for computational research using HPC. Moreover, Freniere et al. (2016; Computing in Science & Engineering, 18(5), 68) used the same benchmark to compare their local cluster with AWS EC2 and reported performance degradation in networking. With these results, we are updating the readers of CiSE on the improvements of a public cloud such as Microsoft Azure.
• We do not know the cause the variance in runtime for the Poisson benchmark. However, the benchmark was executed 5 times within a single job task (and thus, on the same physical node), so the variation is not due to Job/Node placement. We have updated the caption of Fig. 3 (Poisson benchmark), detailing the number of MPI processes and the number of GPU devices used per node.

Changes

• Add information about the number of MPI processes and the number of GPU devices used per node for the Poisson benchmark (98fccec).

[labarba]

Comment 7

Section Four - Explores the cost analysis for a typical set of computational science experiments and insights on the user experience.

Notes:

• [x] For the user experience, how much did free computing and augmented support from Microsoft influence the authors' determination to utilize this environment. If it cost 20K, would the authors have completed the project?
• [x] Pricing changes every month, every week, maybe every day. Will any of this analysis be relevant by publication?
• [x] Table 5 will be outdated by the time this article is published. It does not add value.
• [x] How does this cost compare to buying a 2 - 8 node equivalent cluster?
• [x] Singularity containers appear out of the blue in this section with no description.
• [x] The reference for Table 5 in the text is just (5). Is this a cite, a figure, or a listing.

---

Reply

• We would not have attempted this work, were it not for the sponsored cloud credits. We were indeed curious about reproducibility benefits of cloud computing, and that's why we wrote the proposal to Azure Research. But without the sponsorship, we would not have had research budget for this work. In our opinion, this is one of the reasons to publish this paper: other researchers curious about using cloud need this kind of information before they will jump in. With regards to support, we used the standard channels (support tickets and GitHub issues), and it was not "augmented" support. We edited the acknowledgement section, which vaguely used the verb "help," to avoid the impression that we received extra support.
• Price will of course change. We feel it needs to go down. The relevancy of publishing the "snapshot" of today's pricing is precisely as back up that prices should drop. However, the cost of the NC24r instance we used in this study has been 3.96 USD per compute hour for the last two years and it's unlikely that it will change by a lot in the next 6 months.
• Table 5 will be outdated, yes, but it is today's situation, under which we operated. It's also the case that the patterns will continue (i.e., the ratio between the different pricing options).
• Cost compared to buying a cluster? We don't really know. We tried to get some helpful data from our IT department, but were denied the information.
• We added a short description about the Singularity container technology at the end of paragraph 3 of the section "Cost Analysis and User Experience."
• We fixed the reference to Table 5 in the manuscript.

Changes

• Properly reference Table 5 of the manuscript (db21f03).
• Add description about Singularity container technology (df1e0d5).
• Edit acknowledgements (9772f42).