search

barbagroup / cloud-repro

Reproducible workflow on a public cloud for computational fluid dynamics
BSD 3-Clause "New" or "Revised" License
9 stars 6 forks source link

Reviewer comments in "Accept" decision #3

Closed labarba closed 4 years ago

labarba commented 5 years ago

The editor's decision came as "accepted for publication in an upcoming issue of Computing in Science and Engineering, subject to a final light copyedit." Final manuscript files are due on Sep. 11.

Editor's Comments

With Accept and Accept with Minor Revisions, I'm moving to Accept. Please make sure to address as many comments as possible in the final version of your paper. I will work with the reviewers to get them to enter any updates to their public reviews.

Reviewers' Comments

Comments 1: I have read the author's response to my comments and the updated paper. The authors addressed every one of my comments, and, where they agreed with my critique, I found the corresponding changes clearly addressed in the final document. In my view, the article is ready to accept for publication.

Comments 2: The reviewer would like to thank the author's for revising their manuscript and for providing a detailed response letter.

Specific remarks:

"issues regarding deploying containerised applications within an HPC context" — Broad portability of our software and workflow is not our objective, and would (arguably) not be justified. Our software is currently only used in research projects within our lab: what reasons would call for the highly skilled and labor intensive software engineering required to deliver broadly portable software and workflow?

The reviewer is old enough to remember the exact same rational and phraseology being used to argue against: open sourcing software (only our lab would care, and it would take too much effort to maintain the project), using version control (it is only developed in our lab and is too complicated), unit tests and continuous integration (requires highly skilled software expertise to configure), and automated post-processing workflows (we'll only need to post process once so why bother scripting it up). History has not been kind to any of these viewpoints.

Moreover, in order for it to be meaningfully reproducible, there has to be some level of portability. Case in point consider a researcher in Iran (a country with many highly skilled CFD researchers - several of whom the reviewer considers to be colleagues) who wishes to reproduce your results. At the moment - irrespective of their financial resources - they can't on account of sanctions. To the reviewer this does not sit right and promotes the wrong attitude towards reproducible research. Does this mean that workflows should work on every system? No, of course not. But, one should aim for a reasonable broadly representative subset.

However, given the more specific objectives laid out in the introduction, the reviewer does not request any specific changes.

Note that performance is not our highest priority, reproducibility is.

Would it not, then, simplify things to use nodes with (say) 10G ethernet as opposed to Infiniband? This eliminates all of the issues around RDMA support and would make portability substantially simpler.

We contacted Microsoft Azure about that and they told us there is no publicly released information regarding the speed and channels.

This is unfortunate. Given this the next best option is to benchmark the memory bandwidth of the nodes (and the reference Colonial One system) using (say) the Stream Triad. This will provide a meaningful reference point for how much bandwidth is available.

labarba commented 5 years ago

Comment to consider addressing in the paper:

Changes