mesnardo
commented
8 years ago Addressing reviewer comment # 2:
There should be some discussion about other efforts to compare different codes on the same problem.
In the late 90's, a project initiated at the Institute for Theoretical Physics of the University of California, Santa Barbara addressed the reliability of current methods (at that time) used in cosmological gas dynamics to assess the reproducibility of numerical studies (Frenk et al., 1999). The group of researchers compared the results from twelve different codes of the formation of a cluster in cold dark matter universe. They achieved a successful full-replication of the study: "Although there is, of course, no guarantee that any of the calculations gives the correct solution to the problem, the agreement among the various simulations was better than a pessimist might have predicted."
In the field of compressible flows, Dimonte et al. (2004) performed a comparative study of the turbulent Rayleigh-Taylor instability using seven different codes to justify the use of a computationally-intensive approach (interface reconstruction) but believed to reduce the overestimation a certain quantity. "Finally, if anyone is interested in repeating these calculations with their own numerical methods, please contact dimonte@lanl.gov or any of the other authors for an electronic form of the initial conditions."
In the field of astrophysical fluids, de Val-Borro et al. (2006) tested the reliability of current astrophysical hydrodynamic codes using a disc-planet interaction problem. They compared the solution from various grid-based and smooth particle hydrodynamics codes (a total of 17 codes was used). The good agreement between the different codes permitted to provide a reliable reference for future calculations. "In closing, we would like to reiterate that computational work might be regarded as an experiment, rather than a simulation. We have shown that different codes can give slightly different results for the same physical problem. Reproducibility of experimental results is fundamental to the scientific process, and this standard must be applied to those performed with computers. Before a computational result can be regarded as reliable, it must be confirmed by an independent test with a different code."
References:
- Frenk, C. S., White, S. D. M., Bode, P., Bond, J. R., Bryan, G. L., Cen, R., ... & Khokhlov, A. M. (1999). The Santa Barbara cluster comparison project: a comparison of cosmological hydrodynamics solutions. The Astrophysical Journal, 525(2), 554.
- Dimonte, G., Youngs, D. L., Dimits, A., Weber, S., Marinak, M., Wunsch, S., ... & Calder, A. C. (2004). A comparative study of the turbulent Rayleigh–Taylor instability using high-resolution three-dimensional numerical simulations: the Alpha-Group collaboration. Physics of Fluids (1994-present), 16(5), 1668-1693.
- De Val-Borro, M., Edgar, R. G., Artymowicz, P., Ciecielag, P., Cresswell, P., D'Angelo, G., ... & Klahr, H. (2006). A comparative study of disc–planet interaction. Monthly Notices of the Royal Astronomical Society, 370(2), 529-558.
[UPDATES]
labarba
Recommendation: Accept If Certain Minor Revisions Are Made
Comments: ... below is my review of the paper "Reproducible and replicable CFD: It's harder than you think". This is a nice paper, and will be a good contribution to the literature. I have the following remarks for revising the manuscript:
[x] 2. There should be some discussion about other efforts to compare different codes on the same problem. Some of the ones that come to mind are:
Although many of these are compressible (but not all, e.g. for the alpha group), they share the level of differences and in-depth knowledge of the code needed as reported in this paper.
Additional Questions:
1) Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical. : This provides a nice case study on how challenging reproducible research is to investigators, and will be common to all the readers.
2) Is the manuscript technically sound? Please explain your answer in the Detailed Comments section. : Yes
3) What do you see as this manuscript's contribution to the literature in this field?: This is a broad description of the problems one encounters with reproducibility.
4) What do you see as the strongest aspect of this manuscript?: The lengthy personal experience that the authors convey.
5) What do you see as the weakest aspect of this manuscript?: It would benefit from discussion on other attempts on reproducibility in CFD reported in the field.
Please rate the manuscript. Explain your choice in the Detailed Comments section.: Excellent