[¬Rp] Stokes drag on conglomerates of spheres

khinsen commented 4 years ago

Original article: B. Cichocki, K. Hinsen, Stokes drag on conglomerates of spheres, Phys. Fluids 7:286 (1995)

PDF URL: https://github.com/khinsen/rescience-ten-year-challenge-paper-4/raw/master/article/article.pdf Metadata URL: https://github.com/khinsen/rescience-ten-year-challenge-paper-4/raw/master/article/metadata.yaml Code URL: https://github.com/khinsen/rescience-ten-year-challenge-paper-4/

Scientific domain: Fluid dynamics Programming language: Fortran 77 Suggested editor:

Since this is a failed reproduction attempt, no results are presented, meaning that editors and reviewers do not need to be familiar with the application domain .

rougier commented 4 years ago

@khinsen Oh, our very first negative result ! @otizonaizit Can you edit this submission ?

otizonaizit commented 4 years ago

Yes! I will edit this.

otizonaizit commented 4 years ago

Dear @FedericoV , would you be up to review this?

otizonaizit commented 4 years ago

Dear @eroesch , would you be up to review this?

eroesch commented 4 years ago

Yes, I will do it.

otizonaizit commented 4 years ago

Yes, I will do it.

Friendly ping @eroesch :-)

otizonaizit commented 4 years ago

Hi @rougier , @khinsen : is this a submission connected to the Ten Years Reproducibility Challenge? It seems hard to get reviewers...

khinsen commented 4 years ago

@otizonaizit Yes, it's a contribution to the challenge. And since it's a failed reproduction attempt, it doesn't require any domain expertise (in colloid science) to evaluate.

rougier commented 4 years ago

@ReScience/reviewers Any volunteer to review our first failed reproduction for the ten year reproducibility challenge? No domain expertise necessary since it failed...

pdebuyl commented 4 years ago

Well, I like the topic, I'll review.

rth commented 4 years ago

I can also review. I used to work on hydrodynamic codes some time ago..

sabinomaggi commented 4 years ago

If you need a third reviewer, I can join the list.

pdebuyl commented 4 years ago

Given the sudden popularity of the paper, I'll leave it to someone else :-)

otizonaizit commented 4 years ago

Great, so @rth I'd like you to review this. We only need one reviewer. But I can use @sabinomaggi for #32 : I don't think deep domain knowledge is needed there. I'll invte you over there right now. @pdebuyl : I would need you over to #37 !

rth commented 4 years ago

OK, will do a review in the next 7 days.

otizonaizit commented 4 years ago

great, thanks!

rth commented 4 years ago

A quick status update: I have read both papers and run the code, I'll try to finish the review this weekend.

rth commented 4 years ago

General comments

This paper describes a failed reproduction attempt for computing stokes drag on conglomerates of spheres with a Fortran code. The two takeaways for me were that:

that archival of the source code is not enough, as missing configuration files, which are often considered less important, can also prevent reproducibility.
that the Fortran code in question still works as expected 34 years later with mostly no changes, which cannot be said about more modern programming tools.

Overall I think this paper presents a nice prospective on code reproducibility and provides an interesting example of code life cycle for code used in research over 34 years.

Code

I was able to run the code following instructions in the code repository, and obtained identical results to those presented. Some technical questions were answered in https://github.com/khinsen/rescience-ten-year-challenge-paper-4/issues/1 The use of Guix for code reproducibility is very helpful.

A few minor comments:

I think that the use of org-mode notes can be confusing, as some of code comments (that are not rendered on Github) have an impact on environment variables. I does work in the end, bit it's just one more thing the person following these instructions needs to be aware of.
It might have been easier to create one Guix environment with all required dependencies, then run all shell commands there. As opposed to running guix environment multiple times with different dependencies and having to explicitly exit this environment after each step. At least that was the case for me, maybe I was doing something wrong due to my lack of familiarity with Guix. The use or not of org-mode may be the reason behind it as well.
Just a note that the source code for the Fortran code, currently on Github, can also be accessed via the Software Heritage archive
There is likely a typo on the sentence,

The column “final” corresponds to L=4 for the smaller conglomerates (N=55 or less) and to L=4 for the larger ones.

in summary.org file.

Paper

It's an interesting read and I don't have any major comments on the form or the contents.

One comment is that upon reading "reproduction failure" the first though is that the original paper had issues or that the code has portability limitations. It's not the case here (aside from the configuration files needed to reproduce not being published), and I think it might be helpful to mention it earlier in the article (e.g. in an abstract if there was one).

khinsen commented 4 years ago

Thanks @rth for this review!

Comments and actions taken:

I have rewritten the introduction to my code execution notes in order to make them clear even when viewed on GitHub (which strips away much of the Org-Mode markup). Let's hope that GitHub doesn't break this in an update to its rendered.
This is the first time I publish a computational protocol using Guix, so I am still experimenting with the best way to present everything. The reason for using a specific Guix environment for each individual command is the best possible documentation of the dependencies. For example, it's easy to see why sed is needed (for patching the installation script), and that the real computation does not depend on it. It is of course possible to use a single environment containing everything.
Leaving the environment explicitly is never required when working from Org-Mode, so it's not something I thought much about. I wouldn't do reproducible computations in an interactive shell anyway, because I would lose the output of the session. I have never used Guix in any other way than from Org-Mode, so I cannot say which other tools work well for this.
Thanks for the reminder about Software Heritage. Now that citing code has become easy with the recently published [biblatex-software](https://ctan.org/pkg/biblatex-software} package, I have updated my reference to the HYDROLIB library to point to SWH.
The typo in summary.org is fixed.
I have added a sentence about the nature of the failure to the first paragraph of the introduction.

otizonaizit commented 4 years ago

@khinsen : thank you for the revision! It looks good to me. @rth : do you have additional comments?

rth commented 4 years ago

Thanks for addressing comments @khinsen ! Looks good to me. I don't have any other comments @otizonaizit .

otizonaizit commented 4 years ago

Great, so @khinsen the paper is hereby accepted! I'll go about publishing it in the next two days. I'll generate a PR against your repo for updating the metadata before publication.

khinsen commented 4 years ago

@otizonaizit I have added the SWH ID and re-generated the PDF.

otizonaizit commented 4 years ago

Publication is done: https://zenodo.org/record/3889694/files/article.pdf Now waiting for the update to the website to make it official :-)

rougier commented 4 years ago

@otizonaizit Did you add the bibtex to the website bibfile?

otizonaizit commented 4 years ago

Yes, but there are open questions about it and we are waiting for your comments: https://github.com/ReScience/rescience.github.io/pull/81

rougier commented 4 years ago

Sorry, just answered.

khinsen commented 4 years ago

@otizonaizit Since this paper is now published, can we close this issue?

otizonaizit commented 4 years ago

Sure!

ReScience / submissions