search

GraphicsReplicability / replicability.graphics

Code Replicability in Computer Graphics
https://replicability.graphics
BSD 3-Clause "New" or "Revised" License
197 stars 28 forks source link

About the termnology of Replicability & Reproducibility #86

Open VVingerfly opened 4 years ago

VVingerfly commented 4 years ago

HI, Thanks for the awesome work. It's really helpful and useful for CG researchers.

I am not an English speaker. And I am confused about the termnology of Replicability & Reproducibility.

I searched on the Internet, and found a definition from The American Statistical Association (ASA)

Reproducibility: A study is reproducible if you can take the original data and the computer code used to analyze the data and reproduce all of the numerical findings from the study. This may initially sound like a trivial task but experience has shown that it’s not always easy to achieve this seemingly minimal standard.

Replicability: This is the act of repeating an entire study, independently of the original investigator without the use of original data (but generally using the same methods).

But from the abstract of the paper

This process is called “replicability” when using the original authors’ artifacts (e.g., code), or “reproducibility” otherwise (e.g., re-implementing algorithms).

It seems that the two definitions are opposite? Could you help me to solve my confusion? Thanks a lot!

B.R.

dcoeurjo commented 4 years ago

Thanks @VVingerfly for the feedback.

Actually, setting the proper terminology was not easy (mostly because of the confusion between replicability and reproducibility you mention). We definitely agree that opposite usages of these terms may be found in other communities.

In the paper (and the website), we have followed the ACM definition (https://www.acm.org/publications/policies/artifact-review-badging) which was also used for the http://www.replicabilitystamp.org initiative for instance (replicability=using authors' code)

VVingerfly commented 4 years ago

Thanks @dcoeurjo a lot for the quick reply! And thanks for providing the source of the definitions, which is very useflul for understanding these terminologies.

nmellado commented 4 years ago

Let’s keep this issue open, as it might be a good read for other users.

dcoeurjo commented 4 years ago

Let's keep it open to remind us to clarify the def (with the ACM link) in the project description (and close it when it's done, what do you think @nmellado .

nmellado commented 4 years ago

Yes, that was my second thought. We will add here the link pointing to the page/paragraph commenting this aspect in the project description.