explain what reproduce.sh is in the REMARK documentation

econ-ark / REMARK

Replications and Explorations Made using the ARK

Apache License 2.0

19 stars 56 forks source link

explain what reproduce.sh is in the REMARK documentation #101

Open sbenthall opened 3 years ago

sbenthall commented 3 years ago

The REMARK standard (this repository, README) refers to a reproduce.sh file but this reference is incomplete as there is more assumed in the design of reproduce.sh than is described.

Either what reproduce.sh should be described better -- how to create it, the conditions under which it is used -- or the REMARK standard should include something else besides it.

sbenthall commented 3 years ago

The relevant text of the current standard is:

A reproduce.sh script that
Installs the requirements
Runs and reproduces all the results

"Runs and reproduces all the results" is totally ambiguous. This either needs to be more firmly nailed down, or it needs to be resolved as part of an editorial process, see #105

sbenthall commented 3 years ago

CDC on call: the person has to define what the results are.

sbenthall commented 3 years ago

Use case: Suppose you want to build off the paper/code in 5 years. Your steps are:

Get the REMARK, and run the code in its specified environment to see how it works. In principle, you could continue to work in this environment. In practice, you would...
Try to get it working in a updated environment (new Python, new dependencies)

For step 2, you might want a test suite. Really reproduce.sh is about having a test suite.

sbenthall commented 3 years ago

So maybe reproduce.sh should include some asserts?

llorracc commented 3 years ago

It occurs to me that we could start on defining reproducible figures by using the existing makefig utility in utilities.py. At a minimum probably it could figure out how to save the fig file without ephemeral metadata like time stamps.