search

social-science-data-editors / guidance

Guidance by Data Editors
https://social-science-data-editors.github.io/guidance/
Other
13 stars 11 forks source link

Add guidance and requirement/suggestion for creating feasible computation when full computation is long #30

Open larsvilhuber opened 3 years ago

larsvilhuber commented 3 years ago

Some computations, as presented in the paper, may run for a very long time.

Authors should be encouraged (required?) to provide instructions on how to run "feasible" computations (instead of 1,000 bootstraps, run only 10; instead of full sample, run with a 1% sample) and how it might impact the output.

This is different from creating synthetic data.

joanllull commented 3 years ago

This is actually an interesting point.

We do this indeed (only encountered two examples so far, but I expect to encounter more when I take over at Econometrics Journal). In one of the examples, we required a smaller number of simulations, in the other we only checked the solution code but not the estimation code (it was using pre-specified routines and synthetic data, otherwise, we would have required a small function to test that minimization actually works).

When this happen, in the paper we then certify the following in the paper: "Given the highly demanding nature of the algorithms, the replication checks were run on a simplified version of the code, which is also available at [...]"