Verification of all experiment values/initial conditions/parameters/hyperparameters/etc.

[jlperla]

• [x] YOu don't need to run it, but can follow https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/scripts/README.md to see how the full replication would be done. Everything up to the "TODO BELOW" is complete, so just the figure generation/etc. at this point. If you see typos then feel free to fix them.
• [x] #155 can check what I actually implemented there?
• [x] #157
• [x] #158
• [x] Verify all of the *.txt parameter values in https://github.com/HighDimensionalEconLab/HMCExamples.jl/tree/main/src are correct for the pseudotrue. e.g. the observation noise is consistent
• [x] Verify that all of the generate_simulated_data_* in https://github.com/HighDimensionalEconLab/HMCExamples.jl/tree/main/deps are consistent
• [x] Verify all of the baseline runs in https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/scripts/run_samplers/baseline_experiments.sh have the right number of samples/etc. chains/etc. and use the appropriate initial conditions for the experiments
• [x] Same for the _frequentist.sh and _robustness.sh in https://github.com/HighDimensionalEconLab/HMCExamples.jl/tree/main/scripts/run_samplers
• [x] Verify all of the dynare baseline samplers called form https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/scripts/run_dynare_samplers/baseline_experiments.sh have consisetnt initial conditions/pseudotrue/etc.
• [x] Verify the sgu data generation (in https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/scripts/run_dynare_samplers/generate_sgu_data.m) has the same pseudotrue and observation noise. This does not share the same data as the SGU in julia, otherwise everything is shared.
• [x] Verify the robustness stuff in the RBC section in https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/scripts/run_dynare_samplers/rbc_1_robustness.m along with the dynare files (e.g. https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/scripts/run_dynare_samplers/rbc_1_robustness.mod ) are all consisetnt.
• [x] I would just do a general code review of the sampling/etc. code in https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/src/ etc. Most of it is the same boilersplate so the key is to look for things like the x0 being consietnt/etc.

[donskerclass]

Currently all Dynare/matlab code seems not to use fixed seeds for random number generation.

Dynare command for this is set_dynare_seed(INTEGER) and would need to be added to options list in all .m files when invoking dynare (not the Matlab seed).

Should we add it to ensure reproducibility? This requires rerunning all Dynare code, so if we do, defer until all other issues resolved.

[jlperla]

I don't mind rerunning things if you know how to do this. Up to you if you want to make the modificaitons and then just tell me what to rerun. On the other hand, given that the dynare stuff isn't especially crucial it may not be worth it. But game for whatever you wish.

[donskerclass]

No need to rerun things just for the seed issue, but if I find other issues (I will start looking again later today) that need new runs, we can set the seed along with whatever other change might come up.

[donskerclass]

Re the generate_simulated_data_* files, in all of the RBC ones we set σ=0.01, while the paper reports σ=0.1.

Checking the data generation files run as of when the paper was last updated, we already had changed to σ=0.01, so this is a problem with the paper reporting, not a code error. I suggest we leave the data generation files as is and adjust the paper to match.

All other parameters are as reported and fine for the RBC model and the SGU model (and are acceptable for new models like rbc_sv).

[jlperla]

Sounds great! So at this point no need to rerun anything (except potentially the dynare deeds?)

[donskerclass]

Checking https://github.com/HighDimensionalEconLab/HMCExamples.jl/blob/main/src/sgu_1_joint_defaults.txt and sgu_2_joint_defaults.txt, all values are fine, but current version in paper reports acceptance rate as 0.65, while executed runs use target_acceptance_rate=0.9. I can confirm that the code we ran as of the paper release already used this setting.

Again not a problem, nothing to rerun, just something we need to update in the paper.

To be systematic about this, we should put it all in a constants file in the paper repo.

We do this for some but not all numerical values: currently the paper is with Jesús or else I would update it in the overleaf when it gets back, but I will put a little ticky box here to remind myself.

• [ ] Update paper reported acceptance rates in variables file
• [ ] Update paper reported noise variances to σ=0.01 in variables file
• [ ] Harmonize any other paper references to settings parameters by adding to variables file.

[jlperla]

From slack on the comment for constants: We setup the \mapvar ib https://github.com/HighDimensionalEconLab/hmc_dssm_paper/blob/main/docs/hmc_dssm.tex#L40-L41 Which points to https://github.com/HighDimensionalEconLab/hmc_dssm_paper/blob/main/tables/tex_constants.csv Though it doesn't look like many of these were put in as constants. So this is the place to go to consolidate things. Seems like we might as well add those in if we are updating this, but up to you.

[donskerclass]

Yes, that is the constants file to which I was referring. I want to hold off on updating it until we get the revisions from Jesús, because we will want to map most of the numbers in the paper outside of tables back to it.