Problems to run the experiment

bukosabino commented 4 years ago

Hi @jriou,

Congratulations on your work.

I try to run the experiment, but I have several problems to do it.

Some questions:

Do I need data_S_model13.R, or I can generate it?
Do you have any step by step guide to reproduce the results?

Please, note I don't have any experience with the R language.

Best, Dario

jriou commented 4 years ago

Hi Dario,

data_S_model13.R is just a data file that gets created in the script.

I'm sorry I didn't find the time to produce a structured guide for now. Still I got feedback from several groups that were able to successfully use the code for other settings. If you don't have any experience with R it's going to be difficult though...

Best, Julien

bukosabino commented 4 years ago

Hi @jriou ,

We are working at Pi School to port the experiment to Python, so we are interested in matching the output results in Python and R. I think we are pretty close to achieving the goal.

I can run the file format_output/model13_italy/output_model13_italy.R to generate some output figure files succesfully. Also, I can run run_models/run_model13c_italy.R. But, I am not sure where is the .csv output files?

Best, Dario

jriou commented 4 years ago

Oh you're with the Pi School, I see!

run_models/run_model13c_italy.R is not meant to be run entirely. It's more of a control script that allows to format the data and generate the files that allows to run the model on a cluster, so to re-estimate the model on your own machine. Alternatively you can use the csv files that I already generated.

Is it clear enough?

bukosabino commented 4 years ago

Hi @jriou ,

Finally, I can run all the experiments in R and Python using rstan and pystan. The problem is the results don't match.

Some comments about this:

All the variables have the same values (even in consideration of this issue: https://github.com/jriou/covid_adjusted_cfr/issues/4).
I use the same OS and computer.
I set the same seed in the R and Python experiment (at the beginning of the scripts) and the same seed in sampling rstan/pystan functions.

Do you have any idea/clue about this? It looks like some people have the same problem here: https://discourse.mc-stan.org/t/different-outputs-in-rstan-vs-pystan/211

Clue: When I run big experiments (chain=4 and iter=1000), the results are pretty similar (but don't match), and when I run toy experiments (chain=1 and iter=5) the results are different.

Again, thank you for your work!

Best, Dario

jriou commented 4 years ago

Hi Dario,

Indeed there seem to be an issue using the same seed in rstan and pystan, so it's no surprise that the results don't match exactly.

It is perfectly normal that 5 iterations with different seeds don't match. If you get similar results with 4 chains and 1000 iterations that's all that matters to me.

Thanks for your interest again, it's pretty rare to have someone go behind you and check your code in my field! Best regards, Julien

jriou / covid_adjusted_cfr

Problems to run the experiment #3