Base.r that uses cmdstan with cmdstanr + optimized model

ImperialCollegeLondon / covid19model

Code for modelling estimated deaths and cases for COVID19.

MIT License

944 stars 271 forks source link

Base.r that uses cmdstan with cmdstanr + optimized model #74

Closed rok-cesnovar closed 4 years ago

rok-cesnovar commented 4 years ago

This adds another version of base that uses cmdstan (via cmdstanr). The change compared to the use of rstan is minimal (the output is an rstan fit object).

It also adds the model @wds15 optimized.

At the top of the script is the installation instruction.

Install required packages:

install.packages("devtools")
devtools::install_github("jgabry/posterior")
devtools::install_github("stan-dev/cmdstanr")

And cmdstan 2.23 release candidate:

`cmdstanr::install_cmdstan(release_url = "https://github.com/stan-dev/cmdstan/releases/download/2.23-candidate/cmdstan-2.23-rc1.tar.gz", cores = 4)

rok-cesnovar commented 4 years ago

@harrisonzhu508 how long does the FULL mode usually take? With this PR it ran for 12,5 minutes for 4 chains (8 threads per chain) on a c5 AWS instance.

s-mishra commented 4 years ago

Hi @rok-cesnovar thanks a lot for this. I will check this and merge surely. Give me sometime till Monday as busy with something. Anyways this is awesome as non-parallel version took 5 hrs or so.

rok-cesnovar commented 4 years ago

Sure, no rush. All credit here goes to Sebastian (@wds15), I merely switched the Stan interface :)

s-mishra commented 4 years ago

A quick question @rok-cesnovar is installing cmdstanr and rstan allowed on the same installation? I remember using pystan and cmdstanpy in the same environment is a pain or not a possible thing!

rok-cesnovar commented 4 years ago

Yes, both work just fine together. In this case you actually need both as I have rewritten this script so that the output is still a rstanfit object. cmdstanr has its own fit object, but as you already have the post-processing done for the rstanfit object it made more sense.

rok-cesnovar commented 4 years ago

I now also added the first revision of the parallel model (base_rs2.stan) which is more or less the base model parallelized. base_rs3.stan is a bit more optimized.

s-mishra commented 4 years ago

Hi @rok-cesnovar , there is one problem with stuff in here: The new output from stanfit doesn't contain variables like Rt, Rt_adj, prediction, E_deaths. As they are now generated in an outside function country_log_lik. Is there a way to get them.

rok-cesnovar commented 4 years ago

Hey, I am going to defer to @wds15 for that, as he was the one that optimized the model.

wds15 commented 4 years ago

I updated the model once more with these changes:

corrected model (there was an off by one error; thanks to @MansMeg for nailing that one)
modularised the model => now the generated quantities block also uses the optimised code with dot products. This makes the model again somewhat faster (but please have look as E_death[1] is now slightly different, but to me this looks like only a numerical difference, which should not matter)
all former model outputs are now part of the output (E_deaths, predictions, Rt, Rt_adj)
the model is setup to be compatible with RStan and does not use parallelisation as offered by Cmdstan 2.23 - but it can be easily switched as shown in the comments.

I hope that helps.

rok-cesnovar commented 4 years ago

I will close this PR and @MansMeg can add the option to use cmdstan in his PR. I think that will be the easiest way to proceed.