To do - Githubissues

@seabbs put together a to do list - adding a copy here for reference/easier access.

Direct copy of text but I have reorganised into chunks.

[x] Review current implementation and bug/sense check (i.e for things like should the US = sum(states))
[x] Add any additional automated checks that make sense to prevent PA like issues occurring (ideally identifying what the issue was in PA originally).
[x] Automate submission to the US hub (leaving manual work to be a graph check).

[ ] Potentially add a true NULL model into the mix. (see #86)
[ ] Extend ensembling grid to be by horizon and by state (and ideally both of those) and fix the stability issue that makes using findings hard.
[ ] Score the individual models, plus mean, median and a chosen QRA ensemble (see below) and the hub ensemble. Scoring should be both exhaustive (i.e @nikosbosse summary report) and summarised into a few figures for a paper.
[ ] Score ensemble grid and explore what combinations work/don't work. For models that work well (if any) highlight weightings of individual models and potentially explore over time. Again exhaustive + a few summary figures.

[ ] Introduction (background, detail, what are we trying to do (my take simple single epi concepts + ensembling vs building a big hard to interpret model).
[ ] Methods: (data, section per model, ensembling section, scoring section, evaluation).
[ ] Results: Individual model evaluation above, Ensemble evaluation above, deep dive into areas/times/types of bad performance.
[ ] Discussion: set up sections and bullet.

epiforecasts / covid-us-forecasts