search

mrc-ide / covid-sim

This is the COVID-19 CovidSim microsimulation model developed by the MRC Centre for Global Infectious Disease Analysis hosted at Imperial College, London.
GNU General Public License v3.0
1.23k stars 256 forks source link

Explicitly state the purpose and the goals of the modeling in the readme #246

Closed phelps-sg closed 4 years ago

phelps-sg commented 4 years ago

The readme.md describes what is modeled and how, but not why. It is important to understand the purpose of the modeling, and how the model is used, as the usefulness of many features will be contingent on this. For example, models which are used for inference, e.g. understanding whether increasing a particular parameter may result in an increase in deaths, are very different from models which are used for making quantitative forecasts (e.g. trying to estimate the actual number of deaths that may arise from a particular intervention). Depending on the goals of the model, some feature requests such as #206, may be irrelevant (there is less need to validate ouf-of-sample if the model is not being used for forecasting).

I am not authoritative on the purpose of the modeling, and so I have not issued a PR. However, a possible suggestion for the readme would be something along lines of:

This is a simulation model of the Covid-19 pandemic, focusing on the UK. It is used to inform health-policy by making quantitative forecasts of death-rates, and how these will vary depending on which interventions, such as social-distancing, are enacted.

dhogaza commented 4 years ago

The "why" is explained in the title of the very first report that's linked:

"Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand"

Why NPIs? Because there were no pharmaceutical interventions known at the time of writing. It also makes clear a difference between the "why" between their COVID-19 work and the first paper that describes the flu modeling they did 13 years earlier - that paper explains that it explored the impacts of NPIs + prophylactic dosing with Tamiflu (very much a pharmaceutical intervention).

And it seems clear the model is used both for inferential and quantitative purposes.

I'm just a lowly software engineer, but even I know that the linked papers (there are four) are relevant to understanding the model's purpose. Did you read any of them?

phelps-sg commented 4 years ago

Sorry, yes I did read the papers earlier, but didn't get the takeaway. My comment referred explicitly to the readme itself. Perhaps it would still be a good idea to explicitly mention the purpose of the model in the readme or model overview?

insidedctm commented 4 years ago

@dhogaza Since the model has been used to support a number of different events I still think it would still be useful to give an overview of what the model is intended to do. Not least because this evolves over time. I can see from the code and recent updates that support for modelling digital contact tracing is being added although I don't believe it has been used in any published work.

insidedctm commented 4 years ago

It might also be useful to reference this model that has been used for some of the MRC reports covid19model. Under what circumstances would covid-sim be used v covid19model? At least one reason might be that covid19model probably needs more actual deaths to estimate trends.

dhogaza commented 4 years ago

"Under what circumstances would covid-sim be used v covid19model?"

Wouldn't this be best addressed at the MRC site?

phelps-sg commented 4 years ago

@doghaza, I think the goal "[assess the] impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand" could be made more precise.

I think it would be useful to explicitly state how the model assesses impact, and how it is used to reduce mortality. Clearly the model cannot by itself reduce mortality, since it is only model, so I think it is useful to state something like "quantitatively assess the potential impact of interventions on future mortality rate in order to inform policy-makers".

Although it may be clear to you that the model is used quantitatively, I think it is very important to explicitly state that the model is being used to make forecasts, and to do so prominently in the repo. This is because it highlights an urgent and ongoing need to assess the accuracy of those forecasts, as per #206.

dhogaza commented 4 years ago

I think it is reasonable for the README.md document to assume that the readership is competent to understand the implications of the first cited paper, and the others, for that matter.

I've spent quite a bit of time over the past couple of weeks doing a bunch of reading and studying the relevant basics. I think it's reasonable for others to do the same rather than insist that THIS model's documentation serve as a primer.

phelps-sg commented 4 years ago

@dhogaza, I too have also spent time over the past couple of weeks reading the papers, as evidenced by my recent pull request #239. Nevertheless:

  1. as @insidedctm points out, the purpose of the model is evolving.
  2. the statement of the purpose of the model as described previously can be improved on. The papers cannot easily be edited to improve them, but the readme and model-overview.md can.
  3. What is the actual harm in briefly and explicitly stating the purpose of the model in the repo documentation- how will this detract from the quality of the repo? How would it make things less clear?
  4. If your objection is that it would take some work to update the readme, then I would be happy to personally submit the PR and take on the labor.
dhogaza commented 4 years ago

I'm not objecting to the notion of improving the readme, just pointing out that the information is there, even if you have to dig for it.

Your suggested statement looks OK with me, if you drop a few hyphens, but I'm not an expert, either.

I also appreciate your comments on some of the other issue threads, BTW.

phelps-sg commented 4 years ago

tx

I have modified the title of the issue to reflect your comments, and submitted PR #250.