crangelsmith commented 3 years ago

Story Review:

Story Name: Covid-19 vaccine forecasting story

Submitting Author: : Jack Roberts (@jack89roberts)

Pull Request: #138

Reviewers: @crangelsmith @billfinnegan

Editor: @samvanstroud

Reviewer instructions & questions

@crangelsmith @billfinnegan, please carry out your review in this issue by updating the checklist below in a separate comment, and writing new comments in case you have any questions. If you cannot edit the checklist or comment please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/alan-turing-institute/TuringDataStories/settings/access

Any questions, concerns or suggestions regarding the review process please let @crangelsmith, @DavidBeavan or @samvanstroud know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review Checklist

Code of conduct

[ ] I confirm that I read and will adhere to the Turing Data Stories code of conduct.

General checks

[ ] Notebook: Is the source code for this data story available as a notebook in the linked pull request?
[ ] Contribution and authorship: Are the authors clearly listed? Does the author list seem appropriate and complete?
[ ] Scope and eligibility: Does the submission contain an original and complete analysis of open data? Is the story aligned with the Turing Data Stories vision statement?

Reproducibility

[ ] Does the notebook run in a local environment?
[ ] Does the notebook build and run in binder?
[ ] Are all data sources openly accessible and properly cited with a link?
[ ] Are the data open, and do they have an explicit licence, provenance and attribution?

Pedagogy

[ ] Does the story demonstrate some specific data analysis or visualisation techniques?
[ ] Are these techniques well motivated?
[ ] Are these techniques well implemented?
[ ] Is the notebook well documented, using both markdown cells and comments in code cells?
[ ] Does the notebook has an introduction section motivating the story?
[ ] Does the notebook has a conclusion section discussing the main insight from the stories?
[ ] Is the paper well written (it does not require editing for structure, language, or writing quality)?

Context

[ ] Does the story give an insight into some societal issue?
[ ] Is the context around this issue well referenced (newspaper articles, scientific papers, etc.)?

Ethical

[ ] Is any linkage of datasets in the story unlikely to lead to an increased risk of the personal identification of individuals?
[ ] Is the Story truthful and clear about any limitations of the analysis (and potential biases in data)?
[ ] Is the Story unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

AOB

Other comments to focus from the PR #138:

☐ Length and level of detail ok? I feel like it got quite long for something that's actually pretty simple. E.g. is the amount of time describing the code ok - I was trying to make it as accessible as possible but might have gone over the top in places. ☐ Approach reasonable? In particular the forecast with the uncertainty band? It's maybe not the most robust approach but I was trying to avoid adding the complexity of using an "actual" forecasting algorithm. ☐ Is the explanation of how the forecasts decide whether doses will be given as first or second doses clear? And of calculating the delay between 1st and 2nd doses? ☐ Does the code make sense? ☐ Everything looks ok? ☐ Cut off date for the data is currently set to 4th May. This could be updated closer to the time of publishing, but it's also true that the forecasts become less interesting as time goes on and the vaccine programme gets closer to completion.

crangelsmith commented 3 years ago

Reviewers: @crangelsmith

Review Checklist

Code of conduct

[x] I confirm that I read and will adhere to the Turing Data Stories code of conduct.

General checks

[x] Notebook: Is the source code for this data story available as a notebook in the linked pull request?
[x] Contribution and authorship: Are the authors clearly listed? Does the author list seem appropriate and complete?
[x] Scope and eligibility: Does the submission contain an original and complete analysis of open data? Is the story aligned with the Turing Data Stories vision statement?

Reproducibility

[x] Does the notebook run in a local environment?
[x] Does the notebook build and run in binder?
[x] Are all data sources openly accessible and properly cited with a link?
[x] Are the data open, and do they have an explicit licence, provenance and attribution?

Pedagogy

[x] Does the story demonstrate some specific data analysis or visualisation techniques?
[x] Are these techniques well motivated?
[x] Are these techniques well implemented?
[x] Is the notebook well documented, using both markdown cells and comments in code cells?
[x] Does the notebook has an introduction section motivating the story?
[x] Does the notebook has a conclusion section discussing the main insight from the stories?
[x] Is the paper well written (it does not require editing for structure, language, or writing quality)?

Context

[x] Does the story give an insight into some societal issue?
[x] Is the context around this issue well referenced (newspaper articles, scientific papers, etc.)?

Ethical

[x] Is any linkage of datasets in the story unlikely to lead to an increased risk of the personal identification of individuals?
[x] Is the Story truthful and clear about any limitations of the analysis (and potential biases in data)?
[x] Is the Story unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

crangelsmith commented 3 years ago

Hi @jack89roberts, i'm starting my review of the story now. I'll be submitting a number of comments in the PR soon.

For this story we want to give the reader two reading levels, a high level of the story and a technical one, we will distinguish these with the color of the background on the text, so this means that we need to edit sentences/paragraphs in the cases where the two narratives are in the same paragraphs. Many of my comments on the PR are about this, and the easiest implementation is just to break the paragraph in two. We will take care of the colours at the publishing stage.

crangelsmith commented 3 years ago

☐ Cut off date for the data is currently set to 4th May. This could be updated closer to the time of publishing, but it's also true that the forecasts become less interesting as time goes on and the vaccine programme gets closer to completion.

We are fastracking the review to publish it very soon, but something we could do is to fix the data to 4th of May, but to add to the last summary plot the real data values of number of doses for the day of publication. Something like this (looks ugly because i added the points by hand in pages) :

This could be interesting and would bring the discussion of which model is working best. What do you think @jack89roberts ?

jack89roberts commented 3 years ago

Hi @crangelsmith , thanks for the comments so far! I'll get to them ASAP. I really liked your idea of doing a follow-up story to evaluate the forecasts once the (first phase of the) vaccination programme is complete, and the suggestion above could act as a teaser for that future story. I'll play with it and see how it looks.

billfinnegan commented 3 years ago

Review Checklist

Code of conduct

[x] I confirm that I read and will adhere to the Turing Data Stories code of conduct.

General checks

[x] Notebook: Is the source code for this data story available as a notebook in the linked pull request?
[x] Contribution and authorship: Are the authors clearly listed? Does the author list seem appropriate and complete?
[x] Scope and eligibility: Does the submission contain an original and complete analysis of open data? Is the story aligned with the Turing Data Stories vision statement?

Reproducibility

[x] Does the notebook run in a local environment?
[x] Does the notebook build and run in binder?
[x] Are all data sources openly accessible and properly cited with a link?
[x] Are the data open, and do they have an explicit licence, provenance and attribution?

Pedagogy

[x] Does the story demonstrate some specific data analysis or visualisation techniques?
[x] Are these techniques well motivated?
[x] Are these techniques well implemented?
[x] Is the notebook well documented, using both markdown cells and comments in code cells?
[x] Does the notebook has an introduction section motivating the story?
[x] Does the notebook has a conclusion section discussing the main insight from the stories?
[x] Is the paper well written (it does not require editing for structure, language, or writing quality)?

Context

[x] Does the story give an insight into some societal issue?
[x] Is the context around this issue well referenced (newspaper articles, scientific papers, etc.)?

Ethical

[x] Is any linkage of datasets in the story unlikely to lead to an increased risk of the personal identification of individuals?
[x] Is the Story truthful and clear about any limitations of the analysis (and potential biases in data)?
[x] Is the Story unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

AOB Other comments to focus from the PR #138:

[x] Length and level of detail ok? I feel like it got quite long for something that's actually pretty simple. E.g. is the amount of time describing the code ok - I was trying to make it as accessible as possible but might have gone over the top in places.
[x] Approach reasonable? In particular the forecast with the uncertainty band? It's maybe not the most robust approach but I was trying to avoid adding the complexity of using an "actual" forecasting algorithm.
[x] Is the explanation of how the forecasts decide whether doses will be given as first or second doses clear? And of calculating the delay between 1st and 2nd doses?
[x] Does the code make sense?
[x] Everything looks ok?
[x] Cut off date for the data is currently set to 4th May. This could be updated closer to the time of publishing, but it's also true that the forecasts become less interesting as time goes on and the vaccine programme gets closer to completion.

samvanstroud commented 3 years ago

Great work with the story and reviews all! We can colour cells differently using the example here: https://kiranvaidhya.com/fastpages/jupyter/2020/02/20/test.html

alan-turing-institute / TuringDataStories

[Turing Data Story][Review] Covid-19 vaccine forecasting story #140

Story Review:

Reviewer instructions & questions

Review Checklist

Code of conduct

General checks

Reproducibility

Pedagogy

Context

Ethical

AOB

Review Checklist

Code of conduct

General checks

Reproducibility

Pedagogy

Context

Ethical