search

globaldothealth / outbreak-data

For users of G.h data: notes, timelines, sources, etc.
MIT License
0 stars 0 forks source link

How should we go about omitting errors or discarding entries in the timeline? #21

Closed JacqSauer closed 3 weeks ago

JacqSauer commented 4 months ago

The G.h team would like to keep a record of all data entries in the timeline and our line lists, this includes errors such as data that was later reconciled or changed by the original source. We are seeing an issue with the USDA site and data reconciliation (see tickets #20 and #15 ). For past outbreaks, the G.h team has implemented a status tab in our google sheets with an option to "discard" or "omit error" for a case. This would allow us to keep an internal record of the data entry but ensure that the public version of the data would only include active/confirmed cases.

@scarpino is there a way we can implement this strategy into the timeline? If we were to add this feature/strategy do you need additional engineering support from our team to ensure the weekly upload does not include the omitted or discarded entries? Is there another strategy you have in mind?

JacqSauer commented 4 months ago

the last two columns of the sheet should be used to track the status of each entry. There are three options for the status column: "Posted Publicly," indicating that a timeline entry was successfully posted to the public timeline on Think Global Health, "Not Posted," indicating that an entry was not posted on the public timeline, or "omitted" to indicate if an entry was or should be omitted from the public timeline to keep a record of it in our local timeline/google sheet. The next column to the right titled status_comment is a comment on the entry. This is where we can note data reconciliation or discrepancies between our local timeline copy and the public timeline. I am actively running through and adding comments about missing entries, data discrepancies, and omitted data entries.

@scarpino @aimeehan1 @julianasopko @jackie-powers

JacqSauer commented 3 weeks ago

@jackie-powers do you feel comfortable closing this issue?

jackie-powers commented 3 weeks ago

@JacqSauer Yes. The public timeline occasionally includes entries by ThinkGlobalHealth that are not on our timeline. Some of these are noted on our sheet, but some are not. I will do a better job noting these additions by ThinkGlobalHealth in the future.