search

ecohealthalliance / wahis

Animal outbreak data scraped from the WAHIS database
Other
6 stars 0 forks source link

Capture missing data in six-month-scraper #28

Open emmamendelsohn opened 3 years ago

emmamendelsohn commented 3 years ago

There are two differences in data handling between the “old” wahis 6-month scraper (pre-API) and the new scraper (with API).

(1) In the new dataset, we have inadvertently omitted taxa where diseases are “present” but case count is NA. In the previous HTML reports (pre-API) it was easier to see which taxa fell into this category.

(2) In the new dataset, we are sometimes missing the taxa for diseases marked “absent”. These were often labeled as “multiple species” and thus filtered out of the dataset.

emmamendelsohn commented 3 years ago

Ok I believe both these issues are fixed.

"multiple species" will still show up for present diseases in addition to all the taxa, but all quantities are 0 for multiple species. I didn't filter it out because sometimes "multiple species" is all that is reported for a disease, see example rabies here: https://wahis.oie.int/#/report-smr/view?reportId=22030&period=SEM01&areaId=2&isAquatic=false Regardless, "multiple species" is filtered out in our repel data processing.