Functionality - Githubissues

brittanyblouin / ANCRTAdjust

An R package to adjust routine HIV testing data from antenatal care to reduce bias in estimating HIV prevalence trends

MIT License

2 stars 3 forks source link

Functionality #18

Open seabbs opened 4 years ago

seabbs commented 4 years ago

I need more evidence before I can say the package works as intended. This should be in the form of tests, documentation and an example pipeline (all mentioned in other issues).

Linked to #17 and #16

For this # openjournals/joss-reviews/issues/1740 review

m-maheu-giroux commented 4 years ago

We have extensively tested the functions on ANC-RT data from the President Emergency Funds for AIDS Relief in Africa's (PEPFAR) Monitoring, Evaluation, and Reporting (MER) database that contains information on more than contains information on more than 37 millions ANC attendees from 19,527 unique facilities from 17 countries in sub-Saharan Africa, totaling 226,541 observations over the 2015-2019 period. We also informally tested the package/functions on ANC-RT data from a few selected countries.

Unfortunately this data is not in the public domain. However, we summarized our findings in a 63 pages report that we can privately share with you. Ideally, we would be able to make this report public in the near future.

seabbs commented 4 years ago

JOSS has a policy that all confirmatory evidence needs to be in the public domain and be open access.

I can totally understand that there may be data access issues. Is synthetic data a viable alternative? Ultimately your users need to have evidence that the package works as intended.

seabbs commented 4 years ago

Looking again at the new README (which is much improved) has made the package functionality much clearer!

I just need a few more things:

Evidence the synthetic data your are using has the same problems you would see in raw data. This could just be details on its generation.
Fully automated testing with full coverage.
A discussion on why mean imputation has been used. Mean imputation is relatively crude in comparison to multiple imputation or other modelling approaches so I think some justification is needed.

Nice to have:

An evaluation of the reliability of your imputation approach. This could take the form of just dropping varying proportions of missing data from your testing set and seeing how well your imputation approach performs. This would also give you a good basis for improving it in the future.

ellessenne commented 4 years ago

I would suggest including a simulated dataset that mimics characteristics of the data you are not allowed to share, and demonstrating the pipeline using that.