Continuous evaluation to prove or disprove the efficacy of CWA in reducing infection rates

OlympianRevolution commented 2 years ago

Feature description

Conduct a continuous evaluation of the CWA in reducing population wide infection rates using a randomized blind study based on postal codes and suppressing or replacing a proportion of warnings

The experimental setup is as follows:

For a weekly timespan starting on random days random PLZ are selected.
In these PLZs a proportion (say varying between 10%-90%) of true red/green warnings are instead not shown to the user.
Optional (see comment below): In place of real warnings not displayed to the user another random user in the same PLZ receives the warning, such that the overall number of displayed warnings is the same.
The incidences as well as the warning suppression/randomization rate are tracked per PLZ
Over time it should become very clear whether differences in R(t) between similar PLZs are correlated to the suppression/randomization rate by PLZ of the CWA warnings.

In order to maintain trust in the CWA at most 5%-30% of total germanywide warnings should be suppressed or receive random warnings instead of someone else's true warning.

This would be just as ethical or not as other blind medical trials.

Conducting this study would likely require all users to manually enter their PLZ. I doubt many users would have a problem with that if we explain why. Those that refuse could be enrolled in a nationwide weekly or daily experiment. (Although it would be more difficult to exclude confounding variables such as weather in this case)

Ideally some of those conducting the analysis would not even be told the randomization rate, thus becoming a double blind study.

Problem and motivation

There is still very little evidence to show that bluetooth based contact tracing is effective at reducing transmission significantly. Previous analysis (such as the oxford study https://www.nature.com/articles/s41586-021-03606-z) are hampered by sample bias and confounding variable effects. We should aim for the gold standard in medical treatment analysis, the randomized blind study to get rid of these confounding variables and prove or disprove the current efficacy of the CWA.

Some CWA analysis has been able to show that some EDUS users were surprised by warnings and that that those that were warned test positive more frequently than random testing. Still the numbers are not proof that the CWA is effective at reducing population wide infection rates.

Is this something you're interested in working on

Yes but I am powerless to make such a decision although I am willing to give feedback.

I have little hope this will be implemented but wanted to document this experimental setup for the next time someone able to take such decisions is interested in actually knowing the efficacy of bluetooth based contact tracing efficacy.

Internal Tracking ID: EXPOSUREAPP-11476

larswmh commented 2 years ago

Thanks for your suggestion @OlympianRevolution. We have created an internal ticket for it and will raise this topic internally. Internal Tracking ID: EXPOSUREAPP-11476

Corona-Warn-App Open Source Team

OlympianRevolution commented 2 years ago

Although I initially proposed to replace suppressed warnings with random ones, I now think that it would be much simpler to implement and only marginally less thorough to not do so.

Reasons to replace suppressed warnings with random ones:

the same number of warnings is received per PLZ so that any difference in R(t) can be shown to originate from warning the right people and people not just being generally more careful because they were warned

Reasons to only suppress

much easier to implement, PLZ would never have to leave phone, and all promises of privacy and on device computation could be kept. The app could simply download the current suppression probability per PLZ
Much more difficult to implement replacement since you need to communicate suppressed warnings to other users who entered the same PLZ
We would measure the difference between using the app and not using the app not between using the app and warning random people.
Even if we warn random people we would be changing the distribution from those actually in contact with many others to a uniform distribution.
Many users receiving random warning may be able to identify them as such since they had few contacts.

corona-warn-app / cwa-wishlist