RKI: Results of data donation

[MikeMcC399]

What is missing

The results of data donation are missing on https://www.rki.de/cwa.html.

At least some interim results would be interesting to see. How many users are participating, for instance.

Why should it be included

CWA Version 1.13 introduced voluntary data donation (see https://www.coronawarn.app/en/blog/2021-03-04-corona-warn-app-version-1-13/) with the release on March 4, 2021.

The RKI site Infektionsketten digital unterbrechen mit der Corona-Warn-App says "Das Robert Koch-Institut wird für die Öffentlichkeit aufbereitetet Ergebnisse auf seiner Themenseite zur Corona-Warn-App unter https://www.rki.de/cwa.html veröffentlichen."

Where should it be included

The results of data donation should be published on https://www.rki.de/cwa.html, possibly also on the CWA blog page https://www.coronawarn.app/en/blog/ or in the reports about the app usage Übersicht zu aktuellen und früherer Zahlen und Fakten zur Corona-Warn-App.

---

Internal Tracking ID: EXPOSUREAPP-6456

[dsarkar]

@MikeMcC399 Thanks. Internal Tracking ID: EXPOSUREAPP-6456

[MikeMcC399]

@dsarkar Has there been any feedback from RKI about the results of the data collection in the app? The data collection has been running for more than 2 months now.

[Ein-Tim]

@MikeMcC399

https://www.zeit.de/digital/2021-05/corona-warn-app-pandemiebekaempfung-kontaktverfolgung-covid-19, says: „Das Institut wird in den kommenden Wochen die vollständige Auswertung veröffentlichen”

But I think this refers to the survey on high risk encounters only.

[MikeMcC399]

@Ein-Tim Many thanks for posting the link to the Zeit article. I think you are right that the article is about the survey triggered by a high risk encounter, not about the general data donation.

I wish we could get an answer from RKI. It seems that much information is passed only to the press and then a dedicated member of the community, like yourself, posts the information here.

[heinezen]

@MikeMcC399 @Ein-Tim

I've raised the issue again, this time by also mentioning the existing news articles. Let's hope the RKI responds swiftly.

---

Corona-Warn-App Open Source Team

[MikeMcC399]

Thanks to @Ein-Tim who posted the link in https://github.com/corona-warn-app/cwa-wishlist/issues/540 to a presentation from WeHealth in YouTube Does it really work? Collecting data on the German Corona-Warn-App Duration: 01:30:23.

Justus Benzler, RKI (@BenzlerJ), Tim Weihrauch, RKI and Göran Kirchner, RKI (@kirchnergo) present.

---

From transcript of https://www.youtube.com/watch?v=AE9ljDGNAus :

"Since the German Corona-Warn-App launched in June 2020, only limited data was available to evaluate its effectiveness in stopping transmission chains, due both to the decentralized design of the system and its separation from public health authorities. In addition to external surveys, an opt-in data donation for privacy-preserving analytics (PPA) and an event-driven user survey (EDUS) were integrated in 2021.

Since March, app users can opt in to automatic, anonymous, daily submission of status data to a backend server for analysis. This data describes app-calculated risk levels and changes to these. In addition, incidence data sets are submitted (1) for newly identified exposure episodes (exposure windows), (2) when lab test results are obtained in the app, and (3) when users who tested positive submit random identifiers (diagnosis keys) to warn their contacts.

This operational data is complemented by a user survey, triggered when the app notifies of a high-risk exposure. A linked online survey focuses on the user’s perception of this notification, prior behavior, and further intentions. At the end of this survey, respondents can agree to a follow-up survey, which inquires about actual behavior changes and particularly about PCR or rapid antigen tests.

Both instruments, PPA and EDUS, include basic demographic data and have some technical precautions against misuse, e.g. a device check and, in the case of EDUS, a one-time token, which was checked and invalidated when entering the online survey.

PPA-participation has reached 7 million daily submissions, while the initial EDUS has been concluded after two months and 26,000 respondents (14,000 also participating in the follow-up survey). Having proved feasibility and acceptance, EDUS may be used for other evaluations."

---

According to the above, EDUS was terminated in May 2021. (See video at https://youtu.be/AE9ljDGNAus?t=2447).

[ndegendogo]

Thanks for the link!

(still I don't understand why RKI didn't publish this link on their website...)

[MikeMcC399]

@ndegendogo

(still I don't understand why RKI didn't publish this link on their website...)

RKI has not updated the section Fragen zur Evaluation der Corona-Warn-App on https://www.rki.de/cwa.html since 12.3.2021. RKI would presumably know why there has been no update there, but unfortunately we are not getting any feedback from them here in GitHub.

[rugk]

Sad. So as they do not seem to do it voluntary there is no other way, so here we go: https://fragdenstaat.de/a/221876

[ndegendogo]

@rugk very sad, I agree.

I am donating my data. But now I feel unhappy and disrespected, and usually would consider to stop my donation. In this case I don't; still in the hope and expectation that the risk assessment formula can better be fine-tuned and improved with more donations; and that it's worth ...

[Ein-Tim]

FYI: https://www.heise.de/news/RKI-Schaetzung-Warn-App-hat-mehr-als-100-000-Infektionsketten-unterbrochen-6070747.html (h/t to @fynngodau who shared this in our Community Slack Channel).

[MikeMcC399]

There is a new unpublished branch https://github.com/corona-warn-app/cwa-website/tree/science-blog in the website repo, which looks like it may deliver detailed results of data analysis.

It is hard to understand why there is no RKI feedback into this thread though. We often do not know what will happen until it actually happens.

[Ein-Tim]

@MikeMcC399

The branch looks like there will be a whole new page, only for scientistic analysis of the Corona-Warn-App. Will be interesting to see when they will publish this.

[GisoSchroederSAP]

@All, we are trying to hook up a new part of the coronawarn.app web page, that provides detailed information about the performance and efficacy of the app - we focus on transparency here. Those information come from different sources and are consolidated as part of the evaluation report by RKI.

The new section of the webpage will be launched soon and will be updated from time to time with additional pieces of information, so everybody can start an own research project.

Please, be informed: Like other regular reports provided by RKI, the new section is intended to act as an information portal only. Even located in a GitHub branch, it is not meant to work on the report or on the conclusions/statements provided in the report.

[Ein-Tim]

https://github.com/corona-warn-app/cwa-website/pull/1373 has been merged in to master just now! @MikeMcC399

[Ein-Tim]

There we go: https://www.coronawarn.app/de/science/2021-06-15-science-blog-1/ (only available in German currently)

I think there aren't any results of the data donation yet, but I'm sure they'll come soon (see the last paragraph).

[MikeMcC399]

@Ein-Tim

I think there aren't any results of the data donation yet, but I'm sure they'll come soon (see the last paragraph).

At least some of the results of the data donation are used in the evaluation.

RKI now needs to update the page https://www.rki.de/cwa.html where they wrote that they would publish the results. I will leave this wishlist item open until they have done that. Not everybody will be polling https://www.coronawarn.app/ for changes and otherwise only people who subscribe to GitHub notifications including https://github.com/corona-warn-app/cwa-website will get automatically informed. Perhaps though the reactivated Twitter channel might get used to announce the new Science section?

[MikeMcC399]

"Über die Wirksamkeit und den Nutzen der Corona-Warn-App" https://www.coronawarn.app/de/science/2021-06-15-science-blog-1/ has been announced on Twitter

https://twitter.com/coronawarnapp/status/1405095776691404801?s=20

[MikeMcC399]

@dsarkar Since the new section https://www.coronawarn.app/de/science/ has been created to publish scientific research results including from the data donation, we can probably close this issue. The first results have been published and more results are promised.

The only reason to maybe keep this issue open would be if RKI uses the GitHub issue to track the necessary update on https://www.rki.de/cwa.html to announce the results. If RKI and GitHub are not linked, then it would not help and we could just close this GitHub issue (#451) now.

[jucktnich]

Maybe at a link to the blogposts in the ppa settings

[Ein-Tim]

@jucktnich Good idea! Could you open a new issue for this, please!

[jucktnich]

FYI: #556

[MikeMcC399]

I accidently closed this issue, although I had intended to leave it open for feedback.

In any case today, Jun 18, 2021, https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/WarnApp/Warn_App.html has been updated with the information:

"Detailauswertungen veröffentlicht das Robert Koch-Institut regelmäßig im Blogformat auf https://www.coronawarn.app/de/science/."

so the request to publish results via https://www.rki.de/cwa.html has been fulfilled and the issue can be closed as resolved.

[dsarkar]

Thanks everybody !

[rugk]

FYI the RKI also published the CSVs behind the diagrams used in the blog articles in a FOI request now: https://fragdenstaat.de/anfrage/ergebnisse-der-cwa-datenspende-corona-warn-app/#nachricht-617401

They also explain how long it will take until the other data is to be released, which “should be made available via a GitHub repository in the coming weeks, probably by the end of August [and September}” for most of the data.

[Ein-Tim]

@rugk Sadly, they did not publish anything in the meantime (August is over). What is the best process to continue now? Ask them again via the same FOI, create a new one, etc.

[rugk]

They promised it until the end of September for teh "CWA-Store data" at least, only. So let's still give them that time. Though I'd also appreciate new science blog posts… are there even more to come? :thinking:

Anyway, feel free to ping me again at the end of September. :upside_down_face:

[dsarkar]

Hi @rugk @Ein-Tim,

Yes, there will be more science articles, currently in preparation, and also there will be updates on the last one. Also, English versions are being prepared. Indeed, let's have some patience and end of September we can ask about the raw data if not yet published.

Best, DS

---

Corona-Warn-App Open Source Team

[Ein-Tim]

@rugk Hello again 😉

[MikeMcC399]

To repeat the essence of what I wrote in https://github.com/corona-warn-app/cwa-wishlist/issues/451#issuecomment-863938244 on June 18, 2021:

This issue is closed because the original problem description in https://github.com/corona-warn-app/cwa-wishlist/issues/451#issue-857593039 has been resolved.

The problem description was:

"The results of data donation are missing on https://www.rki.de/cwa.html."

This is resolved, because https://www.rki.de/cwa.html points to https://www.coronawarn.app/de/science/ which contains the Science Blog.

---

I suggest that a new issue should be opened if there are remaining points which need clarification.

[rugk]

https://fragdenstaat.de/anfrage/ergebnisse-der-cwa-datenspende-2-corona-warn-app/

[MikeMcC399]

@rugk I suggest that you review the Analysis page https://www.coronawarn.app/en/analysis/ and the blog post announcing this on https://www.coronawarn.app/en/blog/2021-10-04-key-figures-dashboard/.

Once again, if you have any additional requests I would be grateful if you could start a new issue, since this one is closed.

[rugk]

Ah thanks a lot @MikeMcC399, this was not posted before in this issue as far as I see. That indeed looks promising.

The only thing is the data is not actually on GitHub as far as see… :thinking:

Also, having looked into it, it is live data. (which is also very nice and good!) But it seems to be absolutely unrelated to what the issue here asks for, which is the results of the data donation.

[MikeMcC399]

@rugk

But it seems to be absolutely unrelated to what the issue here asks for, which is the results of the data donation.

You seem to have overlooked the green and red warnings from data donation. (Warnungen: rotes Risiko & grünes Risiko). If you select "Gesamter Zeitraum" on https://www.coronawarn.app/de/analysis/ you will see that there is no data before data donation started.

https://www.coronawarn.app/en/blog/2021-10-04-key-figures-dashboard/ says:

These key figures come from various sources:

• Downloads: Data from the Google PlayStore and Apple App Store.
• Test registration, test results, and sharing behavior: Information from the backend database - where tests are registered to allow results to be communicated and others to be warned if necessary.
• Green and red risk warnings: data from Privacy Preserving Analytics (PPA).

https://www.coronawarn.app/en/science/2021-06-15-science-blog-1/

says:

Two methods were developed to collect these data: an event-driven user survey event-driven user survey and voluntarily donation of usage data (Privacy Preserving Analytics, PPA).

[MikeMcC399]

This issue is closed, but just in case anybody missed the Science Blog article: Wer nutzt die Corona-Warn-App, wen warnt sie – und wie schnell?, there is another installment of reporting on data gathered through Data Donation / Privacy Preserving Analytics (PPA) in the Science Blog - in German initially.