Fundamental issues with contact tracing raised by Dutch experts must be addressed to show viability of DP-3T project

[winfried]

Yesterday and today there were in the Dutch parliament hearings about the use of contact tracing apps and also several official rapports on the use of contact tracing apps were published. And though the DP-3T was mentioned several times as benchmark for privacy by design, the issues raised are so large that I think the DP-3T should formulate an answer to those before continuing development. Without it the credibility of the DP-3T project will diminish.

Among the issues are:

• The Dutch scientific board for health concluded that the effectiveness of contact tracing for suppressing COVID-19 is still open for scientific debate and that beside one model calculation showing that it might be possible there are others that show that even with 100% use of contact tracing, it will only slow down the spread of COVID-19 marginally. https://www.gezondheidsraad.nl/documenten/overige/2020/04/20/verslag-wetenschappelijke-discussiebijeenkomst-covid-19-apps-als-onderdeel-van-een-exitstrategie
• The Dutch Data Protection Authority concluded that there are so many questions about the deployment of contact tracing apps, including two proposals based on the DP-3T protocol, that it is impossible to judge if they are legal at all. https://autoriteitpersoonsgegevens.nl/nl/nieuws/ap-privacy-corona-apps-niet-aangetoond
• In the parliamentary hearings all of the experts raised issues about the use of contact tracing apps. Amongst it were issues about the use of bluetooth technology and the amount of false positives it results in, about the amount of tests needed, about such an app results in employers demanding a 'safe status' before going to work, about what people can do when they are falsely marked as 'contaminated', about the app possibly creating extra risks for groups who are already at risk, like elderly. When asked, a part of the experts outed serious doubts if such an ap, looking at the whole picture, would ever be viable. The other part of the experts was shared the opinion that an app like this will never be viable. There was a strong opinion among the experts that it would be better to allocate the resources to different, more promising and less problematic, projects.

I hope the DP-3T community is able to address these issues so we can come to a viable strategy against COVID-19. A possible strategy would be to not deploy contact tracing as suppression strategy but as part of a research strategy to learn more about the parameters of the COVID-19 virus.

[kholtman]

@winfried I have also been looking at the Dutch briefings, but on some aspects my take-away is different, especially about the recommendation in the title of this comment. Note that I have made other references to the briefings in #188

The Dutch government has said that they will not be launching a Bluetooth contact tracing app in the immediate future, and they might never do so. This is all gated on resolving many issues, obiously the question of does it work, but also including (and this is new) passing a full human rights impact assessment, e.g. on the risk that the app may have side effects like employers effectively banning non-users of the app, which limits their freedom of choice. I feel that the Dutch government legal framing which explicitly identifies human rights issues is a very good development for DP-3T: it allows the project e.g. to argue the case for their vision by pointing out in their documents that at least one EU government has gone on record to say they care deeply about the human rights angle.

The legal tests the government is considering as part of its gating process will be applied to the act of the government uploading the app to the app store.

The Dutch government has made statements that it is still encouraging further R&D by open source projects on the matter. So implicitly, they do believe that it is legally possible to do open source R&D on this without breaking the law. (Warning to all open source developers: this does not imply that a medical-type open soouce project does not need to consult any lawyers. Consult a lawyer. Have a long discussion about the risks that some people might believe your app has a medical effect even though you have put in many messages that all data is dummy data. Have a long discussion about avoiding the legal conclusion that you are doing medical trials. Etc etc,)

[cascremers]

Thanks for the input and your opinion.

It would be convenient to have separate issues for the different items, which would facilitate discussion, and could perhaps use more useful descriptive titles. (The current one is not quite in line with what you write later.) Would this be possible?

With respect to your second bullet: Note that if I understood the (rather brief) Dutch process correctly, the interviews did not explicitly consider DP-3T, but two companies who proposed to build apps based on DP-3T, but had not been in touch with us. If this is the case, it is not clear to me how any conclusions (legal or otherwise) about those app proposals can be validly extrapolated to DP-3T.

We highly encourage in-depth comparisons and analysis, but perhaps the timeline of the process you cite didn't facilitate that.

[winfried]

@kholtman thanks for pointing to #188, I made a comment there too. I don't cite the course the Dutch government takes but the opinions vented by a broad line of experts in the parliamentary hearings. They were all very critical about the viability of the kind of project DP-3T embarked on. The current course of the government is to put a stand alone contact tracing app on hold and to start investigating what kind of app would support the manual contact tracing by the regional health services. That will most likely become an administrative support / questionnaire app.

The whole privacy / human rights / legal admissibility discussion will be short circuited if the assumptions DP-3T is build upon are incorrect and the protocol does not offer a way to suppress the spread of the virus. So if DP-3T can't prove the experts in the parliamentary hearing wrong, then everything else is useless. So I think it would be best to halt the project until it is assumable the type of contact tracing DP-3T aims for is viable.

[winfried]

@cascremers I oppose to splitting these issues because they are closely connected. Legal admissibility depends on proportionality. Proportionality depends on the balance between effectiveness and harm done. So all issues are interconnected.

The parliamentary hearings were not part of the overly hasty process of the Dutch ministry of health, but an independent and broad investigation into the viability of contact tracing apps. The stand of those experts was critical to outspoken negative. Continuing development of DP-3T without addressing the viability issues raised by these experts, would be a waste of sparse resources. So I also oppose to changing the title of the bug.

[kholtman]

@cascremers , all: Here is more detailed information and analysis.

Summary of the Dutch contact tracing appathon process for non-Dutch readers

What the Dutch government has been doing in the last 2 weeks is a rapid prototyping exercise, which is a valid software engineering approach if you want to uncover feasibility and risk open issues fast. We can also interpret it as the requirements elicitation process technique of creating a GUI mock-up and showing it to all stakeholders, inviting them to share their first impressions.

The government did not say, in their communication announcing this exercise, that this was rapid prototyping. They were implying that they might ship the prototype in a few weeks, if it passed review. Was this framing an act of stupidity, or a very clever ploy to draw more eyeballs towards the bug finding process? Dutch opinion is split on the matter. The whole thing certainly focused the minds of many stakeholders in Dutch society, including the mind of parliament and the minds of the privacy and human rights activist community.

There was zero chance that the rapid prototypes reviewed would pass review. The government invited as reviewers: privacy experts and GDPR certification experts, oversight bodies, human rights experts, security pen-testing experts, code quality experts, experts in GUI accessibility issues for the elderly, deaf, and blind, and last but not least contact tracing experts, and asked them all how they would feel about shipping this app real soon.

Not just one, but 7 rapid prototype apps were evaluated, these were made by 7 (all non-government) volunteer teams, invited based on an open procurement call. I believe at least one open source team created a quick shell company so that they could meet the open call phrasing. This was all necessary: if not, certification experts would have complained that an open source project based app, because of open source license conditions saying that all liability is with the user, is not an artefact that is capable of ever passing e.g. GDPR certification.

The open call selected 7 teams, and left these teams 1 day of time between being told that they had been selected and meeting the invited experts for a 2-day review of their running app. The expert reviewers were given had 2-4 days to produce a review conclusion with respect to their area of expertise. No surprise what happened. Conclusions: 1) insufficient data to decide, 2) insufficient data and time to decide, 3) this whole idea of 60% voluntary adoption of a data collection app proposed by the government opens up such a big can of human rights effect issue worms that we may never be able to come to any conclusion on, etc etc. The experts explained in detail what they were thinking, after seeing these apps, so this is very valuable information.

One of the 7 Dutch teams submitted an app based on forking the DP-3T project code. It is unclear to me how many changes they made, but they stated that they were implementing the solution proposed by the DP-3T project.

The software quality experts and security pen-testers were very negative about the quality of the code in all 7 apps, which basically should not surprise anybody, given the time the teams had.

It is still unclear if any contact tracing app will ever be shipped by the Dutch government, there is the will it work question, but even if it works technically, what may stop shipment is that the expected human rights effects will be be unacceptable compared to the value that is delivered, or simply undecidable. Manpower capacity to support manual contact tracing is being ramped up: the Dutch framing is that a tracing app needs to have the function of making this manpower even more effective.

Implications for DP-3T and open source in general

All of the above does have important and interesting implications for the source code in the DP-3T project repository. Though this was not said explicitly by the government as far as I know, I can make a prediction. If it ever even ships a contact tracing app, no contract tracing app shipped by the Dutch government will ever contain a single line of code from the DP-3T repository. If any decision is made to 'use DP-3T' by government and approved by parliament, they will pay a commercial development team, which has experience with writing apps for banks and the like, to do a complete clean-room re-interpretation, for security and trust reasons. The code made by this paid expert team will get an open source license, and will be put on GitHub (or other site) where everybody can review it. This will be seen as a way to boost confidence leading to higher voluntary app use. (Note that this is my expert opinion/prediction, there has been no official policy statement about such matters by the Dutch government.)

In the larger European context, one can interpret DP-3T as being another rapid prototyping exercise that is also creating useful insights via expert review, drawing eyeballs by being somewhat vague about its intentions, just like the one I described above. I am not sure if this was planned by the team all along or if it has been more of an accidental side-effect, but the value being created here by the open source community is large, and I would like the project to continue to play a part in delivering this value.

The project is also identifying itself as a brand, as a label for a vision about app based contact tracing, that can be compared to other such open source brands, and to official government apps. So by acting as a stable point reference in an open and democratic triangulation process about what the real needs and solutions are, the project is adding value. It is much easier to find out what you really want if you have several branded open source prototypes with documented protocols and security models that you can compare and contrast. Again, the DP-3T project team has choices ahead of them here, e.g. on how much energy to put into outreach and advocacy that will reach and engage the general public, not just the technical and activist community.

There are deeper and more general consequences here for the role of open source as this process continues. I am working on a deeper analysis that I hope to post in future.

[winfried]

@kholtman thanks for this good write-down of the "fascinating" Dutch debacle. I would suggest to move the discussion about the positioning of DP-3T in the landscape (and politics) of contact tracing apps and the role of the open sources community in that discussion to a separate issue.

This issue is about a broad and highly respected group of experts independently from each other raising fundamental concerns about the viability of contact tracing apps in official documents and official hearings. It would be irresponsible if the DP-3T pushes on without addressing these issues first.

[kholtman]

@winfried Thanks! I think it is valid for the team to push their vision as a brand. I believe you are talking above about the level of irresponsibility if they would ship their current code as an app to the general public, while making actual health benefit claims about it.

I would suggest to move the discussion about the positioning of DP-3T in the landscape (and politics) of contact tracing apps and the role of the open sources community in that discussion to a separate issue.

Sounds fine to me. Maybe this issue title should be updated to something like: Analysis: Dutch government decides that any type of contact tracing app launch in the Netherlands should,,,,

[winfried]

@kholtman The goal of this project states:

Its goal is to simplify and accelerate the process of identifying people who have been in contact with an infected person, thus providing a technological foundation to help slow the spread of the SARS-CoV-2 virus.

The questions raised by the experts (what is something different then a governmental opinion) seriously doubts the viability of exactly this goal. The short summary of the issues raised: it is highly doubtful if it can ever function at all and if it functions it is highly doubtful the benefits outweigh the harm done by it. We can't ignore these issues but have to address them: if the issues raised by these experts are valid then this project can not meet its goal but still has the potential to harm society. In that case it would be irresponsible to continue the project. So it should have the highest priority to address the issues raised by these experts, so the DP-3T project demonstrates it is a responsible response to COVID-19.

Maybe this issue title should be updated to something like: Analysis: Dutch government decides that any type of contact tracing app launch in the Netherlands should

Would the following suit you better? Fundamental issues with contact tracing raised by Dutch experts must be addressed to show viability of DP-3T project.

[kholtman]

@winfried

Fundamental issues with contact tracing raised by Dutch experts must be addressed to show viability of DP-3T project.

Yes, that title sounds fine to me, better than my proposal.

However, note that the experts invited by the government had fundamental issues with all 7 apps shown, and some of these used a more centralised protocol architecture than DP-3T has. So for me, it is not just DP-3T that still has a lot of work to do.

[burdges]

Israel stops using phone tracking to enforce COVID-19 quarantines

[bartpreneel]

As for effectiveness and privacy: the current proposal on the table is manual tracing. This is highly privacy invasive (with risk of abuse by the interviewers), and so far has shown to be not very effective. Manual tracing can only identify contacts you remember and you can identify (hence not a stranger you sat next to on the train for half an hour). While we may not be sure that contact tracing apps will work (that is a perfectly valid scientific statement that a scientific board can make), we are quite sure that manual tracing will not work. Hence a combined system has a good probability to be more effective. As for the human rights impact: IANAL, but the EPDB has made a clear statement.

I don't know what the motives were for the Dutch rushed approach, but everyone could have predicted that this approach would fail badly and that it would tarnish the reputation of all contact tracing apps. As a consequence, public opinion and politicians in the Netherlands are now perhaps somewhat biased. I believe that other countries are taking a more careful approach. While the questions raised are of course legitimate, personally I do not think that it is appropriate to start exporting this negative experience to other countries by suggesting that all developments should be stopped until we are 100% sure that everything will work perfectly. Just my 2 cents.

[winfried]

@bartpreneel investigating is necessary for other countries too, because of the first bullet point I mention: there are from an epidemiological point of view serious doubts if contact tracing (in any form) can lead to a significant reduction of R0 with the SARS-CoV-2 virus. It worked for other infectious diseases, but it is very uncertain if it can work for the SARS-CoV-2 virus.

Continuing development of DP-3T without first securing the assumption that contact tracing can be effective in reducing the R0 of SARS-CoV-2, would be about as harmful and unethical as suggesting that drinking disinfectants would clean you from the virus.

[kollokollo]

As far as I have understood, R0 is a fundamental constant of the virus (acting in a specific society) and cannot be modified. Maybe you mean R_eff ? The reductions do not directly come from the app, but from isolation and quarantaeine. And this process is hoped to be improved by the app.

[winfried]

@kollokollo Lets not start a semantic discussion on R0 or R_eff (can spend hours on it itself, but that won't be productive here).

The ultimate goal is a reduction of the spread of SARS-CoV-2. You correctly say that that process is hoped to be improved by the app. But what if the the whole idea of contact tracing and quarantine fails to reduce the spread of the virus? Then the hope underneath the app would be false.

The concerns raised by epidemiologists are exactly about that: can contact tracing be effective at all? DP-3T can't continue without addressing those concerns first. It makes the difference between justified hope and false hope.

[baryluk]

Just wanted to give my input, I am not an epidemiologist by any means.

Wouldn't doing comprehensive Monte Carlo simulations with various underlying assumptions, and people behavior, provide insight on

1. effectiveness of the DP-3T (or any other wide spread mobile phone based tracing, centralize or no centralized, privacy safe or not, doesn't matter),
2. how it depends on various other factors like delay between being infectious to diagnosis and isolation,
3. how it compares to control (with no DP-3T), or manual tracing, with only part of contacts identified,
4. how big part of population need to be using DP-3T to make any difference in effectiveness,
5. if there is a difference, how much more looser population regulations (increased social contacts, especially in public spaces) can be to still make tracing effective,

?

All I see in this and other threads it that you "hope" it will work, or a intuitive believe that it is not worse than manual tracing. But that is not enough, and can be done much more objectively.

Certainly modeling and running realistic simulations is not easy, but not impossible either. It could be a synthetic spatial model based on some generic metrics from real world, or semi realistic model based for example on people / mobile phone movements data from various other studies. Non of these really need to be fully accurate or unbiased for good quality of modeling or determining effectiveness of DP-3T vs other methods.

Of course it will not solve the other problems, like privacy and legality, as even if the underlying DP-3T protocol is proven to be secure, there are still plenty of places in the implementation and deployment that can leak privacy sensitive information, be abused, could be attacked using side channel attacks, can have problems in key generation methods, or be deanonimized by sufficiently big adversary, at least when targeted.

[kollokollo]

Ok, point taken. But simulations which I have seen so far on the spread of SARS-CoV-2 are based on multiple assumptions as well. Each of them can be questioned as well. So one would need very good ideas how to do it.

[claustres]

I am not a security expert neither an epidemiologist, however it seems to me that we are expecting a lot for all contact tracing initiatives so far while we don't expect much from less technical initiatives. In order to provide a deeper discussion I would like to recall what is going on in France as an example and to put into perspective some specific points.

For more than a month, and still for a couple of weeks, we have been under containment, which is probably the most important deprivation of freedom since the world war II, yet a few people asked government to "prove" it was the right thing to do. As far as I can see only a couple of countries has been served as an example and we are asked to take experts at their words although we don't have any history of containment. A lot of people are arguing contact tracing apps will not target the at-risk population but most of the population under containment is not at-risk either, yet everybody implicitly accepts that it indirectly serves the purpose of saving at-risk people by stopping infection chains. A lot of people are arguing contact tracing apps might divulge infected people identity if hacked while infected people are already forced by health authorities to quarantine themselves, tell their relatives, employees, colleagues, etc. they are hill in order to stop infection chain.

In a couple of weeks containment measures will end, yet a few people ask government to "prove" it is the right thing to do. Nobody actually knows if the reopening of schools is a good idea, if the fact that there isn't enough "certified" protective masks for the whole population is a big problem, if all people who have recovered from COVID-19 have antibodies, if we are able to test a sufficient number of suspected cases per day, etc. All experts seem to agree on a fact however: there will be a second (and maybe a third) wave of the outbreak. On the 30th of March, for a while during the night, there was no room in the hospitals around Paris to take care of a couple of patients, fortunately the peak was near and things have gone better after that. A new wave higher in either cases count or duration will be for sure a disaster and we should at least allow to test if contact tracing can help on a voluntary basis. Indeed, what is also pretty sure is that existing isolation process is not efficient enough otherwise we shouldn't have had to enter containment.

My humble opinion is that of course we can debate about pros/cons of contact tracing apps and try to ensure they are built using the best security practices but we should stop asking them to be better than we actually are. Contact tracing is inherently "invasive", whatever the technical measures in place to protect against deanonymisation, like containment or preventive measures are as well, so the key to protect privacy as much as possible is also in the operational processes, the governance and oversight of the authorities conducting it. Of course technology should only be one piece in the puzzle of the investigative work and keeping human-in-the-loop is important.

[kugelfish42]

Nobody knows if Bluetooth based proximity tracing in an operational deployment is ever going to be working effectively and we won't know until quite a few months after some governments take a leap of faith on partial data and educated guesses. Technically it looks like a long shot, a real desperate Hail Mary Pass. Something we would probably not seriously consider in less desperate time and many reasons for why have been brought up in this forum and elsewhere.

Because of that uncertainty, it should be important to minimize the potential of the system to do harm such that the net impact should be expected to be positive. The distributed approaches proposed here might not maximize usefulness as it deprives the government of some information it might be able to use, but it also fairly well protects the vast majority of users (those who don't submit their keys as infected) from involuntarily disclosing information other than a possibly increased Bluetooth radio signal noise.

But in the end, some people in political leadership positions will have to make a call and everybody in their jurisdiction will have to decide whether to go along with it or not.

[claustres]

What I am saying is that in this times of uncertainty people in political leadership positions (or even experts) take a leap of faith on some decisions every day. Often things we would probably not seriously consider in less desperate time either (eg containment, always wearing protective material). Often things that can be a lot more harmful than contact tracing because it can cost lives but that we can't prove to be right before trying.

Contact tracing is only one problem among many many others. So it seems to me that the energy spent by some people focusing on the minor technical details of contact tracing is far more important than the energy they spent to question others aspects of our response to the outbreak. Maybe my impression is biased because as open source developers we are used to discuss issues like this with adequate tooling easing the freedom of expression.

Anyway, don't get me wrong, I don't know if contact tracing will work, but just like many others measures that have been taken we will only know after trying. Maybe it would be better to allocate the resources to make it real sooner with the best "quality" possible so that we will also know sooner if it can help.

[winfried]

The best scientific epidemiological discussion I have seen so far about the feasibility of contact tracing was in the Dutch scientific board for health. The report is here: https://www.gezondheidsraad.nl/binaries/gezondheidsraad/documenten/overige/2020/04/20/verslag-wetenschappelijke-discussiebijeenkomst-covid-19-apps-als-onderdeel-van-een-exitstrategie/Verslag-wetenschappelijke-discussiebijeenkomst-COVID-19-Apps-als-onderdeel-van-een-exitstrategie.pdf (In Dutch unfortunately, you may have to run it through a translator.) They open the discussion with the input of two modellists, one saying that contact tracing may result in a reduced R0, one saying that any form of contact tracing is ineffective. The scientific board concludes that both are models with errors, but has the most questions with some of the assumptions of the model saying it is possible. The final verdict: it is not yet clear enough if contact tracing can contribute at all in reducing R0.

When I read the paper stating contact tracing can reduce R0, then it states some very clear but also very tight limits to the contact tracing for it to be effective:

• The time from somebody getting first symptoms to putting the contacts of that person in quarantine has to no more then 24 hours. Otherwise the contact tracing is ineffective (that is why they say it is only possible with an automated app and not manually).
• The success rate of detecting new cases has to be at least 50%.
• The success rate of putting contacts into quarantine has to be at least 70%.

It is hard to compare the parameters of the models because they have different structures, but when I put the parameters commonly used in other models into the model saying that is saying that contact tracing is possible, then the margins disappears and it becomes an 'impossible'. But let assume the the model is correct, then lets evaluate the constraints:

The 70% success rate of putting people into quarantine depends on a lot of factors. For example if people can separate them from others (living in dormitories, homeless, willing to go into quarantine or forced quarantine) that comes on top of the success rate of detection of contacts. Lets make the very positive assumption 90% of the people indicated that they should go into quarantine go into quarantine. Then you can miss only 20% of the contacts of the new cases before the contact tracing comes dysfunctional. But in the Netherlands for example only 80% of the population has a smartphone, so even with an install base of 100% (where 25% has to be proven realistic) you can't detect enough contacts. But lets fix this by handing out smartphones to the last 20% of the population, making the contact tracing app obliged and force quarantine by sealing peoples doors locked. Then still the detection of what is a 'contact' doesn't allow for more then that 70% false negatives. That is already close to the estimate of contaminations that goes via surfaces like door handles (15-30%). But lets ignore that one too and look at the false negative margin of 70%. To reach that, you have to overshoot: make sure the circle to mark somebody to be forced into quarantine not too small and make sure the time to trigger quarantine is not too long. Result: you create more false positives to avoid the false negatives. The study doesn't estimate the amount of false positives, but when I try to estimate those, it results in excessive amounts. With the current amount of positive tests (what is if far below the needed 50% detection rate) all of the Netherlands has to be forced into quarantine. So we need a strategy to reduce the false positives. Testing all contacts is not an option for two reasons: first of all you need to have the testing capacity to weekly (or so) test all of the population, second the time frame is 24 hours and with the current tests, you need all of those 24 hours for the first test. The same is with manual sifting through the contacts: that would lower the precision and also introduce a delay. So we need to find some very smart methodology to detect with more then 70% precision contaminations with only something like 100% false positives at max. I can't think of any way of doing so, but my compliments if somebody can. Bluetooth won't be the solution here, I can already tell. And then comes the amount of tests needed for this kind of work flow. In the Netherlands we would need 5 times our maximum test capacity (of which we currently use only 25%). It would probably (as indicated by some models) be much more effective to use these tests for groups at risk, people with occupations that can potentially make them super spreaders or just for at random selected people.

So even when I take the most favourable model, go with its favourable assumptions, assume the from epidemiological point of view most favourable workflow (which is not acceptable in western societies), then still I can't get the maths added up.

Hence my question to the community here: are we sure we are not creating snake-oil that will only have harmful effects?

[claustres]

Hence my question to the community: are we sure ending containment without any viable contact tracing option (as you seem to highlight that manual tracing is not effective as well) will not have harmful effects only ?

It seems to me nobody can be sure either so be ready for the status quo ;-)

[kollokollo]

Hm, maybe one can make it more plausible. Therefore a process description of the process which is used currently (and "manually", means, without app) by the national health organizations could be used as a starting point. (can someone step in here?) If this would be a graphic (like a process diagram), one could circle out the parts, or the process links where the app can (possibly) improve or change something. This could be: speed up a step, help double-check, help remembering, quicker inform, etc... It would be best to have numbers there, but i fear, for the moment we dont. But isn't it obvious, that e.g. a speedup of any part of the process can only help? Maybe not.

If at some point e.g. the number of false positives is increased by the app, one would need to identify another compensation effect, and then one can leave it to the numbers (to be measured later) to prove the point. I hope I can make myself clear: For example: If you increase the number of false positives (to be put in quarantaine) and(!) increase the speed of the process, this might result in soon having half of the population in quarantaine (unnecessarily). So in this case the compensation measure would be to reduce the speed.

Or another example: If the app can help remember more contacts than the (diagnosed) patient would otherwise remember, this would increase the hit rate, but if you do not also increase the process in informing them (the contacts) in time, the quarantaine (for the contacs) would start too late, so that the contacts have already infected others; so the increased hit rate has not helped at all, but produced unnecessary overhead.

It is really hard to describe all of these paths in text. A process map could really help. And also teach us (especially all non-medical contributors) how at the moment the process of tracking down contacts of infected people is done. The app then would probably not completely change the process, but just modify specific steps of the process.

[kholtman]

All: great to see that the issue title is also drawing some non-Dutch perspectives into this discussion. I am allocating most of my time to some other issues right now, but I will try to summarise my current personal stance, based on remarks made and analysis I did myself (see e.g. #188 )

• As noted in my earlier writeup above, Dutch opinion on appathon has been split. Sampling Dutch social media discussions in the last few days, Dutch opinion on `can the app ever work' is still split as well.

• This very splitt-ness tells us the result of a kind-of experiment that was run in the Netherlands to try to predict the voluntary adoption rate of a contact tracing app if introduced immediately. Experimental result: there is absolutely no way this will reach 60%, or even 10%, if I am reading Dutch public opinion right now.

• My professional opinion is that this negative experimental result will need to be addressed by doing more work, it should not be used as a reason to end work on this type of App by the broader community.

• My reasoning relies on a Battle of Britain ( https://en.wikipedia.org/wiki/Battle_of_Britain ) analogy: contact tracing app technology R&D is like the R&D on radar in the Battle of Britain: radar was based on just-new tech, initially highly speculative, initial experiments were disappointing, nobody knew if it could ever work, or how well, but if all problems could be resolved it would be a great force multiplier for the defending Air Force.

• Where the analogy breaks down is that radar did not require a 60% adoption rate by all British households owning a radio to be effective.

• In the remarks made after the appathon by Dutch epidemiologists involved, there was the quite interesting remark that, though different models studies varied wildly, they showed that the effectiveness of digital contact tracing as a force multiplier depends strongly on the availability of testing capacity. I believe this refers to the tests for people who self-report as possibly infected, but I am not sure what test type (viral particles in swab? antibodies?) or test accuracy the speaker had in mind. Testing Capacity is very low now in most (all?) EU countries, but who knows where we will be in 5 months: a lot of effort worldwide is going into ramping up testing capacity.

• So my mental model is that the community is doing R&D for an app and protocol that might only become effective and used 5 months from now, or 2, or never, who knows. There are plenty of other issues with this newfangled form or radar, like getting privacy/human rights right.

• There is a danger though that the Bluetooth contact tracing app idea will draw attention away from R&D on other types of app that might be equally effective while having little or no privacy impact at all. To give a somewhat contrived example: Apps for hand-washing technique or hand-washing reminders.

[winfried]

I think this discussion is moving the right way: we can't say anything sensible if we haven't described a workflow of how an app will be deployed (@kollokollo). And there are indeed many uncertainties, so yes this is a speculative technology (@kholtman)

We should not close our eyes for how speculative this technology is. It is not given it can work at a theoretical level, it is far less given that it can work in practice.

To give an example: I have never seen supporting evidence for the 60% needed penetration grade for an app that is often mentioned in the Dutch discussion. The epidemiological models I have seen suggest way above 70% or even way above 100% (aka impossible). I suppose they took the 60% group immunity that is a rule of thumb in epidemiology and equalled app usage to group immunity.

So we should avoid to make any claim that this is more then a highly speculative technology to avoid false expectations with policy makers and with the general public. I propose to put a disclaimer in every communication about DP-3T on place were people can't avoid it. Agreed?

Disclaimer for DP-3T: Contact tracing by mobile applications is a highly speculative technology. It is far from certain it can contribute to stopping the spread of the SARS-CoV-2 virus at all or to what extend. Relying on this technology as important part of the strategy against SARS-CoV-2 is likely to harm to the health of a population.

We know the technology, we know the issues it has, we know how speculative it is. It is our responsibility to be upfront about it, otherwise we may cause serious harm.

[kholtman]

@winfried I support your proposal for a disclaimer; this would avoid a lot of confusion. But I based on my experience with open source legal theory and medical law applying to software and systems I propose improvements, as follows.

Disclaimer related to DP-3T project designs and software

Contact tracing by mobile applications is a highly speculative technology. It is far from certain it can contribute to stopping or managing the spread of the SARS-CoV-2 virus at all or to what extent. The open source license of the project implies that the project can offer no warranty and disclaims any liability for the use of its designs or software.

Any statements about e.g. GDPR certification, medical certification, and aimed-for medical effects that are made to by the project or contributed to the project should be interpreted as being speculative in nature, applying to a work in progress.

If you are planning to use the software or designs in a trial deployment or production deployment of a contact tracing system, you should take great care, and do your own independent evaluations, to avoid breaking any laws like the GDPR or applicable medical laws, and to avoid creating any harm, medical or otherwise, to the population targeted by your deployment.

Note to the project on the nature of this contribution: I am not a lawyer, so I recommend that the DP-3T team finds a volunteer lawyer to check and probably improve the disclaimer above. It is unclear to me if there is any legal theory at the moment that can support a conclusion that the MPL 2.0 license used by the project or the text I wrote above combined will be enough to fully shield everybody from prosecution under country-local or state-local medical regulations designed to prosecute specifically against misleading the public about medical matters. I am not sure if any other project has already consulted a lawyer -- maybe something better than what I wrote above is already available in another project repository.

[kholtman]

@Ibarman @winfried I just realised: maybe the above project disclaimer discussion/input is better moved to a new issue with its own title. If you feel so too, please go ahead and open the issue, copying text into it.

[baryluk]

I see a lot of claims here, including for these Dutch epidemiologists, but what they are based on? They supposedly had some model or experience to draw the conclusions and the numbers they presented. Is there some publication or literature to back this up? Some open source code to run their models and verify when it is effective and when it is not? Or is it just all based on intuition of these epidemiologists?

[kholtman]

@baryluk Good question. The Dutch epidemiologists I have been quoting are in, or closely related to, the government team of lead epidemiologists working full-time to manage the covid outbreak for the entire country, so they are not just anybody, and they have whole university departments and international peer networks feeding them data. You might say they have been publishing their preprints, and comments on agreement and disagreement between different models, in briefings to parliament broadcast live on Dutch national TV, and in the appathon livestream. You can find all this video archived on YouTube, but it is all in Dutch, so I have been extracting and sharing the relevant-to-apps points made in English here.

So I have some trust in these people when they make forward-looking statements, but an orderly peer reviewed scientific process this is not. They have their own models specifically tuned to the Dutch situation, models which are currently secret, but they also compare the outputs of their own models to those of other models made abroad. The current state of closed-source and non-peer-reviewed play can (still) be excused by the urgency of the current phase of the crisis. But there have already been loud calls by prominent scientists to set up a second team independent of government, which is also more multidisciplinary, to restore some more openness and diversity of viewpoint to the process as lock-down restrictions are (hopefully) lifted.

[mattijn]

@kholtman, I might be wrong, but I did not see RIVM participating in that event

[baryluk]

@kholtman That is all fine, and there are reasons for some models and decisions to be made in secret or in a rush. But there is no way to verify these claims only on trust and past expertise of these people. They can still make mistakes, they can have hidden agendas and biases, it might be based on specific assumptions that only work well in specific areas or in specific countries (and be influenced by hundredths of factors, including density, age distribution, transport modes, region connectivity, culture in general, even weather, etc. etc).

I am not saying their claims are false, I am saying there is no way me, or anybody else can verify it by reproducing their methodology, or finding out where it applies and where it doesn't. And that is the problem, because there is no way even to find flaws or do any kind of discourse about these claims. Also, maybe it place X it will help, and maybe in place Y it will make things worse. How would I know?

[kholtman]

@mattijn Can you unpack your comment for a European audience? I think you are trying to make the same point as @baryluk , but I am not sure.

@baryluk You make valid points about things like lack of political oversight, transparency, and accountability of the Dutch outbreak management process. But sorry, I am not going to engage with them here, in the comment section of a pan-EU open source project. I believe that such a discussion is better conducted elsewhere -- I see plenty of people discussing it in the Dutch media already.

On this site, I am trying to contribute expert opinion and information to the open review process of DP-3T that the DP-3T team is hosting.

[winfried]

@baryluk you are right that there is still scientific debate going on, but the evidence is getting clearer and clearer. There are some things to note: the modelling work of Frank Dignum, the expert heard by the scientific board is not secret or so, the model is open source and you can examine it and run it yourself: https://simassocc.org/ The same goes for the model behind the article: "Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing" by Luca Ferretti, Chris Wymant, Michelle Kendall, Lele Zhao, Anel Nurtay, Lucie Abeler-Dörner, Michael Parker, David Bonsall and Christophe Fraser. Close reading of that article and comparing it to other articles about modelling the SARS-CoV-2 outbreak reveals several things: the article uses some parameters and deduces some others that are very optimistic. Especially the percentage subclinical contaminations assumed in the article, 10%, is way lower then all other evidence suggests. Even with this optimistic estimate the model of that article states that contact tracing is only possible within very tight constraints: 50% success in detecting contaminated persons, 70% success in putting the contacts of those persons into quarantine. And the whole procedure must be done within 24 hours from the moment the contaminated person gets the first symptoms. If the delay gets bigger (delay before taking the test, time to perform the test, administrative delays), the contact tracing becomes ineffective according to that article.

The current estimate (lots of articles about that) of the percentage subclinical infections is in the range of 40%-60%. Entering that value in the model, or even an optimistic 30%, immediately indicates that contact tracing is not possible any more.

But even so: to reach the 70% of the contacts of a contaminated person in quarantine, you need a usage percentage of the app that is at least 85% of the population, what is higher then the percentage of the population having a smartphone in the Netherlands. But if you manage, lets say 90% penetration of the app, then you can barely afford any false negatives, so you have to compensate with overshooting, making the contact circle bigger and so putting more people into quarantine (Remember the maximum delay? There is no time for a confirmation test, contacts have to go into quarantine immediately, otherwise it won't work.) Add some errors in the distance measurement of the bluetooth to it and it results in an unacceptable amount of false positive contacts in quarantine (a significant percentage of the population). And then I even didn't estimate the testing capacity needed for the whole workflow (higher then almost any country has available). So even if I assume that the assumptions of the article are correct (what they aren't), it doesn't result in a realistic contact tracing scenario.

And that article was the only one I could find that is suggesting that contact tracing can reduce the spread of SARS-CoV-2. All other sources I have found just add to the evidence it is not possible and that is also the judgement of the experts I have spoken recently.

We can't ignore such serious doubts and must address them: it would be irresponsible to continue the project without a good chance it can be effective at all.

[burdges]

I'd phrase this differently:

There are nations like the U.S. and U.K. that already selected private surveillance contractors like Plantir to provide their contact tracing app services, meaning their decision was driven more by mass surveillance goals than by public health. DP-3T should exist so that nations could choose to experiment with a contact tracing app without engaging in such extensive mass surveillance. This is not because contact tracing apps might be effective, but because politicians may choose them regardless.

At the same time, there are poor odds that contact tracing apps actually contribute meaningfully to public health, but surveillance proponents may continue pushing them even after failures. I'd suggest that internally the DP-3T team should know compelling arguments that additional data like GPS coordinates provides nothing. There are many obvious arguments like install rate, battery consumption, etc. of course, but maybe some passing familiarity with the arguments cited here helps too.

[winfried]

@burdges alright, I understand your issue now, thanks for restating it.

But what would be the best narrative to counter it: "it is snake-oil with adverse effects, here is snake-oil with with fewer adverse effects" or should it be: "it is snake-oil with adverse effects, stop using snake oil"? It would be my take, but that is a highly subjective assessment from western European perspective, that it would be more effective to abandon the DP-3T project with a bang as loud as possible, stating that it can be done in a privacy friendly way, but that it will not be effective. That would undermine both parts of the 'palantir' approach. Usually it is stronger to play the effectiveness card then the privacy card.

[mattijn]

@kholtman, its weeks ago already, but in this event you refer to where no people participating from the institute responsible for the Dutch modeling (RIVM). So maybe what you’ve seen was very informative, but I’m not sure if it’s possible to be so certain about these statements.

[noci2012]

Snakeoil or not: https://www.reuters.com/article/us-health-coronavirus-india-app/india-orders-coronavirus-tracing-app-for-all-workers-idUSKBN22E07K This is force fed: https://play.google.com/store/apps/details?id=nic.goi.aarogyasetu&hl=en Mandatory for all workers there.

[winfried]

@noci2012 very scary. Image that that somebody is false positive indicated as possibly infected by such a state mandated app (a very likely scenario) and then not allowed to work and left without family income (also a very likely scenario, certainly in India). That is quite a responsibility for such a deployment. And on top of that come some hairy issues like who is liable: the employer for unjust excluding? The app builder or the organisation running the app for inferior contact tracing? The state? We really don't want to be part of something like that...

[winfried]

@mattijn lets forget about the Dutch saga (we can discuss that endlessly) and have a look at the modelling work that is public. There is a bit of uncertainty there. Partly because it are models, not reality. Partly because there is still a lot unknown about SARS-CoV-2. But the consensus among the modellers seems to be moving in the direction of: 'not possible'. Or to state it differently: 'possible' has more uncertainties then 'not possible'. How much certainty do we need to responsibly continue a project like this?

[kholtman]

@mattijn OK, fair enough. I think you are making a different point than @baryluk was making. Your point is:

but I’m not sure if it’s possible to be so certain about these statements.

I was not trying to claim any absolute certainty for the epidemiologists statements made in the appathon, I was reporting them as best current expert guesses.

no people participating from the institute responsible for the Dutch modeling (RIVM).

I believe there was a person from RIVM one in the appathon expert panel, but more to the point: the RIVM is not 'responsible' for Dutch modelling, they are one of the many parties who make such models, and in the end the political responsibility for the decisions driven by the models is with the government.

[noci2012]

@winfried

very scary. Image that that somebody is false positive...

Well the employers can point to the government: ==> mandatory.... Goverment: probably can't care less about such minute details..

[peterboncz]

Arriving late to this party @winfried

You mention that the Oxford Ferretti et al 2020 Science paper, estimates "subclinical contaminations" as 0.1 (10%), and find that low. I assume you mean "environmental infections", correct? Would you have references that put this higher? You also mention sources (hopefully scientific) that add to the evidence that contact tracing apps would not work. Would also be good if you could post (some) of these.

You also point to the ASSOCC simulation by Frank Dignum. It is so far unclear why his results are so different, and he has not published a textual or mathematical description of the infection model, implemented in his simulation. Recently his group's Netlego model (1000 or even 300 people simulated, not much) was put on github.com/lvanhee/COVID-sim, so maybe someone can reverse engineer that model out of the code. It seems it is quite a bit in motion right now. The recent preprint arxiv.org/pdf/2004.12809.pdf also does not describe the used model in sufficient detail to validate its epidemiological properties, and eschews the app controversy and discusses only school closures. Meanwhile on the 25/4 blog entry (rofasss.org/2020/04/25/the-assocc-simulation-model) we read that Frank Dignum is comparing his simulation with the Oxford group and their model. So far no conclusive statements, but my suspicion is that certainly in families the original ASSOCC model on which the claims were based did not implement significant isolation on app notification, nor modeled contagion in age groups well (e.g. children without phone, playing a small role).

You can of course doubt the detection precision of apps, their adoption, as well as indeed the effectiveness of isolation. These factors would diminish the effectiveness of an app. Certainly below the point that it alone would keep the epidemic in check. Still, even if the effect would be only a moderate reduction of R (e.g. by 20%, rather than a factor 3), an app could still be useful, in combination with other measures (e.g. scaled-up manual contact tracing, working from home when at all possible, mouth caps in public transportation, family-stay-at-home-on-fever). We won't know until we try, and meanwhile there are not so many other options -- always open for suggestions, though.