search

cds-snc / covid-alert-server

Exposure Notification: Diagnosis Server implementation / Notification d’exposition : Mise en œuvre du serveur de diagnostic
Apache License 2.0
298 stars 31 forks source link

Provide for improved key performance indicator: app adoption rate #391

Closed LeCanardQuoi closed 3 years ago

LeCanardQuoi commented 3 years ago

As a user, understanding the adoption rate of the app is vital to understand how I can rely on it and realistically expect more to adopt it in my geographical area. The number of downloads Canada-wide or province wide is inadequate as such a confidence builder indicator, as well as the number of diagnosed user reporting back. In support of App enhancement request #1305 using the 3 first digits (FSA) of our canadian postal code, the server should provide for reporting back to the app over and above existing minimum: number of mobile app users downloaded in my FSA, number of mobile app users active ed in my FSA, number of diagnosed users reporting back in my FSA and number of notified users following through. Ideally this data whould be reported alongside the corresponding virus transmission rate. People would then have far more ownership of this game changing app.

This data reporting enhancement would also be tremendously useful to assess the effectiveness of the app -vs- the adoption rate in targeted field trial.

CalvinRodo commented 3 years ago

This is on hold as it depends on a decision on the app side of things: #https://github.com/cds-snc/covid-alert-app/issues/1305

LeCanardQuoi commented 3 years ago

Cool! Would be great however to understand how #1305 will make it through as a "pull request". In the absence of any follow up on #1305 after several days, are escalations at the ministerial levels required? I have not been able to get the attention of any staff at CDS, including the product manager in spite of numerous good mannered follow ups. The holiday season is coming and it is most unclear how could #1305 not be further avoided. The clock is ticking away on readyness for the 2nd wave. Réjean B.

LeCanardQuoi commented 3 years ago

1305 was closed and deferred to discussion within the development group without any public visibility whatsoever, along with two other social distancing ERs #1264 and #1304. Reqs for social distancing basic heartbeat capabilities were flagged via a staff member to the development team (i.e. without an ER) over a month ago and there never was any follow up until I formally entered ERs on the app side and then the server side. Unclear if this is taken seriously at all in order to strive to a higher app adoption rate #75percentAdoptApp and save so many more thousands of lifes than what a 20% adoption rate can realistically accomplish. The "development team" is unlikely going to discuss over the upcoming holiday. And the clock keeps ticking as winter has not even started yet... :-(

CalvinRodo commented 3 years ago

I wish I could give you an easy answer regarding this feature request, however there is a lot of stuff to juggle regarding feature requests from the public.

For instance for this particular request to be addressed we need to review the potential impacts of tracking this information against what was put forth in our Privacy Assessment for Covid Alert. While the first three digits of the postal code does in most cases narrow it down to a large area we still need to review the potential impacts of this data now being tracked.

We also promised that we wouldn't pull a bait and switch by tracking things after users install the app so we have to assess the optics of a change like this as well. Then there is discussions that have to happen with various stakeholders (provincial and territorial partners, privacy commissioners, MPs, etc...). We also have to assess the financial costs associated with a change to how the system works and the increased data that is getting transferred back and forth. Plus if we do decide to go ahead with this feature request we have to design changes to the system, research those changes to make sure we don't negatively impact the application, implement it, draft comms plans, etc...

Needless to say it's not as simple as just writing some code.

We also have to prioritize any requests from the public against existing work that is currently happening on Covid Alert and new features that are in the pipeline. I would like to say we have infinite people to throw at this tool but the reality is that's just not the case, the app team is currently working at full capacity as it is.

What I will say is that we do take each and every single request that is submitted to our repositories seriously, I do sincerely apologize that we aren't being as responsive as you'd like.

Please continue to send any ideas you have our way. Github is a great way to engage with us for these things as it allows us to track it against other work that we are doing, also we can be reached on twitter @CDS_GC or @SNC_GC.

CalvinRodo commented 3 years ago

That being said I am going to close this issue and will re-open if we decide to implement this feature just to keep my backlog clean.

LeCanardQuoi commented 3 years ago

Lessons from #Ebola: "Moving quickly is essential and avoid aiming for perfection before making a move. Perfection is the enemy of the good."

I have proposed two game changers to get this app to a much higher adoption rate and save the lives of far more Canadians during the 2nd wave. I am sure there are many great reasons (mainly bureaucratic and reminiscent of Phoenix) to avoid such game changers, to keep holding current lame course and keep contending with disappointing lives saved marginal results of a 10-20% app adoption rate at best.

LeCanardQuoi commented 3 years ago

Any committed date by which a decision will be taken to provide for an improved key performance indicator, the app localized adoption rate?

LeCanardQuoi commented 3 years ago

It might make more sense to review this enhancement request as part of a plan to relaunch (along with a social distancing capability), given current low app adoption rates, missing field trials at various app adoption rates and poor support by local health authorities, including Alta and BC still showing no sign whatsoever to signing on. In addition, a more contagious new strain may expedite the need to get to a much higher adoption rate sooner rather than later in order to limit casualties. Many countries have had to relaunch, as just reported by the MIT Tech Review. https://www.technologyreview.com/2020/12/23/1015557/covid-apps-contact-tracing-suspended-replaced-or-relaunched

LeCanardQuoi commented 3 years ago

Nearly a month after raising an important performance related app requirement, it should be realized by now that configuring the app with FSA rough regional info is as harmless to personal privacy as configuring province info. Can we then start talking about a "release 2.0" (major release) including major new features, important interface changes and app config info, i.e. FSA info?

CalvinRodo commented 3 years ago

At the moment we have no plans to leverage FSA's in Covid Alert.

LeCanardQuoi commented 3 years ago

At the moment we have no plans to leverage FSA's in Covid Alert.

Does this mean that it is still deemed at CDS, HC and other partners (including the canadian app advisory council) that such an app user config would infringe on personal privacy, regardless of the benefits of getting a much improved handle on the app performance? Decision making process seems very opaque on such an important matter.

And does this also mean that there is no plan in a foreseeable future for a next major release of the app components?

CalvinRodo commented 3 years ago

So I can't speak to discussions between CDS, HC and other partners as I personally haven't been privy to that, I am currently very busy just working on running the Covid Alert backend infrastructure and continuing to do my other tasks as a developer in the Government of Canada.

However in regards to the following statement:

that configuring the app with FSA rough regional info is as harmless to personal privacy as configuring province info.

I've done a bit of analysis on Forward Sorting Areas and I personally disagree that tracking this data is entirely harmless to personal privacy.

In the 2016 Census done by CRA (Link to 2016 population and Dwelling Count) there are 25 FSA's that have a reported population of less then 100 people and 17 of those with 10 or less. Due to that fact I believe it would be possible to determine who in that area either does not have CovidAlert installed or to identify who potentially exposed an individual to COVID-19.

As well FSA's can range in size from less then a kilometer to large portions of Northern Canada covering thousands of square kilometers in size, this fact would reduce the effectiveness of this type of feature.

At the same time due to the entirely voluntary nature of the app and assuming that the FSA is entered by a user of the app we would have no way of verifying if someone entered in the FSA that they actually live in.

This feature also breaks down when you factor in folks moving from one FSA to another, I personally would question the value of this feature when you cannot be sure that folks that entered an FSA are actually staying in that location.

All that to be said that we have higher priority features that we are working on at the moment and as mentioned we currently have no plans to implement this feature. We unfortunately do not have the capacity to make every change recommended to us by outside contributors. Nor do we currently have the capacity to respond with lengthy replies as to why we decided not to implement those features, though we do try to respond as best we can.

As for major releases CDS does iterative changes to the application so I would be hesitant to call any of our changes to the solution major. We are continuously improving the functionality and our priorities are constantly changing based on guidance from Public Health as well as changing science.

LeCanardQuoi commented 3 years ago

One month later, I can only reiterate this lesson from Ebola: "Moving quickly is essential and avoid aiming for perfection before making a move. Perfection is the enemy of the good." Using FSAs as a basis to assess more locally the performance of the app was not ever intended to be a perfect solution, but a most reasonable compromise (-vs- benefits) to meeting the requirement, if it can be agreed that there is such a requirement. A Health Canada spokesperson was cited by ipolitics.ca with: "We are currently exploring options for how we might gather more information on the app’s effectiveness while still respecting privacy”. Rf. https://ipolitics.ca/2021/01/11/two-per-cent-of-canadians-with-covid-have-logged-it-in-contact-tracing-app/ Other medias were similarly regretting the lack of relevant provincial data within the app to assess its performance geographically. Rf. https://www.cbc.ca/news/canada/new-brunswick/covid-alert-app-new-brunswick-atlantic-key-code-positive-notify-1.5868973

Potentially infringing on the personal privacy in 17 FSAs with less than 10 residents seems insufficient a reason to reject proposed scheme, certainly in the absence of a better one more commonly used or understood for instance (note that FSAs are used in the COVID-19 app!). Same for the existence of very large (mostly uninhabited areas), or validity of FSA for the user misconfiguring home base or being mobile. These are most marginal conditions at the edge useful to be aware of in a design review, but certainly not sufficient to justify a blunt reject without further due process. Coarse metrics are being sought here in the absence of most basic ground-truth info. If deemed such an important privacy matter (most unclear), less populated FSAs can simply be omitted in reporting upon when assessing the app performance and still report on the bulk of the population and meet the requirement.

Is an escalation mechanism review to the Canadian app advisory council (and other bodies) really necessary here, given the Ebola reminder, in the absence of their active participation on this github server site, and the absence of core users or any other commentators such as a personal privacy stakeholder?

CalvinRodo commented 3 years ago

I apologize if that came across as a blunt rejection of your proposal, as mentioned that was purely my viewpoint into this specific issue.

I just want to make it clear that I personally don't have the authority to reject or authorize large functional changes to the Covid Alert system as a whole. While I can provide input and insight into how the system works my job is as Acting Product Manager and Technical Lead for the Covid Alert Server. This consists of all of our backend server infrastructure, while I do have delegated technical authority over this portion of the system and the ability to prioritize work in that domain I do not as stated before have authority to dictate the current direction of Covid Alert.

As mentioned we currently don't have enough resources to work on every single issue and must prioritize our work. Unfortunately at this moment we don't have any policy that I'm aware of that outlines our responsibilities regarding responding to contributions from external sources, we also don't currently have a public backlog for all of the systems that consist of Covid Alert.

We currently have a documented Code of Conduct as well as some lightweight Contributor Guidelines.

The increased visibility of Covid Alert from folks that are outside of our organization is I think bringing to light the need to have some more formal guidance and possibly some explicit statements regarding how we plan on responding to these issues in the future.