search

corona-warn-app / cwa-app-android

Native Android app using the Apple/Google exposure notification API. The CWA development ends on May 31, 2023. You still can warn other users until April 30, 2023. More information:
https://coronawarn.app/en/faq/#ramp_down
Apache License 2.0
2.44k stars 495 forks source link

Major data usage (2,25GB mobile data in one month) #4753

Closed dadosch closed 2 years ago

dadosch commented 2 years ago

Avoid duplicates

Technical details

Describe the bug

The CWA used 2.25 GB (!) of mobile data in 28 days. Usage in WiFi is unknown.

Steps to reproduce the issue

Expected behaviour

The data usage should not be this high.

Possible Fix

Additional context

gv-3lbtuRo6e98_MNUuP_g_14331af7bd541d962c415788dbb92e202b14be8e KqZf_em7QwisSN2pbc7_ug_ad2d07a0fafb96dea800381817fd31fe2b378d90

Internal Tracking ID: EXPOSUREAPP-11611

Ein-Tim commented 2 years ago

How often are you connected to the internet via Mobile Data?

dadosch commented 2 years ago

How often are you connected to the internet via Mobile Data?

The phone is connected to the mobile network every workday 10-17 Uhr, the other times to a regular wifi

Ein-Tim commented 2 years ago

@dadosch

Okay, FYI: In general the app is doing exposure checks on mobile data only once a day, see https://www.coronawarn.app/en/faq/#risk_update_24.

It should be investigated if a usage that high is plausible.

MikeMcC399 commented 2 years ago

@dadosch

In case your concern is about charging for the data volume, there is an FAQ article specifically about this point

https://www.coronawarn.app/en/faq/#mobile_data_costs

It is referred to in the link above given by @Ein-Tim.

dadosch commented 2 years ago

@MikeMcC399 @Ein-Tim very interesting. The data usage definitely counted towards the data cap, as the cap was reached and the person had to buy a additional data plan.

I queried the mobile data usage of the app, as reported by dumpsys netstats package de.rki.coronawarnapp (removed all WiFi usages from the output) and made a quick graph showing the data usages and packet numbers.

Beforehand a few things I've noticed: The bytes sent and received are equal, probably an Android issue. (Was an issue with the plot script) The data usage according to the output of dumpsys for the CWA is 3590 MB, so higher than what the system tools says. Might be because of a different method of counting, I don't know.

traffic_cwa

packets_cwa

Only since 15th January: traffic_cwa_january

MikeMcC399 commented 2 years ago

@dadosch

I am surprised about the amount of data. For comparison, on Android 11 with a device which is mostly connected to WiFi (so it is doing checks about every 4 hours) the WiFi data usage by CWA over the past month as shown in the Samsung Android UI is 237MB. The mobile data usage is 164KB.

The second point is about the data being charged against the plan. Who is the mobile network provider? Is it a German network provider?

dadosch commented 2 years ago

@MikeMcC399 the provider is AldiTalk, so E-Plus/Telefonica.

Just one additional note: the storage of the phone is nearly full, this might have something to do with the issue 🤷

EDIT: The issue isn't limited to mobile data. According to the UI a whopping 7.82GB of data has the CWA used since 31.12.2021.

vaubaehn commented 2 years ago

There definitely seems to be an issue. I'm on wify nearly 24/7. I made a rough check:

mobile data - 200 MB used in the past month. Hi data usage started at Jan 20. On 26th alone >80MB have been used, since then only a small amount of KB. The data usage of CWA obviously did not count against my mobile quota (provider vodafone).

wifi data - usage of ~7.2GB in the past month, compared to 171MB in December...

I'm also very low on internal storage, currently around 400-600MB free.

One idea could be, that CWA can't complete some (download) tasks (due to lack of internal space?) and in consequence repeats the downloads of diagnosis keys.

@dadosch for the phone you're taking care of I'd suggest to activate 'mobile Hintergrunddaten einschränken' for CWA until the problem is solved - updates for risk matching via wifi could be enough.

vaubaehn commented 2 years ago

Maybe also associated with #4732 ?

dsarkar commented 2 years ago

Hi @dadosch and everybody! Thanks for report and analysis. Internal Tracking ID: EXPOSUREAPP-11611

Best wishes, DS

Corona-Warn-App Open Source Team

thomasaugsten commented 2 years ago

A error report would be helpful to confirm the theory of redownloads of failed downloads because not enough space is left.

vaubaehn commented 2 years ago

@thomasaugsten in my (and probably all other) case it's problematic, as for good reasons the log-file is stored inside CWA's cache directory which gets wiped when the OS has the feeling it should free some space. The past error reports that I started were hence 'killed' after a couple of hours. I'll try to time error report with the expected downloading window, maybe this can work.

vaubaehn commented 2 years ago

There may also be a timeout problem in the download task, that is connected to the lack of internal storage: my observation is that the whole mobile gets into a state of laggy to unfunctional responsitivity the more free internal storage decreases, possibly because of operations to free memory. So we should look not only to directly memory related hints in error logs, but also download timeouts.

Jo-Achim commented 2 years ago

Just for info: A quick look at my data usage (Samsung Galaxy Note 10, SM-N970F/DS; Android 12 (One UI 4.0) with Android security update: January 01, 2022 (since 3 days); CWA Version: 2.16.2; ENF: 18214815000) shows for the period from 01/01/2022 to 01/29/2022 (today): Mobile data usage: 0 B WiFi data usage (Corona-Warn-App): 224 MB.

178 GB of 256 GB storage free; 3.8 GB of 8 GB RAM free.

vaubaehn commented 2 years ago

@thomasaugsten Are the DiagnosisKeys files downloaded also stored in a cache directory? Because this could mean, CWA is downloading everytime all files instead of just hourly packages when cache is cleared by OS on a regular basis due to low memory.

DerVogel2020 commented 2 years ago

Just for info: A quick look at my data usage (Samsung Galaxy Note 10, SM-N970F/DS; Android 12 (One UI 4.0) with Android security update: January 01, 2022 (since 3 days); CWA Version: 2.16.2; ENF: 18214815000) shows for the period from 01/01/2022 to 01/29/2022 (today): Mobile data usage: 0 B WiFi data usage (Corona-Warn-App): 224 MB.

178 GB of 256 GB storage free; 3.8 GB of 8 GB RAM free.

My data usage is nearly the same (Android 8, Galaxy A5 (2017).

So this is not a general problem.

vaubaehn commented 2 years ago

@thomasaugsten error log started somewhat after 12:00 UTC, download should have happened around 12:22 UTC, CWA received EW and did risk calculation around 12:33 UTC. Looks like everything worked fine, so I'm not sure if you can find something. Did some disk cleaning last night, since then there's 600-700MB internal storage free. I assume the threshold when OS jumps in to do cache wiping is around 500MB free memory. Error log: 2022.01.29 - 14:44:28 ID B3FB247BF67E4D7C90FC

vaubaehn commented 2 years ago

Additional note: before the last risk check, the wifi data usage counter for CWA in January was 7.311MB, now it's 7.312... Don't know how much in size is an hourly DK package currently.

vaubaehn commented 2 years ago

@thomasaugsten After I sent the error log for the successful risk check in background, meanwhile there had been another one. This time 58MB of data had been downloaded, what looks suspicious to me. However, I can't send an error log now, because obviously OS again purged the cache directories, so error logging had been stopped. Free internal storage is 580MB currently.

Looks like, it's hard to impossible to log memory issues.

I was once discussing with @d4rken , whether the error log could be stored inside the data directory, but he explained good reasons not to do so: when something goes wrong with error logging (continous crashes of CWA) and the error log is inside the app's data directory, there's no way to recover than erasing CWA's data via system settings what may be hard for some users. Beside this, erasing CWA data also clears collected RPIs due to Google's data privacy policy. Third reason is, that when users enable error log and forget to disable it after a while, internal storage can run full and OS can't do anything about it (i.e. freeing storage by clearing cache).

So I think there's not much that can be done to clearly diagnose the root cause. It would need a fix by altering the download logic (and in parallel maybe changing the logic of providing DKs to ENS like I proposed elsewhere: https://github.com/corona-warn-app/cwa-app-android/issues/4732#issuecomment-1021095635) just by relying on the symptoms.

Edits: correcting words printed from my mobile's insane auto-correction

vaubaehn commented 2 years ago

Since my last comment 2 days ago, 600MB have been downloaded by CWA. That would sum up to 9GB per month.

Let's say we have around 15.000.000 active Android users, and 10 percent have slow/old mobiles that may be affected. Then the affected users currently contribute to 13.500.000GB per month in the overall traffic. (=13.5 Petabyte/month)

Jo-Achim commented 2 years ago

Just for info: A quick look at my data usage (Samsung Galaxy Note 10, SM-N970F/DS; Android 12 (One UI 4.0) with Android security update: January 01, 2022 (since 3 days); CWA Version: 2.16.2; ENF: 18214815000) shows for the period from 01/01/2022 to 01/29/2022 (today): Mobile data usage: 0 B WiFi data usage (Corona-Warn-App): 224 MB.

178 GB of 256 GB storage free; 3.8 GB of 8 GB RAM free.

Today (01/31/2022): Mobile data: 0 B WLAN (Corona-Warn-App): 234 MB.

176 GB of 256 GB storage free; 3.7 GB of 8 GB RAM free.

Note: with very few contacts to outside.

MikeMcC399 commented 2 years ago

@vaubaehn

Are you running Android 7.0 (API24) like @dadosch? The issue may be API level dependent. My oldest compatible device is Android 8.0 (API26) (same device as @DerVogel2020) and I am not seeing any high network usage on that device.

vaubaehn commented 2 years ago

@MikeMcC399

Thanks for your good idea. I'm running on Android 6. My thought is, that it's indirectly dependent from API level: the lower the API level, the older the mobile, the slower the system, the more limited the storage. System speed and free storage might contribute to this issue, but for sure I would not exclude anything that is directly related to API level. If we could break it clearly down by API level, then it would need to be API level <=24.

@Jo-Achim Thanks for your feedback - did I get it right, in the past two days your CWA downloaded 10MB? Then the issue contributes by factor 60 to increased traffic.

vaubaehn commented 2 years ago

@mlenkeit and @thomasaugsten In case that can get fixed somehow, this might probably also justify a forced-update like suggested in https://github.com/corona-warn-app/cwa-documentation/issues/732, as this likely reduces substantial costs for bandwidth and traffic.

Jo-Achim commented 2 years ago

@Jo-Achim Thanks for your feedback - did I get it right, in the past two days your CWA downloaded 10MB? Then the issue contributes by factor 60 to increased traffic.

Yes, that's right: 10 MB in the last two days. (Or 234 MB from 01/01/2022 until 01/31/2022.)

MikeMcC399 commented 2 years ago

@vaubaehn The module DeviceStorage.kt treats anything older than API26 (8.0 Oreo) as legacy and runs differently. That is to say API25 (7.x Nougat) and earlier are handled as legacy. You have said that you are running Android 6, which is API 23 (Marshmallow) and therefore is also handled as legacy.

vaubaehn commented 2 years ago

@MikeMcC399 Good catch! Just from a fast screening I'd say it's about how APIs allocate necessary storage. Maybe in allocation something doesn't work as expected? Definetly something the devs should have a look into. I'll try to find something in my error logs with keyword "requestStorage".

MikeMcC399 commented 2 years ago

@vaubaehn Have you updated to the latest version 2.16.2 or are you still on an old version?

vaubaehn commented 2 years ago

I'm still on 2.6.1 for some reasons... Still waiting for the forced-update. Have the storage allocation functions been introduced in a later version?

Edit: ~yes, they have. 2.6.1 was still "legacy mode" obviously...~ new 2.16.2 and old 2.6.1 seem to be identical

dadosch commented 2 years ago

A question for the people operating the backend: Do you see any increased traffic size? Depending on the amount of legacy devices out there this should probably be visible…

MikeMcC399 commented 2 years ago

@vaubaehn The 2.6 version you are using was released in July 2021 and your test results may not be representative of the current version. (The one module which I mentioned was last changed in April 2021, however so far there is no proof that this module is involved in the root cause of the high data volume issue.)

vaubaehn commented 2 years ago

@dadosch I pinged one dev from the server department 30 minutes before you asked, let's see if he responds.

@MikeMcC399 What do you mean by test results? The amount of data downloaded in one month (>7GB)? This fits perfectly with results observed by @dadosch for WiFi data on CWA 2.16,2 - although the two of us are a small sample size and not representative for the proportion of possibly affected devices. - What is a bit tricky with this issue: I would not have recognized it as such, if I had not read @dadosch 's issue. That let me have a look and finding out similar issue for me. Likely only a few of affected users are aware of it (if nearly all data goes via WiFi).

By the way, the log message in this block seems to be wrong, https://github.com/corona-warn-app/cwa-app-android/blob/658f702021dfce94278be40fdf9c43f386574ec3/Corona-Warn-App/src/main/java/de/rki/coronawarnapp/storage/DeviceStorage.kt#L70-#L75 it should be "requestStorageLegacy(path=%s, requiredBytes=%d)",

vaubaehn commented 2 years ago

A question for the people operating the backend: Do you see any increased traffic size? Depending on the amount of legacy devices out there this should probably be visible…

@dadosch related parties are working on the issue: https://github.com/corona-warn-app/cwa-server/pull/1703#discussion_r795977457

MikeMcC399 commented 2 years ago

@vaubaehn

By the way, the log message in this block seems to be wrong, https://github.com/corona-warn-app/cwa-app-android/blob/658f702021dfce94278be40fdf9c43f386574ec3/Corona-Warn-App/src/main/java/de/rki/coronawarnapp/storage/DeviceStorage.kt#L70-#L75 it should be "requestStorageLegacy(path=%s, requiredBytes=%d)",

Will you open an new issue and / or submit a PR to correct the text of the log message?

vaubaehn commented 2 years ago

Shifting discussion from there (https://github.com/corona-warn-app/cwa-server/pull/1703#discussion_r798318349) to here:

@mlenkeit

@vaubaehn there are ways to download files only if they are different from what the client downloaded last time (i.e. e-tag, if-none-match header, etc.).

That was actually a way what I was thinking about and thus pointing my question to @hilmarf . Logging the hashes into a CWA-internal db as a control mechanism could then prevent superflous downloads of keys as well as non-necessary key provisions to ENF (as they're obviously not matched anymore by ENF due to a kind of suspected logging mechanism - but the inspection/file operations on provided key files need much time on older phones like mine thus contributing to timeouts in the workflow).

However, CWA is using other mechanisms that ensure that files are not repeatedly downloaded.

If you have time, in one sentence, could you elaborate?

We are stills struggling to understand why these repeated downloads happen on some devices and I'm not yet convinced that changing the caching approach in CWA will fix it.

I can share my observations: on older phones with less internal storage, the OS is wiping all app caches frequently to free some space. Hence DiagnosisKey-files are deleted (was confirmed by Fynn Godau in an internal chat, you might confirm by performing your own simple experiments), and if CWA's mechanism to prevent repeated downloads is not functionally here, it may lead to the observed problem. What might contribute to the problem of low memory is, that there might be a bug in ENF (https://github.com/corona-warn-app/cwa-app-android/issues/4732), but that's a weak suspect currently. Other observations indicate that former DK-RPI matches ("risk contacts") are cached by ENF and always returned by getExposureWindows() during the 14/15-day validity period. So it was a win on performance to only provide DKs to ENF that not have been submitted before. To confirm this behaviour, it would need more effort to experiment, as it either needs a whitelisted account or a rooted OS to inject crafted RPIs with matching TEKs (DKs) into ENF. If my theory applies to reality, this possibly also mitigates issues where risk detection stops on older phomes (due to CWA and ENF timeouts) when incidence rates are high (resp. larger DK packages).

Anyway, this PR here was not motivated by the repeated downloads but rather by the size of recent diagnosis key packages in general.

Yes, I got this from the beginning, but I wanted to share and exchange my thoughts also with @hilmarf , as I assumed he may contribute to a solution with his professional expertise. I hope everyone is fine with that, but we may also shift further discussion to the related Android issue.

PS: error logging/report is not effective to detect/confirm this issue here, as the error log itself is stored in the app's cache directory, hence purged together with DK-files when OS wipes all app caches - I was already explaining this problem to Thomas. This might be the reason why it's diffcult to find out the root cause, as no error log may detect the problem...

vaubaehn commented 2 years ago

@vaubaehn

By the way, the log message in this block seems to be wrong, https://github.com/corona-warn-app/cwa-app-android/blob/658f702021dfce94278be40fdf9c43f386574ec3/Corona-Warn-App/src/main/java/de/rki/coronawarnapp/storage/DeviceStorage.kt#L70-#L75 it should be "requestStorageLegacy(path=%s, requiredBytes=%d)",

Will you open an new issue and / or submit a PR to correct the text of the log message?

@MikeMcC399 I will do at a later time.

vaubaehn commented 2 years ago

@dsarkar The WiFi data usage for the time frame Januar 10 to February 7 is now approx 13 GB for my phone due to repeated downloads.

Please investigate.

dsarkar commented 2 years ago

@vaubaehn Forwarded ...

vaubaehn commented 2 years ago

@dsarkar Thanks...

vaubaehn commented 2 years ago

Good morning @mlenkeit , @thomasaugsten and @hilmarf ,

I now have an indirect proof, that the high mobile/wifi data usage is caused due to interaction between low storage on device, Google's ENS data storing policy and an unfortunate handling of processing Diagnosis Keys packages by CWA.

As written above, CWA submits all available Diagnosis Keys packages (past daily packages and current hourly packages) via provideDiagnosisKeys() to ENS in each API call. ENS is storing all provided Diagnosis Keys for each call in a system folder for the ENS for about 3-4 days. As each API call provides packages up to 100MB (sizes from 6 days ago), with up to 6 calls per day, the persisted data in ENS could reach between 1.8 and 2.4GB internal storage (e.g., https://github.com/corona-warn-app/cwa-app-android/issues/4732#issuecomment-1040081787). Todays's sizes may be smaller due to decreasing incidence rates (didn't measure today).

As many old phones have very limited storage capacities, the internal storage may run full quite fast due to that mechanisms. In case Android OS detects free internal storage <500MB, it automatically purges caches of all apps. For CWA it means, that already downloaded Diagnosis Keys packages that have been stored inside the app's cache directory get cleared, and so that packages are completely downloaded again before the next ENS API call. About one week ago, that could mean a download volume of 600MB per day, when the cache had been cleared multiple times per day. In my case, the download volume was about 13GB per one month.

As my phone was rendered nearly unfunctional due to lack of free internal storage, I decided to give CWA's fork CCTG a trial. From discussions with @fynngodau and from a short peek into the microG code, and also after some information of @Ein-Tim it was clear, that microG does not store all provided Diagnosis Keys packages in their .bin files. Instead, after each API call with provideDiagnosisKeys() the TEKs are inserted into a sql db and stored there for further processing. There are no redundant doublettes what keeps necessary storage space in a minimum. I turned off ENS but did not clear data in Google Play Services, and then installed CCTG. In the first day, I still had low storage (as DK packages are still stored in ENS), so Android OS was still clearing all app caches regularly. As CCTG also stores DK packages into the app's cache directory and uses the same provideDiagnosisKeys() call like CWA, the clearing of app cache caused CCTG to download Diagnosis Keys packages repeatedly, which caused a download volume of >300MB for that day. During the next night, ENS cleared some storage of older Diagnosis Keys .bin files what released ~500MB. That was enough free space to stop automatic cache clearing by Andorid OS. Since then, CCTG is only downloading the hourly Diagnosis Keys packages, what contributes to <10MB of daily download volume.

As written in another comment, it's unfortunately not possible to capture these problems in an error log, because the error log is stored inside CWA's cache directory. Because of this, the error log gets deleted every time Android OS is purging the app caches. And if there is continously enough free space in the system (>500MB) the problem wouldn't occur.

Possible solution to decrease TSI's bandwidth for Diagnosis Keys downloads:

=> the only question is: will it be enough to provide each .bin file only once? Google says: Description of provideDiagnosisKeys():

[...] When using the ExposureWindow mode, all provided keys are accumulated under the ExposureWindows. [...]

https://developers.google.com/android/exposure-notifications/exposure-notifications-api#methods

and

Description of getDialySummaries() (ExposureWindow mode):

[...] This function returns a list of DailySummary objects for the last 14 days (or less, if specified in the DailySummariesConfig). [..]

https://developers.google.com/android/exposure-notifications/exposure-notifications-api#methods

If taken these two descriptions together, we may assume that all exposures are stored for 14 days in ENS and can be retrieved by CWA for that time period. Unfortunately there is no clear statement, whether exposures of 14 days are also returned by getExposureWindows(). But logically, when there is no token used (like recommended!) in provideDiagnosisKeys(), or the token stays the same which each call, it can be assumed that the list of retrieved Exposure Windows contains all exposure windows from the last 14 days.

So, validating assumptions above by experiments (or simply asking Google directly) and a small refactoring of the DK provisioning can solve many different problems.

EDIT: Please see also following comment for necessary but likely small adaption of the server.

These are the moments, when I'm missing passionate team members like @d4rken with who I was discussing many of these aspects on GitHub in the last year. But maybe you can convince the POs to put this issue with high priority into the backlog and someone else can take care of it.

-- Thanks to @fynngodau , @mh- and @Ein-Tim for discussions, and to @darkmattercoder , @reinob , @6e6e58c4 , @klic , and @kille for providing valuable information by looking into root:/data/data/com.google.android.gms/app_en_diagnosis_keys on their rooted phones. FYI: @dsarkar

vaubaehn commented 2 years ago

... for the proposed solution

download from server and provide each .bin file to ENS only once. This would mean that CWA needs to log the eTag of the .bin file (stored in a db inside CWA's data directory, not cache directory), and only download and provide .bin files that not have been downloaded and provided before.

I forgot one thing: to make this work smoothly, we would need to make hourly Diagnosis Keys packages available for all the past period (currently 10 days), not only for today and yesterday. So in case a phone was off internet for a longer time, the single missing packages can be picked and downloaded. Daily packages then are necessary for backwards compatibility only. As far as I remember, persisting hourly packages on the server for a longer period than 2 days only needs a change of one parameter in the server config file, so it should be minimum effort at that place.

vaubaehn commented 2 years ago

P.S.: @mlenkeit To cite you from above:

We are stills struggling to understand why these repeated downloads happen on some devices and I'm not yet convinced that changing the caching approach in CWA will fix it.

To reproduce the problem/to find out how low storage/caching affects repeated downloads you may try the following steps:

  1. Take a phone with plenty of free internal storage where CWA is active for at least a week. The problematic behavior is reliably observable with Android 6, but all other versions may be affected similarly.
  2. Go to settings and check how much wifi/mobile data was used by CWA until today, calculate the daily average. Write down both numbers.
  3. Important: First clear all app caches! Now fill up the internal storage of your phone with arbitrary files, until there is only 400-500MB free storage left.
  4. Leave CWA/the phone alone for 24 hours, but do some typical user activity like internet browsing (e.g., chrome) from time to time (this fills up cache again). Avoid time travelling! Otherwise ensure, that after time travel you wait until Android cleared app caches before CWA downloads DKs. You may also clear app cache manually and force CWA to start the download after.
  5. Go back to settings and check how much wifi/mobile data has been used during the last 24hrs. Compare to the daily average in step 2.

To reproduce how much CWA's provideDiagnosisKeys() mechanism effects internal storage in ENS, you may try the following:

  1. Take the same phone in the state of step 5 from the previous experiment.
  2. Go to settings > apps > Google Play Services and check how much storage is reserved for "data".
  3. Disable ENS via CWA or Google Account settings.
  4. Leave phone alone for 24 hrs. See hint regarding time travelling above.
  5. Repeat step 2.
  6. Repeat step 4.
  7. Repeat step 2.
  8. Re-Enable ENS.
  9. If you now check data usage of mobile/wifi, you should be able to observe the same quota like in the previous experiment step 2.
  10. Repeat step 2 (in this experiment).
  11. Repeat step 4.
  12. Repeat step 2.
  13. Repeat step 4.
  14. Repeat step 2. Now the free internal storage should be somewhere be between 450 to 700MB - as Android OS tries to wipe app caches as often as possible...
  15. If you now leave the phone again alone for some time (>24hrs) you will notice, that the mobile/wifi data usage has increased again, probably around factor 60.

Hope that helps.

Edit: added an important move to step 3 in the first experiment. Edit2: added an important note to avoid time travelling. Problem is reliably observable with Android 6, but other versions may be affected similarly.

dsarkar commented 2 years ago

@vaubaehn Many thanks for the analysis, all forwarded to the internal ticket.

vaubaehn commented 2 years ago

Hi @thomasaugsten , @GisoSchroederSAP and @dsarkar ,

in this issue here it was reported that CWA causes a very high internet data usage on single devices, mobile data (>2.2GB) as well as wifi (>13GB), and in some cases CWA's mobile quota counted agains the users' data plans. I just found a recent report in Google Play Store, where the data usage lead to limited connection speed after the data plan had finished:

In this issue, we analized, that an interaction between CWA's mechanism of caching Diagnosis Keys packages, low internal storage on the phone and Android's automatic cache cleaning may cause repeated downloads of Diagnosis Keys from TSI's servers. The risk of low storage (hence repeated downloads of Diagnosis Keys) is increased due to the way how CWA provides Diagnosis Keys to the ENS (with each call, all available packages of the past 10 days plus the hourly packages of today are provided) and ENS storing policy of provided .bin files (for each call, all provided .bin files are stored for 3-4 days), as the storage consumption can currently contribute with ~1.5GB.

We provided a guide on how to reproduce the problematic behavior, and we proposed some ways on how to mitigate the issue.

What we forgot is to discuss some legal aspects: https://www.coronawarn.app/en/faq/#mobile_data_costs claims that

[...] Mobile data that is transmitted via the Corona-Warn-App when diagnosis keys are uploaded or when the server notifies you in case of exposure is free of charge and will not be deducted from your data plan. The zero rating applies to all SIM cards that are distributed by German network providers. [...]

According to the report in the issue here and on Google Play Store, that claim is not completely true. Obviously, there may be some exceptions from some providers in general, or the providers did not guarantee a "full flatrate for CWA data", but they technically implemented a limited zero-rating for the IP addresses used by CWA per device. The limitation might have been set on base of practical estimations (e.g., 1GB per month per device or even less) some time ago. The problem is, that the publisher of the CWA likely does not know that at least CWA's FAQ does claim characteristics about the zero-rating that are not true in their completeness due to the unfortunate interaction between CWA, Android and ENS. There may have been similar statements published in other places as well. I would not dare to exclude that this issue may have some legal implications.

I therefore strongly recommend to inform the publisher accordingly and change the FAQ temporarily until the issue has been fixed/enhanced. The FAQ could state that exceptions from the zero-rating might occur currently for some devices and that steps are taken to ensure a full zero-rating again. The paragraph of https://www.coronawarn.app/en/faq/#mobile_data_costs could state that, "as long as more than 500MB internal storage is continously available on your device 4 days after the installation of CWA, the zero-rating most likely will be in effect entirely". I also recommend to supplement paragraphs about the pre-requisites of the installation of CWA accordingly:

https://www.coronawarn.app/en/faq/#in_short

How do I ensure error-free use? To ensure error-free use of the Corona-Warn-App as part of contact tracking, always use the latest version of the app and keep your device's operating system up to date. In the settings of your device, activate the background update, Bluetooth and for Android devices up to and including Android 10 also the location detection. Then, within the Corona-Warn-App, enable risk detection and carry your device with you when you leave the house! Open the app once a day to check for regular risk status updates. To limit downloads of data for risk detection to the minimum necessary, we kindly ask you to ensure that after the installation of Corona-Warn-App the internal storage of your **Android device** continously provides significantly more than 500MB free space.

https://www.coronawarn.app/en/faq/#minimum_requirements

What are the minimum operating system requirements for my phone? UPDATE February 10, 2021 iOS - Minimum operating system requirements The minimum operating system level for iPhone 5s, iPhone 6 and iPhone 6 Plus devices is iOS 12.5. For iPhone SE (1st generation), iPhone 6s, iPhone 6s Plus and newer iOS devices the minimum is iOS 13.7. These versions of the iOS operating system include the necessary v2 version of the Apple Exposure Notification Framework. See also [Apple/iOS]: Why is iOS 13.7 a minimum requirement? and the blog articles Telekom and SAP to make Corona-Warn-App available for iOS 12.5. and Corona-Warn-App version 1.12 comes with two new features Android - Minimum operating system requirements The minimum operating system level for all Android devices is version 6 of Android. In addition, the device must have Google Play services installed, with a minimum version 1.5 of the Google Exposure Notification System (ENS). See [Google/Android]: Which version of the COVID-19 Exposure Notification System is currently installed? for instructions about how to find out which version of ENS is installed. Before the installation, please ensure that there is at least 2GB free internal storage left.

The suggested 2GB calculate as follows: ~100MB for CWA including all data downloaded subsequently and stored inside CWA's data directory and cache directory + 1.4GB for temporarily stored .bin files inside the ENS provided via provideDiagnosisKeys() + 500MB buffer necessary to prevent Android OS from automatic app cache clearing.

Please provide information until the end of the week, on how you will further proceed on this issue, in terms of

If we won't hear anything from you until Friday evening, it may be the best, that community informs publisher directly next week. as well as public media, to prevent users to get in trouble with their mobile data plans.

Thank you in advance!

dsarkar commented 2 years ago

@vaubaehn Thanks, we will have a look at this

MikeMcC399 commented 2 years ago
MikeMcC399 commented 2 years ago

The app gives no choice about using shared European diagnosis keys. Would it help users to be able to choose between German-only and European exposure checks, if their device is low on free storage space?

vaubaehn commented 2 years ago

The app gives no choice about using shared European diagnosis keys. Would it help users to be able to choose between German-only and European exposure checks, if their device is low on free storage space?

That is an interesting idea. I calculated some data: DK packages from 2022-02-14 to 2022-02-23 sum up to 46.1MB for EUR and 39.9MB for DE. Daily packages together with all hourly packages of today until midnight, we may estimate a data load of 50.71MB for EUR and 43.89MB for DE. When for 3 days there are 18 calls (6 per day) for provideDiagnosisKeys(), .bin files would sum up to 913MB for EUR and 790MB for DE. For some devices the difference of ~120MB may already make a difference, but 120MB is not that much in all. The avarage proportion of DE Diagnosis Keys to EUR DKs in the past 10 days is ~86%. I don't know how much is the impact of Austria that will stop their service end of this month, and whether there are other countries quitting contact tracing which could decrease the number of DKs additionally.

However, the app would need to be adapted to leave the user the choice to download the type of DK packages (EUR/DE). For some portion of the user population it may not be easy to take a decision about it. Also, when changing between EUR and DE, there will be a delay of up to 3 days (storage time in ENS) until an effect could be visible, but the effect will be under the radar of most of the users likely. But for TSI it could be a gain of overall bandwidth for their servers, hence reduced costs for TSI and mobile providers.

Jo-Achim commented 2 years ago

But for TSI it could be a gain of overall bandwidth for their servers, hence reduced costs for TSI and mobile providers.

Even if it were a gain for TSI in overall bandwidth for their servers and lower costs for wireless carriers, I don't think the idea is viable. At least as long as the savings are not actually quantified.

In addition, the CWA represents a European solution in the sense that diagnosis keys are also exchanged across borders, which I think is very important given the increasing amount of travel. (I myself recently brought a red warning from Italy.) And explaining to a CWA user that he should think about changing his CWA configuration before and after traveling abroad, ... if this cannot be automated, I do not think it is very realistic or practicable.

Of course it would be nice if we didn't need an app like the CWA anymore, but I'm not sure yet - next winter will definitely come. And the virus still has plenty of time and opportunities to mutate.

vaubaehn commented 2 years ago

Hi @hilmarf , there are some not so good news with impact on bandwidth of your servers that I'd like to share with you. According to this simulation study: https://depositonce.tu-berlin.de/bitstream/11303/16461/4/2022-02-23_MODUS-COVID_Bericht.pdf we can expect a rise of incidence rates again starting from end of February due to the spread of Omicron subtype BA.2. From the modeled data, the best case scenario predicts incidence rates that will stay somewhat below the rates that we had two to three weeks before, or they will be similar. Hence, the server infrastructure should handle this case well, as it did in the past weeks. But now for the worst case secenario: this model predicts incidence rates of factor 2.5 within a few weeks. What does this mean for the server resources?

With Omicron subtype BA.1 we had Dialy Diagnosis Keys packages up to a size of 10MB. With factor 2.5 for BA.2 worst case scenario, Dialy Diagnosis Keys package size would increase to 25MB. With the current storage policy of Google's ENS and CWA's caching/providing mechanism, the storage for .bin files in ENS may increase to 3 days x 6 calls/day x 25MB per daily packages x 10 daily packages per call = 4.5GB. To prevent Android from automatic cache cleaning, Android mobiles would need free internal storage of >5GB. If the storage with the predicted incidence rates is less, it would mean that a single Android device will download Diagnosis Keys packages up to 6 times per day (on wifi), with 250MB per download and an overall download size of 1.5GB per day. Calculated for the whole month, a single affected device then would download 45GB per month. For a device that is solely on mobile, it would just be 250MB per day and hence 7.5GB per month. A critical question is now, how many devices could be affected? Let's assume, that the free internal storage on all Android devices follows the normal distribution (Gauss...). We don't know what is the average free internal storage of all Android devices out there, so we can only guess, what proportion of devices has less than 5GB free internal storage left. What it be ok to estimate, that around 20% of Android devices have less than 5GB free? Then we may estimate that CWA currently has 15.000.000 active Android users. That result in ~3.000.000 Android devices with less than 5GB free internal storage. If they were all on wifi, in a whole month in the worst case scenario, they would download 3.000.000 x 45GB per month which is roughly 130 Petabyte/month. The real life sizes would be smaller, as many devices are also running on mobile data, thus downloading less often. All other devices with >5GB free internal storage would only download 25MB per day, which is 750MB per month. Means, 12.000.000 devices x 0,75GB per month are roughly downloading 8.6 Petabyte/month on wifi. If there was no issue with Android devices with <5GB internal storage, monthly downloads were about 11 Petabyte for all Android devices.

So if the worst case scenario of increasing incidence rates due to Omicorn BA.2 came true, and the issue reported here was not fixed before, there is a chance that you would need to prepare servers for download rates of ~140 Petabytes for the Android devices, plus what is necessary for the iOS devices.

Hope that estimations can help you for your planning and preparations.