App does not work if the device UTC time differ more than 2h from the correct UTC time

[Tho-Mat]

Describe the bug

According to: https://github.com/corona-warn-app/cwa-server/issues/723 @michael-burwig reports: "We do not check whether an end device has its time set correctly."

As far as I understand: The correct UTC-time of a device is very important. If the times of two meeting devices defer more than 2h, i.e. |t1-t2|>2h then no key-match will be found, if one of them reports his/her/its infection. Moreover: if the device of the infected user with an UTC-time & date ti that defers more 2h from the correct UTC-time tu, i.e. |ti-tu|>2h, then all of the transmitted keys are invalid / useless for others. On the other hand, if the device time of an CWA user is wrong, he will receive the keys but no match will be found, also. Note: Even the device shows the correct time and date the UTC-time may be wrong.

I assume IOS and Android devices are effected?

Expected behaviour

The user-set device time should not be important for value matching. I could not find any information about that.

Possible Workaround

As long as the OS uses the (user-set) device time and not the correct time (from internet, GPS or mobile network) to store the received BT-keys and their own active day-keys, the CWA app should check if the device time is set correct. The user should be informed that he has to set his device-time to the correct value to ensure the app will work. This could be done by calling an external time server or a one to be implemented on the CWA server.

Additional context

https://blog.google/documents/69/Exposure_Notification_-_Cryptography_Specification_v1.2.1.pdf page 9 https://app.slack.com/client/T019BKJF80K/C0194ML0MLN/thread/C0194ML0MLN-1598688186.242400 Thanks to Michael Huebler and Paul-Olivier Dehaye

---

Internal Tracking ID: EXPOSUREAPP-2591

[daimpi]

Somewhat related: https://github.com/corona-warn-app/cwa-wishlist/issues/88

If this issue is affecting both iOS and Android it should probably be moved to the documentation repo.

[Marco2907]

Hi @Tho-Mat, thank s for sharing your content. We will keep and discuss this point with our development team and will come back to you as soon as possible.

Thank you.

Best, MP

---

Corona-Warn-App Open Source Team

[Marco2907]

Hello @Tho-Mat Hello @daimpi , if both of you agree I would close this issue here because this topic is similar to corona-warn-app/cwa-wishlist#88 . Based on corona-warn-app/cwa-wishlist#88 the community discuss this with our development team and I will come back to you as soon as possible.

Thank you.

Best, MP

Corona-Warn-App Open Source Team

[Tho-Mat]

@Marco2907 I do not agree. If time is not correct the app is useless. So it's not a wish, its the same as if BT is off. So this should result in: Risikoermittlung nicht aktiv

[daimpi]

I also don't think closing this issue is right… https://github.com/corona-warn-app/cwa-wishlist/issues/88 is about a quite different issue, which is only related in so far as that both issues concern the time-setting in your phone…

If anything I think this issue here should probably be moved to the documentation repo, as it affects both iOS and Android.

[Marco2907]

Hi @daimpi , sorry for the late response and misunderstanding, I will keep this issue to discuss with our development team. I will come back to you as soon as possible.

Thanks, MP

Corona-Warn-App Open Source Team

---

Internal Tracking ID: EXPOSUREAPP-2591

[Tho-Mat]

This may be the next wrong time issue https://github.com/corona-warn-app/cwa-server/issues/764#issuecomment-693209906

[Tho-Mat]

At the moment 2 of 4100 users that submit their keys seem to use the wrong time. If i assume the same rate for the 18.000.000 users that download the app, we get round about 8800 users, that do not know, the app is useless for them.

[Tho-Mat]

And again https://github.com/corona-warn-app/cwa-server/issues/764#issuecomment-695217330

So, if the assumption of wrong date/time is right, this should be fixed

soon.

[Tho-Mat]

and again... https://github.com/corona-warn-app/cwa-server/issues/764#issuecomment-695956079

[Tho-Mat]

@JoachimFritsch so at the moment we have 5 of 4871 users with wrong time. That means 1‰ or 18000 of 18.000.000 with useless app

[Tho-Mat]

and another 2020-09-21-hour-09.zip

[mlenkeit]

Hi @Tho-Mat ,

thanks for raising this. We are currently looking into this with CWA Engineering and our partners in order to better understand the impact of this.

As you already suggested, we could interact with an external time server to check whether system time is off by more than e.g. 2 hours. While this sounds simple, integrating external services requires lots of iterations and approvals, depending on their service agreement.

Alternatively, we could share the server time with the client, for example in an HTTP header, but this also comes with the challenge of accuracy on slow network, etc.

My assumption is that in the end, we might only be able to inform users about the side effects of incorrect time settings and the risks associated with this.

We’ll keep you posted as we proceed in the discussions.

Oh, and btw, I think 5 of 4871 is rather 0,1% than 1% 😉

[Tho-Mat]

@mlenkeit

Oh, and btw, I think 5 of 4871 is rather 0,1% than 1% 😉 forgot ‰ sorry :-), corrected.

On the other hand with the missing 6 only those users are counted that have a time far in the future. (1/2 to one day.) All users, that have a time in the past also could not be found by this method. So at least the number has to be doubled.

One easy solution would be to transmit the device time with the key-upload. That would be an easy method to find wrong time devices.

Theoretically the server then could refuse the data, if device time is incorrect. Correction of the time is not possible, since you do not know the start-time of the time-difference. But better take the keys, even they do not work.

What about: ntp1.t-online.de "service agreement" should be included?

Time-share sounds nice for the moment. Accuracy on slow networks is not a problem, since we have a time-period of 2 hours.

Best solution could only be made by Google/Apple, if they save the correct time and not the device time. Are you in contact with them about this issue?

But at the moment we do not have this, so a solution has to be found with the things we can make ourselves.

[Tho-Mat]

@mlenkeit

some new sets arrived at 2020-09-21-hour-17 2020-09-21-hour-20 2020-09-22-hour-00 so in the last 24h at least 5 of 175 users are assumed to have the wrong time.

That looks to much. Maybe the assumption is not correct. Since only the sets with keys in the far future could be identified, there could be 10 or more with the wrong time.

[Tho-Mat]

don't understand me wrong. The time issue has to be solved. But the reason of the missing 6 could be something else. The first time the missing 6 occurred was the 22.08.2020. But since then nothing important happend on the client side.

With https://github.com/corona-warn-app/cwa-server/issues/764 the server crew tries to find out more details. but on the client side we only talk.

What are the plans to solve this? "To be reviewed" should be changed to "in progress" or "milestone".

[ndegendogo]

A question on this missing key. Do we know if it was even sent to the server? Does the App perform some sanity checks or filtering on the keys retrieved from the ENF? Have you verified that ENF does not suppress this key? And possibly the behavior differs here between Google and Apple implementation of ENF? and/or between the many iOS versions available meanwhile, including the betas?

[Tho-Mat]

@ndegendogo Good questions. The first may be answered after 30.09 when the new server version is published.

[Tho-Mat]

again we got the issue on: 2020-09-22-hour-08 2020-09-22-hour-15 2020-09-23-hour-03

Does anybody know when the first IOS14.0 beta was released, that supports cwa? The massive increase in the issue count may be related to IOS14?

[daimpi]

Does anybody know when the first IOS14.0 beta was released, that supports cwa?

Beta 4 was the first iOS 14 version which supported ENF afaik. I think it was released around the time when I made this post to update the respective FAQ entry (4th August).

[ndegendogo]

The massive increase in the issue count may be related to IOS14?

Or, possibly to Apple's new implementation of the ENF that supports the API version 2. That would be since iOS 13.7.

[GisoSchroederSAP]

This topic is now getting attention when we move this into the development planning. As of today, I do not have any date or release that I can share yet. However, the number of links to other time-related issues gives a good indication about the importance - and now needs to be converted into priority. I'll keep you posted.

[Tho-Mat]

Update for the missing 6 "Risk 6 key is missing in reported keys" https://github.com/corona-warn-app/cwa-server/issues/723 I report here since the issue is closed.

I now believe this is NOT time related.

Too many of them occur in the hour files the last days: 2020-09-24 08:00 1 User 2020-09-24 09:00 2 User 2020-09-24 11:00 3 User 2020-09-24 12:00 2 User 2020-09-24 13:00 1 User 2020-09-24 14:00 1 User 2020-09-24 15:00 3 User 2020-09-24 16:00 2 User 2020-09-24 17:00 1 User 2020-09-24 18:00 2 User 2020-09-24 19:00 1 User 2020-09-25 07:00 4 User 2020-09-25 08:00 2 User 2020-09-25 11:00 1 User 2020-09-25 13:00 3 User 2020-09-25 14:00 1 User 2020-09-25 15:00 2 User 2020-09-25 16:00 1 User 2020-09-25 17:00 1 User 2020-09-26 06:00 1 User 2020-09-26 07:00 1 User 2020-09-26 08:00 9 User 2020-09-26 09:00 3 User 2020-09-26 10:00 4 User 2020-09-26 11:00 3 User 2020-09-26 12:00 3 User 2020-09-26 13:00 10 User

so at the moment we have 73 Users since 2020-08-22, if I count right. It it is getting more and more every day.

That could not be related to a wrong time.

I more assume it is related to IOS 13.7/14 or some changes in android.

[ndegendogo]

Thanks @Tho-Mat for your update.

So, what is now the plan how to proceed with the analysis of these missing keys? I assume the improved logging at the server might give more insights? Is there a ticket for this?

[thomasaugsten]

Missing means also there are not in the next day 3am package included ?

[Tho-Mat]

@ndegendogo https://github.com/corona-warn-app/cwa-server/issues/764

@thomasaugsten https://github.com/corona-warn-app/cwa-server/issues/764 https://github.com/corona-warn-app/cwa-server/issues/723

Missing means also there are not in the next day 3am package included ? yes: I assume you mean the 2020-mm-dd-hour-00.zip file I don't think the keys of a user are publish in different hour files. But I may be wrong with this assumption.

[Tho-Mat]

I also identify a special case in 2020-09-24-hour-16.zip

risk	date	keys
1	17.09.2020	70
1	16.09.2020	75
1	15.09.2020	70
1	14.09.2020	75
1	13.09.2020	75
1	12.09.2020	75
1	11.09.2020	75
3	18.09.2020	70
3	17.09.2020	5
5	19.09.2020	75
6	23.09.2020	70
8	23.09.2020	5
8	22.09.2020	75
8	21.09.2020	75
8	20.09.2020	75
8	12.09.2020	5
8	11.09.2020	5

Whatever you try, you will always get 2 key-chains with a missing 6. One of them will have a start date 23.09 with risk 8 One of them will have a start date <=22.09 with risk 8

One explanation would be: The first user set his time 1 day in the future The second user set his time 1 day in the future AND has switched off his phone on the day before he submits his keys.

Making 2 assumptions to explain a case is too much for me.

If I assume that the missing 6 is not related to a time shift it could be that all submitted keys have the correct UTC date but only the wrong risk value (not sooo bad) it also could be that the submitted keys have the correct risk value but the wrong date (very bad, all keys are invalid)

[thomasaugsten]

The keys of a user are publish in different key files if he submits future keys this keys will publish start date + 26h this means you should find you risk score 6 keys in the 3am packages

[Tho-Mat]

@thomasaugsten Ok. Let's have a look on the hour-files of the 26.09.2020 (yesterday) You will find 60 key-chains with a missing 6 With the multiplication of 5 that would result in 300 keys with a 6 for the 26.09. The first hour file of today is 2020-09-27-06 with 215 key and looks like:

risk	date	key
1	21.09.	5
1	20.09.	15
1	19.09.	15
1	18.09.	15
1	17.09.	15
1	16.09.	15
1	15.09.	15
1	14.09.	15
3	22.09.	5
3	21.09.	10
5	23.09.	5
5	22.09.	10
6	26.09.	20
8	26.09.	5
8	25.09.	20
8	24.09.	20
8	23.09.	10

If you assumption was true there should be 295 more 6es. only 295 since we only have one "lonely" 6.

a possible distribution

Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys		Risk	Date	Keys
8	26.09.	5		8	25.09.	5		8	24.09.	5		5	23.09.	5		3	22.09.	5		1	21.09.	5		1	20.09.	5		1	19.09.	5		1	18.09.	5		1	17.09.	5		1	16.09.	5		1	15.09.	5		1	14.09.	5
6	26.09.	10		8	25.09.	10		8	24.09.	10		8	23.09.	10		5	22.09.	10		3	21.09.	10		1	20.09.	10		1	19.09.	10		1	18.09.	10		1	17.09.	10		1	16.09.	10		1	15.09.	10		1	14.09.	10
6	26.09.	5		8	25.09.	5		8	24.09.	5
6	26.09.	5

I have to check the 59/295/290 again I could have mad a mistake and they may +-1/+-5/+-5 corrected:=> should be 60/300/295

[ndegendogo]

@Tho-Mat both tickets you mention are closed. I am happy if you use this ticket for further investigations, no need to create yet another. I just wanted to be sure, because the title of this ticket suggests a slightly different focus.

[Tho-Mat]

@ndegendogo Since both are close we unluckily talk about 2 different problems.

1. like the title say.
2. I no longer believe the missing 6 is caused by the problem in the title.

History: First i opened https://github.com/corona-warn-app/cwa-server/issues/723 A possible cause was the wrong time of a user device. Then i opened this ticket to ensure all users have set their device to the right UTC time. In the meantime a large number of missing 6s has occurred that can no longer be explained with the wrong device time.

[thomasaugsten]

ok can you extract the risk level 6 keys of all 3am packages or the first hour file with keys in the last days to see there is a higher distribution of level 6 keys

[Tho-Mat]

@thomasaugsten mist, mist, mist: Oversaw the -03 file from today. Will have a look on it Ashes on my head..

[Tho-Mat]

2020-08-22-hour-17: one missing 6 => 2020-08-22-hour-08 one lonely 6 => ok 2020-09-15-hour-22: one missing 6 => 2020-09-16-hour-05 one lonely 6 => ok 2020-09-17-hour-14: one missing 6 => 2020-09-18-hour-05 three lonely 6 => ok 2020-09-19-hour-08: one missing 6 => 2020-09-20-hour-06 one lonely 6 => ok 2020-09-21-hour-06: one missing 6 2020-09-21-hour-09: one missing 6 2020-09-21-hour-17: one missing 6 2020-09-22-hour-00: one missing 6 2020-09-21-hour-20: one missing 6 => 2020-09-22-hour-03 five lonely 6 => ok 2020-09-22-hour-08: one missing 6 2020-09-22-hour-15: one missing 6 2020-09-22-hour-03: one missing 6(or 14 keys) => 2020-09-23-hour-03 four lonely 6 => ok 2020-09-24-hour-08: one missing 6 2020-09-24-hour-09: two missing 6 2020-09-24-hour-11: tree missing 6 2020-09-24-hour-12: two missing 6 2020-09-24-hour-13: one missing 6 2020-09-24-hour-14: one missing 6 2020-09-24-hour-15: tree missing 6 2020-09-24-hour-16: two missing 6 2020-09-24-hour-17: one missing 6 2020-09-24-hour-18: two missing 6 => 2020-09-25-hour-03 twenty-one lonely 6 => ok 2020-09-25-hour-07: four missing 6 2020-09-25-hour-08: two missing 6 2020-09-25-hour-11: one missing 6 2020-09-25-hour-13: tree missing 6 2020-09-25-hour-14: one missing 6 2020-09-25-hour-15: two missing 6 2020-09-25-hour-16: one missing 6 2020-09-25-hour-17: two missing 6 => 2020-09-26-hour-03 Seventeen lonely 6=>ok 2020-09-26-hour-06: one missing 6 2020-09-26-hour-07: one missing 6 2020-09-26-hour-08: nine missing 6 2020-09-26-hour-09: two missing 6 2020-09-26-hour-10: six missing 6 2020-09-26-hour-11: three missing 6 2020-09-26-hour-12: three missing 6 2020-09-26-hour-13: ten missing 6 2020-09-26-hour-14: two missing 6 2020-09-26-hour-15: five missing 6 2020-09-26-hour-16: two missing 6 2020-09-26-hour-17: two missing 6 2020-09-26-hour-18: three missing 6 2020-09-26-hour-19: three missing 6 2020-09-26-hour-20: seven missing 6 2020-09-26-hour-22: one missing 6 => 2020-09-27-hour-03 Seventy lonely 6=>ok 2020-09-27-hour-03: one missing 6

@thomasaugsten So you are totally right. The missing 6s are published in the first hour-file of the next day. I already was wondering about the high amount of 6s on 25.09 and 26.09, but did not see the relation. Thanks!!!!! chapeau.

So no key is missing. But the question remains: are they valid?

[thomasaugsten]

Yes they are valid. My assumption is the exposure check is independent of the start date of the TEK.

[Tho-Mat]

@thomasaugsten In the past a user could not submit a key for the current day. Did this change?

[thomasaugsten]

This is still the case sameDay TEK s not yet possible but we are working on this. But if you change your date in the future then the 6er key is a normal yesterday key but with longer wait time because the startdate is in the future or today. The problem with sameDay TEK there multiple TEKs per day possible.

[Tho-Mat]

But if the uses have changed their date to the future days before they submit their keys, all of the later keys are invalid or better: will not match on devices with the correct UTC time.

But I can't believe so many users have changed their date to a future date.

[thomasaugsten]

The keys should match. I will double check on this but the validness of a TEK is independent of his startDate.

This is a common thing in games to reduce the waiting times. I have no other explanation for this, the ENF will not return sameDay TEK and the sameDay TEK has a riskLevel of 5.

[Tho-Mat]

But as far as I know the key is not independently from the date. So if the date/time is more than +-2 wrong no match will be found. https://blog.google/documents/69/Exposure_Notification_-_Cryptography_Specification_v1.2.1.pdf page 9

You could also check, if a call to ENF -to give the own stored keys- will not give you keys for today. For iOS and android maybe one of them has changed this.

[daimpi]

But as far as I know the key is not independently from the date. So if the date/time is more than +-2 wrong no match will be found.

But aren't the RPIs which the user with the wrong date broadcasts also shifted by the offset? In this case the RPIs another user records will still match with the DKs the infected person (with the wrong date) uploaded.

[Tho-Mat]

@daimpi Don't know. But I think this easily could be tested. AFAIU: The contact person only gets the TEK and the start period of the TEK and will only look in this period. With the start period and the TEK the receiving device can calculate possible RPIs to find a match.

Let make things easy: sending and receiving device have the same date but sending device is in period 1 and receiving in period 30 the function for calculating the RPI = TEK+1

1. sending device sends RPI=TEK+1 to the receiving device
2. the receiving device stores the RPI in period 30
3. the sending device uploads his TEK and the start period (normal the day)
4. the receiving device downloads the TEK
5. the receiving device calculates the RPI=TEK+1...TEK+144
6. a macht will only be found if TEK+1 will be found in period 1..12 (=2h) but it can't, since it is in period 30.

Correct me if I'm wrong!

[daimpi]

2. the receiving device stores the RPI in period 30

I don't think the time of storage is important in this context. Afaiu the validity period of a stored RPI is encoded in the ENIN, which is part of the PaddedData in the encrypted RPI (documentation, p.7):

I.e. in your example above the recorded RPI would have an ENIN value which corresponds to period 1 not 30, because the ENIN is provided by the sender who lives in period 1.

[Tho-Mat]

@daimpi you are right. For the receiving device it would be possible to find the RPI(i,j) in his storage. But then every time all received RPI had to be checked against all possible RPI. For 14day that would be 14000*5=70000 TEK = 10 000 000 RPI to calculate and assume only 10 received RPIs a day that would lead in 100 000 000 compares. So they limit the compares to a 2h period on page 9 of the document: A +/- two-hour tolerance window is allowed between when a Rolling Proximity Identifier derived from the Diagnosis Key was supposed to be broadcast, and the time at which it was scanned. That can only be done, if the receive RPI are stored in a time slot (the time at which it was scanned).

[daimpi]

So they limit the compares to a 2h period on page 9 of the document: A +/- two-hour tolerance window is allowed between when a Rolling Proximity Identifier derived from the Diagnosis Key was supposed to be broadcast, and the time at which it was scanned. That can only be done, if the receive RPI are stored in a time slot (the time at which it was scanned).

Good point I forgot about this. I remember we recently had a conversation on this topic on Slack 🙂.

In this case you're o/c right: the scanning/storage timestamp of the receiving device also matters for the RPIs validity and having your clock out of sync with real time by more than 2h will render your CWA pretty much useless no matter whether you're the sender or the receiver.

[thomasaugsten]

We will setup a test three receiving device with wrong date and time

All devices a reseted with not collected keys

device01Android = -4h in the past device01Iphone = -4h in the past device02 = -24h in the past device03 = correct time

device 01a, 01i, 02 and 03 are close together for 1h device 03 will submit his keys and we will check the exposure on device 01 and 02

Will this answer all questions? My opinion is only the RPI is important not the startdate of the key. This means the device with exact 24h will receive an exposure.

[daimpi]

@thomasaugsten thanks this sounds great, I'm curious for the results 🙂.

My opinion is only the RPI is important not the startdate of the key. This means the device with exact 24h will receive an exposure.

Are you saying that your expectation is that device02 will receive an exposure, but not device01a/i ?

I would expect that none of the devices would receive an exposure, as they're all outside the 2h tolerance window.

[ndegendogo]

Will this answer all questions?

@thomasaugsten I suggest to use one iOS <= 13.6.1 and another >= 13.7. Apple has reimplemented their ENF to support the new spec version, and we have already seen several issues that they broke the backwards-compatibility ...

[Tho-Mat]

@thomasaugsten there should also be 1 android and 1 iOS device with the correct time and most resent os. This will show if they submit a key for today.

All of the four other devices should also submit their keys not only receive.

To get more info about the "shifted" 6 the devices should be on for at least 2 days.

It would also be interesting if the devices use the device time or the correct time.

[thomasaugsten]

We are 100% sure the devices are not submit the sameDay TEK Risk Level 5. On Android only beta accounts and whitelisted bundleID on Apple only in ENAPIVersion 2