Decision threshold for distance below two meters

niclasku commented 4 years ago

Where to find the issue

In solution_architecture.md under the topic LIMITATIONS: "we assume that an attenuation <50dB translates to a distance below two meters".

Describe the issue

Assuming an attenuation of 50dB to be the decision threshold may not be sufficient. Measurements we have done during a master students project at TU Berlin show that the decision threshold (path loss) should be much higher to catch all contacts below a distance of two meters. You can have a look on our measurement data as well as on our measurement report for details. If you have further questions or remarks regarding the measurements feel free to answer here or contact us privately.

Questions

Are there measurements planned or done to determine the decision threshold?
If yes, is it possible to publish those measurements or results on how precisely the distance can be estimated?
Are there any correction steps planned to normalize the attenuation for example based on the smartphone model?

Internal Tracking ID: EXPOSUREAPP-2145

mh- commented 4 years ago

From reading the Apple / Google specs I got the impression that the 50dB attenuation threshold is mentioned in the description of attenuationDurations, but the app has no influence on this. You have to work with Google / Apple to modify that. All the app can do is configure ENExposureConfiguration. Any correction steps have to be done by Apple / Google inside their frameworks, because all the relevant data points never reach the app.

What we know already today, however, is that the Associated Encrypted Metadata (which contains the TX power value) is apparently not considered a fixed value, but recorded and stored by the Google framework for every single BLE reception. So we could anticipate that they probably try to detect the state of the mobile phone (on a table, in a pocket, driving in a car, ...) and a) potentially change TX power based on that, and/or b) send the state in the currently reserved space of the AEM (2 bytes), so that the receiving device can use this for a better estimation, in future versions of the framework.

tklingbeil commented 4 years ago

It is correct that the two fixed attenuation buckets are not ideal. Therefore two threshold values were introduced, forming three custom attenuation buckets.

Those new buckets are already used in the calculation of the combined risk (see solution architecture document), figure 13.

SebastianWolf-SAP commented 4 years ago

Dear @niclasku,

the previous answers already state precisely what has already been done on our side (and what can by done on top of the Apple/Google framework). In the end, it's up to Apple and Google to integrate and continue developing their framework based on research data. We will continue to do our best to incorporate and document additional aspects that we can influence - as already outlined by @tklingbeil.

As a consequence, we will close this issue now.

Mit freundlichen Grüßen/Best regards, SW Corona Warn-App Open Source Team

niclasku commented 4 years ago

Thank you for the detailed answers! You are right that the framework does not allow anyone to modify the attenuation directly. However it is possible to set weights for the eight different attenuation ranges with attenuationLevelValues and the thresholds for the three attenuation buckets with attenuationDurationThresholds. Those configuration values will affect attenuation risk, exposure score and therefore the combined risk score. I am interested in the following (hopefully better formulated) questions:

How do you determine the attenuationLevelValues and attenuationDurationThresholds?
Are you able to test the configuration and are you going to test it?
Are you going to provide a test report on how precise the risk score would be for an ideal environment?

JonasDaWi commented 4 years ago

Dear @mh-, @tklingbeil, @SebastianWolf-SAP,

for the documentation of this issue: We are glad that you "acknowledged" this issue at least in the documentation & improved this issue 12 hours ago in this commit via lowering the threshold to 58 db.

Another problem still exists: without device-specific calibration, RSSI values can have different meaning on different devices. Eg. measuring 58dB on a Samsung Galaxy has a different meaning to measuring the same value on an iPhone.

Best Regards!

mh- commented 4 years ago

@JonasDaWi You are right that device-specific calibration makes sense. It would have to be done by Apple / Google, though, because the CWA app has no chance to look at single beacon transmissions / receptions, this is all encapsulated in the Google / Apple framework. The app can only assume that Google / Apple do their best effort to estimate the actual attenuation as good as possible.

keugens commented 4 years ago

@mh-

It has to be done by Apple / Google, ...

Why? Do they have an order for this?

Do they have an order to provide something to have a calibration-app, to make use of a large community to help?

mh- commented 4 years ago

@keugens What do you mean by "order"? I believe Apple / Google provide the Exposure Notifications Framework on their own account, they pay for that themselves.

keugens commented 4 years ago

Order in the sense: the client specifies what he needs.

keugens commented 4 years ago

@mh- Do you think this is a new or unexpected issue? For me not, if I am looking to the Singpore app, for example.

lukastrm commented 4 years ago

If we trust in what was said the Corona-Warn-App press conference even the whole "It's on Apple and Google"-Topic becomes obsolete, since it was explicitly stated (by Dr. Müller, SAP) that the German development team deeply cooperated with the engineers at Apple and Google in order to improve the functionality of this app. We should therefore not primarily concentrate on having discussions if Apple and Google have an order for this or not but more if the Corona-Warn-App team is somehow aware of this issue and if they even tried to address it towards Apple/Google. So however an official response would be suitable that tells us whether Apple/Google have been contacted regarding device specific corrections and if yes, whether a better solution approach is on its way or was denied (with or without any reason).

In addition it is very shady that this issue asked for official precision measurements, was never answered in that way but closed a day after its creation as if nobody wants to admit that the classification reliability is way worse as expected. But suddenly at the press conference precision measurements were mentioned that promise 80% classification precision, however again neither with sharing on how the measurements were performed nor how they got these numbers. The phrase "will be published within the next weeks" is not only suspicious because just days ago nobody knew about the measurements but also focusing the fact, that everything else was published on time and almost every other measurement that has been shared on the internet states a significantly worse reliabilty for the use of Bluetooth in distance estimation, especially in real-world scenarios.

SebastianWolf-SAP commented 4 years ago

Dear colleagues,

first, it's of course a combination of efforts by many parties - Google/Apple, the Robert-Koch-Institute, Fraunhofer IIS and many more. From our side, we can only repeat our statements that we will continuously update the parameters which we can influence. We also added the configuration values now explicitly to the risk assessment document, see https://github.com/corona-warn-app/cwa-documentation/issues/307

Moreover, the other statement also still remains true: "We will continue to do our best to incorporate and document additional aspects that we can influence". At the point of time when we closed the issue no detailed documentation about the respective precise measurements was planned - this is still the case. But we will continue to ask the responsible teams if and in which form they might be able to document their findings publicly which lead to the configuration values.

On the other hand, we have the cwa-wishlist repository for such requests in place by now. Consequently, we will reopen this ticket and move it to this repo for voting, further discussions etc.

Mit freundlichen Grüßen/Best regards, SW Corona Warn-App Open Source Team

SebastianWolf-SAP commented 4 years ago

After the consolidation, @pdehaye opened another issue with https://github.com/corona-warn-app/cwa-documentation/issues/336 to ask more in detail what should be provided in addition to the existing documentation. As this issue was opened earlier, I copy the content to this issue and close the newer one:

why is there this clipping at 73dB "on average"? what does "on average" mean? (this was https://github.com/corona-warn-app/cwa-documentation/issues/307, left unaddressed there, even though the issue is now closed)
why isn't the weight 0 for exposures between 10 and 15 minutes? This differs from the Immuni configuration, for instance (this is https://github.com/corona-warn-app/cwa-server/issues/601 on the server repo)
what are the app_defined factors? what do they mean, beyond a numeric value of risk?
what is the idea behind the deault-bucket-offset (currently set to 0, not really explained in the solution architecture)
given that default-bucket-offset is 0, I would have expected that your configuration of attenuationThresholds should match the Swiss one for that part, but it differs significantly. With the same weights as you of [1, 0.5, 0], they chose [50, 55] while you chose [55, 63]. Given what I said in the Background above, I am very surprised by this difference. What am I missing?
How was the Risk Score classification threshold value of 15 chosen?
what does "the attenuation is weighted by the duration at each risk level and averaged for the overall duration" mean?

In general we can now tell where each factor is set, and what the value is, but not how it was decided to set it to that value (even once the solution architecture was chosen). The risk scoring documentation does not answer those questions either.

pdehaye commented 4 years ago

As an additional part of corona-warn-app/cwa-documentation#336 that might provide helpful background, I had written this:

The configuration of CoronaWarn, as described in the solution architecture document, leverages two parts of the GAEN API: an "Exposure Score" (based on a per exposure summary) and a "Normalized Total Risk Score" (based on an aggregate) to calculate the risk. The interesting bit is that SwissCovid relies on the first, while Immuni relies on the second. So in a perfect world, each component of this risk score would match each of those other deployments (no?).

The reason to ask it like this is that I get CoronaWarn, Immuni, SwissCovid, etc are all built on top of a foundation they don't control (GAEN). But presumably they are studying the same electromagnetics and same epidemiology. So any significant difference in parametrization might be interesting to trace to its origins, which is probably going to be open data.

SebastianWolf-SAP commented 4 years ago

FYI, in case you haven't seen that already, please have a look at the recently published information from Google: https://developers.google.com/android/exposure-notifications/ble-attenuation-overview

corona-warn-app / cwa-wishlist