CMFA FAR/FRR?

[woodbe]

For biometrics we show "quality" by measuring FAR/FRR values (or FNMR/FMR). For CMFA, is there some sort of equivalent measure that could be shown, given the number of inputs (and possible combinations), something that could boil it down to a single number/ratio?

[n-kai]

I think that it depends on which sensors CMFA uses.

If CMFA use only WiFi and GPS to continuously authenticate a user, there is no authentication error as long as a user stay at a known location and connect to the know WiFi network.

However if CMFA use WiFi, GPS, voice and gait to authenticate a user, CMFA can incorrectly recognize a user because of recognition error introduced by voice and gait. To measure performance of such CMFA, for example, vendor gathers 10 employees working at the same office, ask them

1) to use the device without the CMFA one month to record the number of device unlocks made by users 2) to use device with CMFA another one month to record the number of device unlocks 3) to exchange the device with CMFA among the employees and measure mean time to detect (mean time that CMFA need to detect other user start using a device and lock device)

CMFA performance can be measured by reduced number of unlock from 1) and 2), and mean detection time. But another parameter can be devised for the different type of CMFA and there is no standard for CMFA performance so vendor should devise their own matrix to measure the CMFA performance considering target environment

[gfiumara]

If I'm reading the question and @n-kai's response correctly, only biometrics will affect the FAR/FRR equation. Other signals (has the device moved, did the SSID change) are much more trivial math (?) and would only obtain error via signal spoofing.

In that case, I can say that in other biometric studies, some have done "fusion" of algorithms where, at a very basic level, similarity scores from multiple recognition algorithms can be averaged (similary_score1 + similiarity_score2 / 2) to get a fusion similarity score, which could get you a single FAR/FRR for the fused algorithm.

[gregott]

A fusion score would need to be developed that takes into account a number of factors. These may include error rates for the biometric factors, perceived “strength” of the biometric factor, e.g iris would be given more weight than gait. What to do with non-biometric factors that are essentially binary such as are you within the GPS geo-fence or on the approved WiFi or attempting access within allowed hours, etc. Consideration needs to be given to how many and which ones of the available factors are needed to maintain access, what to do when individual factors are giving divergent scores, etc. At a minimum you would hope the biometric portion of the fusion error rate, if that can be calculated, would be no worse than the worst error rate of the individual biometric factors being fused. There has been a lot of research in how best to do fusion in a multimodal biometric system. Last time I looked at this there was no single agreed-to answer. From what I remember how to do fusion depends on which modes are being combined. The research papers always seemed to include non-trivial methods of calculating the fusion score. We also should say something about how often the re-authentication is conducted. It cannot be truly continuous given battery constraints. The periodicity of authentication will likely be variable based on a number of factors. For example, while within the geo-fence and on an approved WiFi, the period can be longer. When off the WiFi the user may need to be authenticated more frequently. These decisions are so use case and authentication method specific it will be impossible for us to specify them here. I think we might be better served by requiring the device maker to provide flexibility in their CMFA scheme to allow the user to specify all the parameter values as suits their needs. The device should give feedback to the user (really the administrator who will configure the device for the user) on the impact to the range of fusion scores (trust level) based on the choices made. I can envision a CMFA Policy defined by the company IT department that is sent to mobile devices with access to company IT systems. The Policy will set the parameter values used by the device. (This Policy would be included in the resources to be protected.) I would also like to see a method to have the CMFA scoring scheme evaluated by someone to assure it is “good”. This will be a hard thing to do.

[woodbe]

Review this as an optional performance metric requirement

[woodbe]

A thought here about a performance metric. Maybe the answer is in speed of detection of changes, not in the FAR/FRR type of value. So for example my solution reacts in .3 seconds, so you can be confident that it will react quickly to changes.

A time-based metric would be more about the balance the vendor is able to achieve with detection of changes and battery life on the device than on some specific check.

[gregott]

I think time-to-react could be a good metric for the vendor to specify for some signals. This might make most sense for things like detecting an unauthorized WiFi or BT connection, time of day and other signals that are instantaneous in their transition from one state to the other. Location, signal strength, on body detection, ambient light level, etc. where gray areas exist as to whether the device is within its allowed parameters are harder to apply time-to-react metrics. In a case where the location changes too quickly for a device to travel that far or fast may trigger a CMFA response. The time to detect such “unnatural” movements may be suited to a time-to-react metric. Crossing a geo-fence boundary may not lend itself to this approach. As for biometrics and PAD, since these are performance metrics with which we are all familiar and have a relatively established meaning, they should be included in the claimed performance metrics.

We may end up with a list of performance metrics. This may not be a bad thing. Users may want high performance biometrics but not be too concerned about how fast the device reacts to the use of unauthorized BT. In the case where the user prohibits the use of BT, this metric is of no use to them. It is like shopping for a car. A recent college grad may want a car that has high torque and goes fast. A father of triplets might want a car that seats 6 and gets at least 35 mpg. The customer is going to shop around for the device whose specs most closely align with his requirements. Could we come up with a single performance metric for automobiles that is useful to all customers?

I suggest we ask the vendors to provide a list of metrics and include a minimum required set for them to supply. For example: Biometric FAR & FRR, PAD error rates specified in 30107 (such as APCER, BPCER, APNRR, etc.), time to lock screen after detection of unauthorized connection (Wi-Fi, BT, ANT, etc.), geo-fence margin of error, etc.

[woodbe]

So maybe the requirement (for us), is to provide an expectation of the metric for the components then. So for any biometric the is used/allowed, then the appropriate FAR/FRR/PAD (or equivalent) rates must be stated. For anything else, some expectation of response time based on the parameters of what is being measured. Obviously we don't want to specify every possible input, but we would maybe need to create some sort of selected assignment (so you select the broad category and then assign the detail).

For a more immediate question, should this be part of the ESR? Should we have a statement that this must be noted, or should this just be part of the PP-Module requirements that we write? I'm not sure this is an Essential Security Requirement as much as it is information we expect to be provided for clarity. As such I'm ok not putting this directly in the ESR (i.e. while the BIOPPM has an ESR for FAR/FRR, we would not add this to the CMFAPPM).

[n-kai]

I think that accuracy or performance of continuous authentication should be part of the ESR but we should also choose the cost-effective way of evaluation. We can take the same approach as BIOPPM does. We can ask the TOE developer to test the TOE based on their own matrix and the evaluator only needs to check whether or not the content of report meets some pre-defined requirements (The evaluator doesn't need to test the TOE to evaluate the performance of the TOE).

[woodbe]

I think that for the ESR, this is handled by the requirement that the level of confidence be determined continuously. An SFR from that can require the user to define the performance parameters of the input on determining the confidence level. I do not think this needs to be explicitly called out in the ESR.