New types of tests

[woodbe]

This is a start to the new artefacts test plan.

This will close #342

This update changes the assumptions section in the text. Other changes still need to be updated.

[woodbe]

I moved the new artefact text that we will want to use in the toolbox process into the biometricITC/Administration#41 pull request to update that document with more details on the expectation of the changes.

[woodbe]

@n-kai While I agree that the iTC does not have the ability to control the evaluators actions, as we are the ones in control of the Toolbox, I don't see how it can be acceptable to allow someone to add new tests to the toolbox and then say that it is comparable to an evaluation that didn't have those.

My concern is something along these lines:

• vendor 1 tests to face v1
• the lab for vendor 2 decides there are new tests that should be done, and performs them, but does not add them to the toolbox, so still claiming support to the same face v1
• after the evaluation, the lab submits the new tests to the iTC, and we determine they should be changed, so the way they were run is not acceptable

So now vendor 2 has run the tests that meet v1, but also a set of tests that don't meet anything in v1 or anything in the approved v2 from the iTC. How would this be handled? How would this be repeatable (the NIAP requirement)?

This is why I want to say that the process for adding a new toolbox artefact would run something like this:

1. lab presents full set of docs for new artefacts to iTC (using toolbox TD process I will write up)
2. iTC will take 2 weeks to review the docs, and probably make some suggestions (likely working with the lab on it)
3. docs are added to the toolbox and version is iterated/published
4. full normal review proceeds on artefact addition, which could lead to no changes or additional updates to the docs. if changes are made, another revision is published, on whatever schedule the iTC takes

So this would mean going from v1, the lab would provide updates that we would use to create a v2, which would be accepted as the iteration for that modality. If the iTC decides additional changes are needed, then a v3 would be created, but v2 would be able to be used for the evaluation, and would be in effect as the current set of tests for all new evals running (so if another vendor started 3 months later, they would also have to use these docs with the new artefacts.

What I am trying to avoid is having 2 vendors testing the same modality and one of these scenarios happening (because the lab can do what it wants and still be approved):

• one lab decides that there are new artefacts to be used while the second one does not (in this case the second one happening a month or 2 after the one that did propose the changes)
• 2 labs each see the new artefact type and create individual plans that are different, and so while both vendors test against the new artefacts, they don't do so in the same manner, so the evaluations aren't technically equivalent

The way less ideal way to handle this scenario, I would think, would be to to say that until the new artefacts are approved, even if there is testing to the new artefacts, the certificate can only claim compliance to the current published version of the toolbox. If the scheme wants to list additional details about tests that is fine, but there wouldn't be any compliance claim to anything except the current version of the toolbox. The problem here though is that there is no consistency for any new artefacts, and likely this will be frowned upon (especially by NIAP).

[woodbe]

I have added another line based on this comment while merging that PR.