General Conformance Claim Accuracy and Consitency

[xahun]

General Conformance Claim - While there are downloadable PDFs for the module, supporting document, and toolbox overview. The versions of the module and supporting document are the same. The version of the toolbox overview is much earlier, but maybe doesn't matter. My concern is in regard to conformance claims - it seems the module, supporting document, and toolbox content all needs to be uniquely and unambiguously identified in terms of the specific versions/content used for a given evaluation. The supporting document refers to "[Toolbox] Toolbox Overview, November 9, 2021, Version 1.1." but "https://biometricitc.github.io/v1.1/1.1PRE/BIO-PAD-Toolbox-Overview-v1.1.pdf" yields version 1.0. But regardless of that problem, I don't see how the specific toolbox content is associated with a specific module version. That needs to be cleared up somehow so that accurate claims can be made.

[woodbe]

Regarding the version number, that is definitely an "oops" in the docs.

The toolbox content for any modality is by the releases page for each one (see the Eye example), and would be the latest package from here. This can definitely be written more clearly though in the Toolbox Overview.

The idea on the modalities is they will just be versioned as needed individually, so if you had Eye and Fingerprint, you would take the latest version of each of them as the documents to use.

@xahun do you think there is a need for PDFs to be created and published along with the adoc files that have been used do far?

[xahun]

So my concern is that from a NIAP perspective, we claim specifically identified packages, with specifically identified deviations (e.g., specific Technical Decisions) applied. PDFs provide us a means of having a copy of what we actually used (since using hashes we have found that even PDFs occasionally change unexpectedly without any indication), maybe adocs work, but we'd need to have a way to capture and retain a copy of what we used. The notion of using the latest available thing is similarly problematic - can we unambiguously identify what was used for evaluation and perhaps even more important is a question of NIAPs role - since they will presumably endorse this, what exactly did they endorse and what changes are they accepting without further endorsement. Consider NIT interpretations in the Network realm - the NIT issues those, but NIAP follows up with Technical Decisions that serve to endorse (or reject) those.

[woodbe]

As for being able to know exactly what version of the documents are used, that is the great thing about GitHub releases. That is a fixed set once it is created. While you can add files to a release, it is not possible to edit the actual content that is generated automatically when the release is created, so what is there (in terms of the adoc files, as those are managed directly by GitHub), is explicitly the correct version (and any generation of PDFs or HTML is output generated by external tools for better presentation, as opposed to original files).

As for using the latest release, NIAP could of course say they don't want a specific release of the toolbox modality, the point there was that the modality toolboxes are not intended to be released on the same schedule as the PP/SD/Toolbox docs, but to be managed as needed for those toolboxes. So other than any particular NIAP issue with a specific toolbox version, the idea is that if you start an eval today, you would look at the latest published released toolboxes and pick the ones for the devices in evaluation. If you start another eval in 6 months, you would check again, even if the PP/SD/Toolbox hasn't changed, the modality toolboxes may (of course it is also the same in reverse). The main goal was to make sure that the toolboxes themselves were not tied to specific PP/SD requirements (and at this point we do not know of any).

[woodbe]

So looking at this some more, it really isn't clear what we should do, for several reasons. The first is no one has tried to do something like this (but I don't see how we can not do it this way since the individual toolboxes need to be able to be updated at their own pace, and that is why they are separate from the main docs). The second one is that it isn't clear how to link the Toolbox into the rest.

Looking at the NIAP PP-Configuration docs (I used the more recent NIAP ones for a firewall) as an example, they don't even specify the SD in the PP-Config (in this one it is at least there as a reference). I think it may be a good idea to specifically add the SD into it somewhere as it would seem to remove some level of potential ambiguity. If we did that, it would also be possible to add the Toolbox to that section.

An alternative would be to specifically reference the Toolbox in the beginning of the SD in section 1.1 (I don't know if it should have a note about being optional there, though that is clear in other sections but that doesn't mean it isn't part of the SD).

For the individual toolboxes then, what I am thinking is to add a section into the Toolbox that talks about choosing the proper toolbox based and how to reference it. At this point I could also add a table that states the minimum version/date that is expected for any toolbox to be used (i.e. state that you can't use something older than some specified version/date, which could be specified on a per-modality basis). This could also be duplicated in the Toolbox repo (with links to the proper ones (same on the main website which does list the version with a direct link to the appropriate package).

So the linkage would be PP-Config -> SD & Toolbox added (and Toolbox added as part of 1.1 in the SD) -> Toolbox would have a minimum version/date specified to the modalities (with links to the repo)

Further information would be provide in the Toolbox to explain how to choose the proper toolbox for a modality test.

[woodbe]

Add section 4.4 in the PP-config about PAD conformance

Add Toolbox section about how to find the modality toolboxes and how they are versioned