New webfrontend: approvedUrl -> url

[smuellerDD]

After the switch to the new web frontend, I now see approval responses to contain

[
  {
    "acvVersion": "1.0"
  },
  {
    "url": "/acvp/v1/requests/151",
    "status": "approved"
  }
]

It used to be approvedUrl.

[smuellerDD]

After checking the code, I see that approvalUrl used to be used for all responses except for initial or processing status responses. I am fine to consolidate to one keyword. Thus, please confirm that url is used for all responses from now on. Thanks.

[smuellerDD]

On second look, I see that url is the reference to the request URL but not the approved URL. So I think the approved URL is missing.

[Kritner]

Hey @smuellerDD can you tell the request type that this issue occurs on? We should still be returning the approvedUrl for meta data resources that actually have a URL.

The approvedUrl was taken out for certify requests IIRC, since that resource being returned in the approvedUrl for that type never actually existed. Though, thinking about it now I suppose that's the only area you are able to retrieve the validation ID?

Would you be opposed to getting for that request approval a separate property that has just the validation ID? (I don't know if this is what we'll end up doing, but just wanted to throw it out there as a possibility)

[smuellerDD]

Am Donnerstag, 4. Juni 2020, 10:51:50 CEST schrieb Russ Hammett:

Hi Russ,

Hey @smuellerDD can you tell the request type that this issue occurs on? We should still be returning the approvedUrl for meta data resources that actually have a URL.

It is a very old request ID number 151 on the demo server. It likely does not exist any more.

The approvedUrl was taken out for certify requests IIRC, since that resource being returned in the approvedUrl for that type never actually existed. Though, thinking about it now I suppose that's the only area you are able to retrieve the validation ID?

I requested the registration for a module ID long time ago and never picked up the approval. Now I stumbled over that entry and my app flagged an error to obtain the data.

Would you be opposed to getting for that request approval a separate property that has just the validation ID? (I don't know if this is what we'll end up doing, but just wanted to throw it out there as a possibility)

I am not sure I understand - for me, the actual request ID in discussion is irrelevant. So, you can safely dispose of that request. I just opened the issue because the server returned data I did not expect.

Ciao Stephan

[smuellerDD]

Not of interest.

[smuellerDD]

reopen: I see the following for an approved certificate request

[
  {
    "acvVersion": "1.0"
  },
  {
    "url": "/acvp/v1/requests/11858",
    "status": "approved"
  }
]

Shall we leave it with url? If yes, is that url used for all responses?

[Kritner]

@smuellerDD sorry I lost track of this one...

In your comment: https://github.com/usnistgov/ACVP/issues/880#issuecomment-642819988

what is the endpoint you're hitting for the response:

[
  {
    "acvVersion": "1.0"
  },
  {
    "url": "/acvp/v1/requests/11858",
    "status": "approved"
  }
]

I would have thought that was the GET /acvp/v1/requests/11858" is it not?

"url" being "the GET endpoint you hit". Previously "approvedUrl" would represent the endpoint of the "approved" object. However, validations do not currently (and I don't think plan to) have an endpoint, so the "approvedUrl" was removed for approved validation requests. This is probably directly related to https://github.com/usnistgov/ACVP/issues/889.

I'm realizing we need to, at a minimum, return the validation ID of the approved request, as we currently are not.

I can't actually find any description of a "/validations/" endpoint within the specification at https://usnistgov.github.io/ACVP/artifacts/draft-fussell-acvp-spec-00.html, but I don't believe we had an intention of reproducing that endpoint in the new stack, as all the data should be available on CSRC. I'm a bit removed from this portion of the process however, @shaneshaffer / @celic can you confirm what I've stated?

I agree that, at a minimum, we need to return the validation ID, but I don't think that will be in the form of "approvedUrl" since we don't have a "/validations/" endpoint.

[smuellerDD]

Am Donnerstag, 11. Juni 2020, 19:38:27 CEST schrieb Russ Hammett:

Hi Russ,

@smuellerDD sorry I lost track of this one...

In your comment: https://github.com/usnistgov/ACVP/issues/880#issuecomment-642819988

what is the endpoint you're hitting for the response:

[
  {
    "acvVersion": "1.0"
  },
  {
    "url": "/acvp/v1/requests/11858",
    "status": "approved"
  }
]

I would have thought that was the GET /acvp/v1/requests/11858" is it not?

You are right, absolutely.

"url" being "the GET endpoint you hit". Previously "approvedUrl" would represent the endpoint of the "approved" object. However, validations do not currently (and I don't think plan to) have an endpoint, so the "approvedUrl" was removed for approved validation requests. This is probably directly related to https://github.com/usnistgov/ACVP/issues/889.

Uhm, how shall we get the certificate number programmatically? It used to be with the approvedUrl.

Note, in the ever growing list on the web site, it will be hard to manually search for the awarded certificate number. Besides, I just got a certificate for an IUT which in fact extends existing certificate.

So, I think there is a need to programmatically get a certificate number.

I'm realizing we need to, at a minimum, return the validation ID of the approved request, as we currently are not.

Right, we need that.

I can't actually find any description of a "/validations/" endpoint within the specification at https://usnistgov.github.io/ACVP/artifacts/draft-fussell-acvp-spec-00.html, but I don't believe we had an intention of reproducing that endpoint in the new stack, as all the data should be available on CSRC. I'm a bit removed from this portion of the process however, @shaneshaffer / @celic can you confirm what I've stated?

There was never a spec for /validation. It just happened that we discovered that by reviewing the returned information from the server.

Besides, during the initial "approval process" to gain access to the ACVP infrastructure, labs are requested to show the final certificate number to NIST to demonstrate that they are capable of interacting with ACVP.

I agree that, at a minimum, we need to return the validation ID, but I don't think that will be in the form of "approvedUrl" since we don't have a "/validations/" endpoint.

That is fine, but we really need a way to get the certificate number.

Though, it was nice to have the data from the /validations as it listed all information of a certificate. Parsing the web site for this information is a bit of a challenge.

Ciao Stephan

[shaneshaffer]

The old result included the validation ID in a url of an endpoint that the spec did not describe. What was being returned by that endpoint was a convenient byproduct of code that existed to display the very first ACVTS validation list several years ago, before CSRC was able to display ACVTS validations. During the recent rewrite we chose to no longer support this endpoint, as it was not required by the spec and would have required a rather large amount of work only to provide something that is duplicative of the functionality of CSRC.

Returning the validation ID is still important however, so we're updating the response now.

We may however return to supporting that endpoint, albeit with a significantly different output than before. Once CAVS is retired users will need to use ACVTS to perform metadata updates that they used to perform through CAVS Change or Update submissions. Doing so will require knowing the IDs of the metadata items being updated, and there is not currently a good way to get that information - using the metadata search functions is not terribly helpful when the data is not normalized. Upon completion of some internal data model changes we are working on we will likely return the validations endpoint to ACVTS as a means to get the IDs of all the metadata items on that validation.

We do not intend to provide any data about the algorithm capabilities of the validation via this endpoint. Adding endpoints to CSRC to expose the algorithm capability data will be considered if there is sufficient demand and reasonable use cases.

[smuellerDD]

@shaneshaffer I am fully in line with your statement that we do not need the algorithm details. All that the old /validations endpoint was really helpful for was the cert number and the meta data.

But most importantly, we need a way to get the certificate number rather sooner.

[Kritner]

This change had been pushed to both demo and prod:

It is retroactive as well, your requests that were approved certify requests will now display the validationId

[smuellerDD]

Well, I am not sure this validation ID is of help - we need the actual certificate number like A500 or so. How would I get that from the validation ID?

[smuellerDD]

As a side note, with the current response for an approved certificate request, we depart from the response structure used for all other meta data approvals where we have an approvedUrl.

To me, the certificate is nothing different compared to, say, the module meta data. Thus, I would say that an approvedUrl should be returned for a certificate. That url should point to a resurrected /validations or so endpoint. However, the data that you return from that endpoint then may only contain something like

{
  "certificate":"A1234"
}

[AlexThurston]

@shaneshaffer For what it's worth I was trying to get some traction about some clarity on this endpoint in this https://github.com/usnistgov/ACVP/issues/720 so I'm going to poke in this thread as well :)

TL;DR I really think this route is valuable to the users of ACVP and should contain as much information as possible for a particular certificate.

In its previous implementation the endpoint would list all the resources (OE, vendor, capabilities as well as something called a scenario) that went into the certificate in a programatic way. There were a couple of use cases which I could see as very beneficial, and I believe that a lot of that information should remain.

Firstly, is the case where a certificate needs to be updated: adding an OE to an existing certificate, adding a new algorithm to an existing certificate, or changing an existing algorithm in a certificate (obviously with all the required testing). Using the API, it was simple to pull out that information and the resources that were used and better understand how the certificate is going to need to be changed. This is especially valuable when the original certificate was done by another lab. As a silly but true example, making sure that the same vendor is used across multiple certificates. There are sometimes 100s of entires for a single vendor so being able to look up the actual vendor ID that was used in the certificate is valuable. If I wanted to add an algorithm to an existing certificate, I would also want to re-use the OE IDs that went into the validation.

The second is that from what I could see on the CAVP list, some of the details of various resources wasn't shown. It was nice to be able to look up the exact OE (for example) that was used in a validation. This is often helpful when interacting with large vendors where re-use/consistency is valuable. It was quite handy in a few cases where vendors were transition from CAVS to ACVP and want to know what their previous values looked like. It was equally valuable to see the exact capabilities that were registered in the test session that went into generating the certificate.

I'm certain that there are other ways in which programmatically being able to extract the data that presumably is used to populate what appears in this list

If it wasn't clear already (😄 ) I'm advocating strongly that this API not be removed and contain as much information about a certificate as possible.

[smuellerDD]

Just to give you perspective: for the FIPS validation, we need to list in the report all certificates. It would be crazy to manually collect all certificates considering the full automation we have.

Thus, our ACVP proxy downloads the certificate information to store locally to generate the certificate listing we need for FIPS 140-2 TE.01.12.01:

 * ACVP-AES-CBC - Cert. #A494, Cert. #A497, Cert. #A498, Cert. #A499

 * ACVP-AES-CCM - Cert. #A496, Cert. #A497, Cert. #A498

 * ACVP-AES-CFB128 - Cert. #A494, Cert. #A497, Cert. #A498

 * ACVP-AES-CFB8 - Cert. #A497, Cert. #A498

 * ACVP-AES-CTR - Cert. #A496, Cert. #A497, Cert. #A498

 * ACVP-AES-ECB - Cert. #A494, Cert. #A496, Cert. #A497, Cert. #A498

 * ACVP-AES-GCM - Cert. #A496, Cert. #A497, Cert. #A498

 * ACVP-AES-GMAC - Cert. #A498

 * ACVP-AES-KW - Cert. #A497, Cert. #A498

 * ACVP-AES-OFB - Cert. #A494, Cert. #A497, Cert. #A498

 * ACVP-AES-XTS - Cert. #A494, Cert. #A497, Cert. #A498

...

[shaneshaffer]

Sorry all, I've been mistaking this validation endpoint for another one. Longer reply coming once I make sure I know exactly what this was doing...

[Kritner]

I assume @shaneshaffer is still working on his response, but in the meantime, I'll be putting in a minimalist version of the "/validations/" endpoint, we can obviously revisit what's contained in the response in the future.

For the moment however, this is the intention:

Now returning approvedUrl again from the "/requests/{id}" endpoint:

New "/validations/{id}" endpoint containing the "validationId":

I'll update this issue once it has been deployed to demo/prod - hopefully this is sufficient for an interim step.

[shaneshaffer]

OK, I think I understand enough of what was being returned... To be honest, we (or at least certainly I) were under the impression that the acceptUrl on a validation request was going to an HTML page, not a JSON endpoint that was returning all this data.

What I was suggesting that we would add is a subset of what was there previously. The cases that @AlexThurston identified are exactly what I wanted to address by bringing back this endpoint. If you were doing ACVTS validations you might have kept track of the IDs you used on all the metadata items, but if you didn't, there was little chance of finding those IDs through the metadata search endpoints. Soon when CAVS is retired there will be a need to see those IDs for CAVS validations for which none of those IDs have ever been exposed, so we definitely have a need for an endpoint like this.

The exact output of the previous endpoint no longer makes sense for us to use, or at least it won't very soon. TL;DR;, some of that data won't exist anymore. Getting into the internals of our system a bit, our original model for a validation included one or more "scenarios"(as @AlexThurston mentioned). Each of these scenarios was a set of OEs that had the same algorithm capabilities. The thought behind this was that we could reduce redundancy in our database by simply saying that a set of algorithm capabilities applied to a set of OEs, and that it would be a convenient way to condense the validation display on CSRC. Unfortunately, doing this grouping required a ton of comparisons as new vector sets were processed, especially on existing OEs, creating a lot of churn in our data. It also turned out that there were some flaws in that process, including a rather major flaw if validation requests were performed a certain way, and we wound up with quite a mess in our data. These issues were some of the main triggers for the recent overhaul of our system.

As part of this, we changed our approach in this area. We're no longer grouping like this, each OE now has its own set of algorithm capabilities. You might have seen the impact of this on CSRC, as ACVTS validations are now much larger than they were a few months ago. We know this isn't good, and once CAVS is retired and we can transform the data we will be doing a major revision of the validation display on CSRC to account for this approach (and other usability issues)

So how does this tie back to the results of the validation endpoint? While we changed our approach to processing, we didn't yet change the data model - that has to be done in parallel with changing CSRC, and after we're no longer doing CAVS processing. We are working on that simpler data model, and there is no such thing as a scenario in that model. A validation will simply have a collection of OE/Algorithm instances, really just flattening the model.

To support the clear use cases we have for metadata, what I had planned to do for this endpoint was to expose the metadata urls that relate directly to the validation and updates that might be done - the implementation url and the set of OEs. I didn't plan on drilling all the way down into the implementations or OEs the way the old endpoint did a full dump of the data - if you want to see the dependencies of an OE, or the address for the implementation, walk through the appropriate chain of URLs. This endpoint would at least provide the context that isn't provided anywhere else.

We don't want to go back to exposing the algorithm/capability data via ACVTS if we can avoid it, as we know we need to do that through CSRC anyway. There's a larger audience for CSRC than ACVTS, and there are use cases there that mean we'll wind up providing an API there at some point to expose more data. We don't want to be providing the same data from two different systems. Basically, if you want the big dump of data that would support a CSRC-like display, get it from CSRC; if you want the data needed to interact with ACVTS, go to ACVTS.

All that said, we definitely need to bring back this endpoint in some form, and hopefully we can find some middle ground across the various use cases and use both ACVTS and CSRC to serve all the needs without undue burden. Let's keep the discussion going if there are any other thoughts.

[smuellerDD]

@Kritner thank you, the validationID is key

[jarnold01]

this has been deployed to Demo (/validations endpoint)

[smuellerDD]

@shaneshaffer I have absolutely no objections to your changes and your reasons. If the old approach turned out to be not helpful, I am the last to object. Thus, go ahead with our redesign of exporting the IDs and do not stick to the old way of doing it.

And I think that overhauled ID listing can be added to the validation endpoint that @Kritner is resurrecting at the moment. Thus, I think @shaneshaffer and @Kritner suggestions will go well together.

PS: And yes, I would not see the need for the detailed cipher algorithm listing that is now non-existing.

[smuellerDD]

@jarnold01 Thank you. But I am not sure we need it on Demo considering that certificates are only accessible on prod!?

(at least I need it on prod as I just got some certs awarded :-) - but it is not super urgent to add it to prod if you want to let it sink for a day or two)

[jarnold01]

everything always goes to Demo first, even if it's just 5 minutes before it goes to Prod ;-)

[Kritner]

@smuellerDD can you confirm on demo that (for now) it works for your needs as a /validations/ endpoint? We can proceed to prod after, it was thankfully a pretty small change, just want to get an idea that it's what you expect.

[smuellerDD]

Confirmed - that is what I need. Thanks

[jarnold01]

Thanks @smuellerDD - this has been deployed to Prod.

[AlexThurston]

@shaneshaffer First off thanks for taking the time to go through all that. It's very much appreciated.

Just about everything you say sounds reasonable and I'm glad that portions of that information will be available.

I guess my final thought revolves around the algorithm capabilities that went into the validation. I asked a little bit ago if there was any appetite to provide that information when you GET on a test session (https://github.com/usnistgov/ACVP/issues/835). It was suggested the way to get that information would be in the validation route but that isn't going to be available. I totally get that replicating the data between two sub-systems is not desirable and is a management nightmare. What if the validation endpoint returned the test session ID the went into the validation, and from the test session IDs, I was able to read what algorithms and the algorithm capabilities that were tested.

Again, I know that in the linked issue it was expected that we maintain the state of what capabilities were entered for what test session IDs but in the case where we never had access to the test session ID in the first place (it might have been another lab), I'd be nice to be able to get it some how.

As for an example use case, if a vendor wants a certificate with the exact same capabilities as a previous iteration or product. The FIPSer(tm) could just just download the JSON payload for the algorithm and use it to create a new test session for the new product.

You mention that this information is available from the CSRC but I don't know what that is :P. Does it provide an API where I can extract that algorithms (JSON) for a given certificate? Or are you talking about the visual CAVP list thing that in linked to in the previous post. If so, that's good and all for a visual representation, but getting the JSON that makes up the capabilities is more what I'm interesting in being able to get. Obviously it could be translated from the webpage but I'm guessing that's going to be prone to error, and if the data's there anyways...

Hopefully I explained this OK. I'm also fending of two small children while trying to write this out :smile:

[smuellerDD]

on prod: for test session 1498 I got

  {
    "url": "/acvp/v1/validations/11861",
    "validationId": "HMAC1133"
  }

[Kritner]

Well that doesn't seem right huh? I will try to take a look at this shortly but may not be able until the morning

[Kritner]

@smuellerDD are you sure that's the correct validation id? that appears to be a request id that was created today, but that's the request ID, not validation ID.

requests/11861 -> validations/32652 -> A494 is what i'm seeing if I treat the ID you posted above as the request ID rather than validation ID

[shaneshaffer]

@AlexThurston CSRC refers to https://csrc.nist.gov/Projects/cryptographic-algorithm-validation-program, the official public site for the CAVP program. This is where every NIST crypto algorithm validation dating back to 1977 can be found. Currently there is no API there for accessing the data, only the validation search and the resulting displays, which we will be revising soon. We do know of some parties who will never be ACVTS users that have needs for raw data instead of a human readable web page, so it is likely that we'll expose an API there in the not too distant future to get all the data.

Before you brought it up I had been considering whether it would be worthwhile for this ACVTS endpoint to include data about the test sessions and vector sets related to a validation. Of course we wouldn't have this data for CAVS validations, or some ACVTS validations before we improved tracking of those relationships, but going forward we should be good. While I like your idea of being able to grab and reuse the JSON registration for an algorithm, one of the premises of our JWT based access control is that only the person who originally created a test session can ever access any data related to that test session, other than the public representation on CSRC (which isn't like the JSON). We'll have to think about this one some more.

[smuellerDD]

@Kritner my bad - due to the shakeup, my tool stored the request ID in place of the validation ID and I manually had to change it. It seems I mixed it up, please disregard.

I am good now with the resolution. The issue can be closed from my perspective.