search

ehn-dcc-development / eu-dcc-schema

Schema for the ehn DCC payload
Apache License 2.0
164 stars 59 forks source link

test-manf.json no cumulative update? #87

Closed miguelzapaton closed 3 years ago

miguelzapaton commented 3 years ago

More than a real PR this is an input for discussion with diff.

From an implementer's perspective I wouldn't have expected two test manufacturers to disappear "overnight" with a valueset update.

Example: If I got tested with 1242 or 1065 yesterday, shouldn't it be validatable today?

I think this is especially important for the business rules, e.g. here:

https://github.com/ehn-digital-green-development/dgc-business-rules

Regarding the ValueSets in general I've got some more questions:

TIA

Miguel

gabywh commented 3 years ago

Background

The test-manf.json valueset is generated directly from the EU Joint Research Council (JRC) list at https://covid-19-diagnostics.jrc.ec.europa.eu/devices/hsc-common-recognition-rat who define this list EU-wide. The eHealthNetwork Semantic Sub-Group generate the format of the valuesets we use here and has generously supplied us with the script they use to auto-generate the test-manf.json valueset from the EU JRC list. You can find this script here: https://github.com/ehn-digital-green-development/ehn-dgc-schema/blob/main/valuesets/source-update/update-rat.sh

Specifics

I do know that with the update some more generic manufacturer entries were replaced with one or more specific entries so it can certainly be the case that a more generic manufacturer has "disapppeared" only to re-appear in a more specific format.

I'll ask about this with the Semantic Sub-Group to get a definitive answer for you.

gabywh commented 3 years ago
  • What's the function of the "active" flag in the valuesets and why don't you use it?
  • How will valuesets be versioned? Globally vs on valueset level (-> valueSetDate = version?) vs on entry level?
  • Will there be one official API Endpoint to get updated valueset data?

Now answered in the FAQ: https://github.com/ehn-digital-green-development/ehn-dgc-schema/wiki/FAQ#valuesets

  • How is the offline use of DCC systems addressed, is this a design goal?

I'm not sure how this fits with the general topic of valuesets. Could you perhaps explain a little more and then I can answer appropriately?

kruzikh commented 3 years ago

IMHO

Regarding the ValueSets in general I've got some more questions: What's the function of the "active" flag in the valuesets and why don't you use it? Active flag should be checked by business rules. Only concepts with active = true can be used for new certificates, inactive values are left in the valueset for backward compatibility. However it is not clear how to deal with inactive entries. This need to be decided. At this moment only RAT manufacturer and product name valueset is expected to be subject of frequent changes and potential inactivation of entries. How will valuesets be versioned? Globally vs on valueset level (-> valueSetDate = version?) vs on entry level? Each valueset has a valueSetDate element with date of last update of the valueSet. Entry level version exist however is currently not used. Use of versioning of the valuesets and entries should be further discussed. E.g. could be used for differential updates of the DCC apps.
Will there be one official API Endpoint to get updated valueset data? All valuesets will be distributed by DCCG. How is the offline use of DCC systems addressed, is this a design goal? Yes. DCC has been primarily designed for an offline use. Valuesets should be regularly (each 24 hours) downloaded from DCCG to the DCC apps to perform correct verification.

gabywh commented 3 years ago
kruzikh commented 3 years ago

As regards RAT tests that has been removed from the common list. The problem is that current JRC api does not return them. So, we should either ask JRC to change the API or we would have to compare a list retrieved from the JRC with prevous version of the value set and mark those entries that had been removed as inactive. I opened this question with Daniel and Konstantin. As I already wrote, we (TIO and SGS) also need to decide how to deal with inactive concepts in the value sets. It does not make much sense to my opinion to merge this PR until we will be clear in that.

miguelzapaton commented 3 years ago

Dear @gabywh, @kruzikh

Many thanks for your answers!

OK, I understand that the current handling of the valuesets is of temporary nature only.

After a quick check of the DCC test files it seems that none of them references 1065 or 1242. So currently nobody cares that they are missing :) Sorry, I just double checked it, two test files are currently referencing "ma" 1242:

As stated at the beginning, this PR was never thought to be merged and mainly to address the issue in general. Please feel free to close it and move it to issues or discussion.

Regarding offline use and valuesets (this might also affect schemas, business rules, dscs, etc.):

What I meant with "offline use" was the fact that a DCC issuance or verifier system could be temporarily offline and not be able to get online updates of valuesets (schemas, business rules, dscs etc.).

I think nowadays that's more accidentally than structurally, e.g. a firewall misconfiguration, a network outage caused by a thunderstorm, etc.

This could either be ignored from your perspective or addressed "formally", e.g.:

A maximum offline window (Dmow) for all DCC systems is defined between 1-7d (the time needed to fix the above mentioned problems).

Having this situation just using a boolean active flag (that would be indeed more comfortable for implementers) on entry level is not enough information to keep the overall system consistent.

The following information would be necessary instead (similar to the FAQ answer):

Calculated from these values:

Where:

NoIssuanceBeforeDateTime >= PublishingDate + Dmow

Other solutions could be imagined on a valueset file or package level in combination with more sophisticated update procedures.

With regard to the final implementation I could either imagine a Rest API or a CDN style mechanism.

Just my two cents.

gabywh commented 3 years ago

Dear @gabywh, @kruzikh

Many thanks for your answers!

OK, I understand that the current handling of the valuesets is of temporary nature only.

~After a quick check of the DCC test files it seems that none of them references 1065 or 1242. So currently nobody cares that they are missing :)~ Sorry, I just double checked it, two test files are currently referencing "ma" 1242:

Ok, good to know, thanks for the input. @kruzikh: one for the discussion on RAT versioning...

What I meant with "offline use" was the fact that a DCC issuance or verifier system could be temporarily offline and not be able to get online updates of valuesets (schemas, business rules, dscs etc.).

So the idea is every 24 hours (or whatever is appropriate, some valuesets far more "volatile" than others) valuesets can be automatically generated and made available for download. Member State in the end to decide on when / how often they want to update their own verifier app.

A maximum offline window (Dmow) for all DCC systems is defined between 1-7d (the time needed to fix the above mentioned problems).

Is an option, amongst others, for sure - DCCG deployment and Member State discussion.

Having this situation just using a boolean active flag (that would be indeed more comfortable for implementers) on entry level is not enough information to keep the overall system consistent.

The following information would be necessary instead (similar to the FAQ answer):

  • NoIssuanceBeforeDateTime
  • NoIssuanceAfterDateTime (optional, once the entry is deprecated)

Should still keep it, allows client to always check validity.

Calculated from these values:

  • NoVerificationBeforeDateTime = NoIssuanceBeforeDateTime

thus ident, is valid_from

  • NoVerficationAfterDateTime = NoIssuanceAfterDateTime + MaxPossibleDurationOfDCC (optional)

MaxPossibleDurationOfDCC - optional and would drop, discussion elsewhere on this one, but dropping also gives an ident with valid_until

Where:

NoIssuanceBeforeDateTime >= PublishingDate + Dmow

Member State decision on how/when to update - but sure, guidance could be given from deployment side if Member State has no particular specific constraints

Other solutions could be imagined on a valueset file or package level in combination with more sophisticated update procedures.

With regard to the final implementation I could either imagine a Rest API or a CDN style mechanism.

Yep, issues for DCCG / deployment. I suspect CDN, but that's up to the deployment people.

Just my two cents.

Thank you - and thank you too for your thoughts and ideas here on these matters - most welcome :)