ehn-dcc-development / eu-dcc-schema

Schema for the ehn DCC payload
Apache License 2.0
164 stars 59 forks source link

valueset for rapid antigen test #76

Closed lcimaglia closed 3 years ago

lcimaglia commented 3 years ago

Hi

here at this link:

https://covid-19-diagnostics.jrc.ec.europa.eu/devices/2?manufacturer=&text_name=&marking=&rapid_diag=&format=&target_type=&field-1=HSC%20common%20list%20%28RAT%29&value-1=1&search_method=AND#form_content

there are the rapid antigen test EU approved. I am wondering how we are going to manage the updates of this list vs the valuesets in test-manf.json file.

As for now there are 66 tests in the approved list compared to the 26 entry in the test-manf.json file.

emmeligross commented 3 years ago

Hi! I'm trying to develop an API for reporting test results. In the API we need to specify which codes that should be used, for example for describing which test that was used. I would need a description of how the connecting systems can get a list of approved tests. Are all systems supposed to fetch them themselves from https://covid-19-diagnostics.jrc.ec.europa.eu/devices manually, or is somebody going to provide the list somewhere in a more machine readable format? If yes, how often will this list be updated?

Thanks!

jkiddo commented 3 years ago

The machine-readable format can be found eg. here: https://covid-19-diagnostics.jrc.ec.europa.eu/devices/hsc-common-recognition-rat

gabywh commented 3 years ago

The intention is that the list is re-generated at least one every 24-hour period from the EU JRC sources and made available for download from a known location, which at least to my thinking would fit with either the Digital Covid Certificate Gateway or an official EU DCC hosted area. The request for an official EU-hosted location was already (re-)raised a couple of weeks back. Whichever physical hosting solution is eventually provided, the same principle applies of a known location where Verifier (and Issuer) apps can download the machine-readable valuesets. By the way, this is valid for all value sets in the schema, not just vacc

gabywh commented 3 years ago

In essence, the valuesets here: https://github.com/ehn-digital-green-development/ehn-dgc-schema/tree/main/valuesets

The schema was so designed as to allow the valuesets to be referenced without being embedded directly in the software. Thus you should be able to download the valuesets on whatever frequency of update you decide is appropriate whilst not having to change the software.

BTW, I will also be adding a valueset for recovery in the near future

emmeligross commented 3 years ago

It sounds really good that you are intending to make a solution that allows software to download the valuesets whenever they want! Next question: Could you please assign unique identifiers for all valuesets? Preferably in the form av URIs (or OIDs or UUIDs if that suits you better)? For example (just making this up) "http://covid-19.eu/ValueSet/vaccine-medicinal-products" Then when we build APIs using these valuesets, the identifier can always be used to make sure that you have got the correct valueset, even if it is moved between different locations (from GitHub to some other site, or republished somewhere else).

That would be a really big help!

lcimaglia commented 3 years ago

From what I know the verifier app verifies the JSON schema, so the valueset versioning should be carefully managed

dirkx commented 3 years ago

It is fairly good practice to use schemas intensely on the sending side - when you create them (be strict on what you sent). But cut the world more slack on the receiving side - doing a best effort. Also as your app or environment can ‘get behind’ the rest of the world.

emmeligross commented 3 years ago

Hi again guys! So, the API I need to provide is for reporting test results to an application that will make the digital certificate. This API will be used by several organizations and different systems. So my use case is reporting data that will be the basis for making certificates. Therefore it is important that it is clear for everyone which data they need to send in; the correct codes for test type, the test that was used and test result. Otherwise there will be a risk that the certificate will be faulty and that the traveler won't be let in when trying to pass the border. Therefore in the API we need to point out these ValueSets in an unambigous manner, even if they are moved for GitHub, or if somebody somewhere decides to publish them on a code server or whatever. You will still know that it's the correct ValueSet referenced as you have the unique identifier of the specific ValueSet. In our API we can reference a unique identifier where we need to point to a specific ValueSet, and in the documentation describe where to find the ValueSets.

gabywh commented 3 years ago

@emmeligross yes, indeed, that would be a good approach - which is why the value sets are defined as they are in the schema via a loose reference from the "main" schema, same model in fact as you are describing here for your API

  1. test - am looking to update the valueset for test even as we speak, also: https://lfpublichealth.slack.com/archives/C01V8GX69K5/p1622107214036600
  2. recovery - based only on NAAT, am working on an update release for today on this one. Also from recent discussions I have had: implicitly any NAAT is therefore an approved test and it is thus not foreseen that there will be a separate valueset of known good tests for recovery.

I'll keep you posted on progress.

gabywh commented 3 years ago

https://github.com/ehn-digital-green-development/ehn-dgc-schema/releases/tag/1.2.0

lcimaglia commented 3 years ago

@gabywh I see in the 1.2.0 schema version the "dr" field for test entry is not allowed anymore.

This change could have a significant impact on the MS ready to start issuing DGCs from 1st June. I wonder if the 1.0.0 schema should be still considered valid for those and how to manage such schema updates in the future. So I assume the schema validation by the verifier app should be based on multiple schema version according to the one stated in the current QR scanned.. am I right?

gabywh commented 3 years ago

Previous versions of JSON docs containing the dr field will still validate against this v1.2.0 of the schema.

Exactly which version of the JSON schema will be used for DCCG and go-live is something I've already discussed with T-Systems / Steffen Schulze directly.

Note that there is a further release pending (I'm just waiting on reviewers to approve) whereby the name DGC goes to DCC - > including filenames and refs - minor and somewhat trivial (even perhaps somewhat slightly irritating!) from a pure development perpsective - but I'm trying to get these changes in before the 1st June issuance date -> the sooner the better as less overall invested effort.

gabywh commented 3 years ago

So I assume the schema validation by the verifier app should be based on multiple schema version according to the one stated in the current QR scanned.. am I right?

Yes, that should always be the case - but like I mention above, I do try for non-breaking changes where possible so you could still use schema 1.2.0 to validate e.g. JSON written against schema v1.1.0.

lcimaglia commented 3 years ago

thanks @gabywh
I am no expert of JSON schema validation.. I assumed a missing field on the schema could result in validation error if the field is present on JSON data

thank you also for the tip about next update!

gabywh commented 3 years ago

next update:

https://github.com/ehn-digital-green-development/ehn-dgc-schema/releases/tag/1.2.1

... and yes fully appreciate that changing file names is sth of a PITA, but I hope to have at least minimized the pain by doing it asap up-front

lab-incert commented 3 years ago

Hello all, I (Luxembourg) am struggling to understand exactly how this value sets for the test name and test manufacturer will be managed, and for me, this is worrying, as DCCs from other EU Member States may be showed "red" (in LU, we use a red/orange/green code), thus rejected. If I understand properly, for NAAT, both info are optional, for RAT, both info are mandatory. In the JSON schema, for NAAT, the field "nm" is filled with the Test name and is optional. For RAT, the field "ma" is filled with the "id_device" which points to the Test name and Manufacturer name. The field is optional, which I don't understand why, as according to the regulation, it is mandatory. When I talk about id_device, I am referring to https://covid-19-diagnostics.jrc.ec.europa.eu/devices/hsc-common-recognition-rat. Currently, in LU, we use only this database. We edited it, in order to have something such as: { "valueSetId": "test-rat", "valueSetValues": { "additionalProperties": { "1341": { "country": "China", "name": "Qingdao Hightop Biotech Co., Ltd", "commercial_name": "SARS-CoV-2 Antigen Rapid Test (Immunochromatography)" }, "1263": { "country": "South Korea", "name": "Humasis", "commercial_name": "Humasis COVID-19 Ag Test" }, "1180": { "country": "Germany", "name": "MEDsan GmbH", "commercial_name": "MEDsan SARS-CoV-2 Antigen Rapid Test" }, "1215": { "country": "China", "name": "Hangzhou Laihe Biotech Co., Ltd", "commercial_name": "LYHER Novel Coronavirus (COVID-19) Antigen Test Kit(Colloidal Gold)" },.......................... and so on and so forth. We do the same with NAAT test. We really need to be aligned with the EU on this. Our solution would be suitable.

gabywh commented 3 years ago

update of table just pending 1 reviewer approval

lab-incert commented 3 years ago

Thanks. And about this point. "For RAT, the field "ma" is filled with the "id_device" which points to the Test name and Manufacturer name. The field is optional, which I don't understand why, as according to the regulation, it is mandatory." Is this normal?

gabywh commented 3 years ago

Thanks. And about this point. "For RAT, the field "ma" is filled with the "id_device" which points to the Test name and Manufacturer name. The field is optional, which I don't understand why, as according to the regulation, it is mandatory."

The JSON schema is about structure - a static thing. The field is optional for NAAT -> thus mandatory for non-NAAT. However, this is a data-driven thing and not easily modelled in a JSON Schema. Data-dependent validation needs to be handled by the business rules software. Since it is optional in NAAT, we have to leave it as optional in the JSON Schema and require the business rules to then enforce mandatory (or not) based on data of test type

gabywh commented 3 years ago

Valueset updated in github for now; however an up-to-date version should be distributed by the DCCG in future.