SORMAS-Foundation / SORMAS-Project

SORMAS (Surveillance, Outbreak Response Management and Analysis System) is an early warning and management system to fight the spread of infectious diseases.
https://sormas.org
GNU General Public License v3.0
293 stars 142 forks source link

Concept for different Schema-Versions in SORMAS #6324

Open markusmann-vg opened 3 years ago

markusmann-vg commented 3 years ago

Problem Description

With SurvNet we have another system which is used in Germany for Surveillance. As they export data, which we import into SORMAS, we have to manage different schema versions in SORMAS as this is important for statistics and the evolution of data. E.g. If a case was created in a older schema and is imported into SORMAS, the user can see all available data from the current SORMAS version. When there is a case with COVID where no disease variants have been known, when the case was created, the SORMAS user should not be able to add a disease variant without updating it to the latest version.

Proposed Solution

First of all we have to add a field for all entities, which describes the schema version, when the entity was created So when importing data from SurvNet, we know for every entity, which schema-version it was made. Only for Admin & National users it should be allowed to make changes to an entity, which is not from the schema version that is currently in place. Other users should see all data from this entity (let's take a case as an example), but are not able to change it. For example there are fields, that have been there in an older schema version but were dropped in the latest one. The users should see the value as readonly and are not able to edit or delete the information. If there are enums, where some items have been dropped from one version to the other, the old values should be shown grayed out and are not selectable. So it can be that an old enum is shown grayed out, but it is no longer selectable as it was only available in older versions. We do not make a difference from which version the old value is. Everything that's not included in the current schema version, will be grayed out. In the case of SurvNet: Once we move to Schema 9, Schema 8 and Schema 7 values will be grayed out, if they are no longer available. If a user decides that he wants to make changes to a case, which was created with an older version, he will be asked to convert this case to the current version.

There is no migration needed. Once we have the functionality, we just have to save the additional property schema version to the fields, values and enums which are no longer available.

In addition to this, we should add filters for the different schema versions and let the user know in the SORMAS UI, with which Schema the entity was created.

Tis is applicable for SurvNet but could also be needed, when we are talking about different SORMAS Versions.

Possible Alternatives

Additional Information

kwa20 commented 3 years ago

Further things to think about:

markusmann-vg commented 3 years ago

Further things to think about:

How do we handle imports and exports? MM: I don't see a problem with imports and exports. If there are fields mentioned which not yet exist (because of the SORMAS version, which is used and is not the latest one), the entry should be ignored. If there are fields and values, that are filled in the import but result from an older schema-version, we import them anyway as the fields are still available in SORMAS (but maybe readonly). On Export, we should always export everything, when it's a detailed export. Maybe we should think of marking fields with the information, when they refer to an older schema version.

How do we handle S2S compatibility with potentially different versions? (if applicable) MM: Should not be a problem. S2S would need the same SORMAS Version. If this is not given, S2S should not be possible

How do we handle major changes like e.g. jurisdictional changes? MM: To be refined

How do we handle major features which are not just enums and values? MM: As mentioned above. Everything is everywhere available and so it shouldnt be a problem.

How do we handle user access and edit rights depending on changes between schemes? MM: To be refined

How do we handle mobile app implications? MM: To be refined

Are schema versions dependent on interfaces? If not, how do we define different schema versions and how do we ensure compatibility with interfaces? MM: As written above. Older entries should no longer be selectable and are grayed out, however this needs to be discussed as there are most likely use cases, where it does not work like this.

kwa20 commented 3 years ago

How do we handle imports and exports? MM: I don't see a problem with imports and exports. If there are fields mentioned which not yet exist (because of the SORMAS version, which is used and is not the latest one), the entry should be ignored. If there are fields and values, that are filled in the import but result from an older schema-version, we import them anyway as the fields are still available in SORMAS (but maybe readonly). On Export, we should always export everything, when it's a detailed export. Maybe we should think of marking fields with the information, when they refer to an older schema version.

This might become more difficult if we consider major changes that require a lot of migration. I doubt that imports will then work consistently. Maybe we might then have to specify a schema version for import files

Something further to think about: Changing the schema version, specifically, how do I upgrade an entity to a new schema version?