search

SORMAS-Foundation / SORMAS-Project

SORMAS (Surveillance, Outbreak Response Management and Analysis System) is an early warning and management system to fight the spread of infectious diseases.
https://sormas.org
GNU General Public License v3.0
292 stars 142 forks source link

Add configs to ignore sex and birtdate for duplicate detection of persons #7759

Open SORMAS-JanBoehme opened 2 years ago

SORMAS-JanBoehme commented 2 years ago

Feature Description

Add a possibility for allowing the health departments to make the duplicate detection for persons ignore some of the values that are being used at the moment. Without having to completely rework the system.

Problem Description

The health departments often face the problem that sex or birtdate are entered incorrectly by the users or in a DEMIS lab message which results in persons not being detectd as a duplicate.

They are asking for a possibility to make the duplicate detection more forgiving.

Proposed Change

Add new server side configurations:

Both are set to "true" as a default value.

If they are set to "false" the duplicate detection for persons will no longer take into account the corresponding information for a person to count as a possible duplicate.

This will result in more possible duplicates found but the feedback from the health departments is that they would rather have too many possible duplicates than too few.

Possible Alternatives

Additional Information

@kwa20 Any comment?

kwa20 commented 2 years ago

@Jan-Boehme only comment would be that if this issue gets implemented before #5583, we should also make this configurable via the UI, preceding the variance options.

bernardsilenou commented 2 years ago

@Jan-Boehme Since age and sex can be registered with error, then any then attribute used in the duplicate detection can also be entered with error, right? What about instead adding a config to use only names in duplicate person detection. Even if name is entered with error, the similarity threshold can take care of that.

SORMAS-JanBoehme commented 2 years ago

@bernardsilenou Only deactivating everything instead of name is not granular enough in my opinion, Even when talking about deactivating birthdate the feedback was mixed. some liked it and some didn't. this is why I split it up into their own configs.

But you are right that if we introduce configs for sex and birthdate we probably should also be able to configure the other two options. They aren't really used in the health departments but being able to deactivate them could be interesting for users outside germany.

I'll edit the issue description to reflect that.

marko-arn commented 2 years ago

@Jan-Boehme @bernardsilenou Thanks for the issue. We are also requesting that change as we are using DEMIS since yesterday and some labs are sending the sex "anderes" - "others", which is in Sormas Germany "divers". The related case cannot be selected during the import of the lab message as the sex is not the same. The workaround is only to search the person if there is one and then change the sex there, import the lab message and change the sex back. The disadvantage is that the person search is very slow for normal users.

bernardsilenou commented 2 years ago

@Marko-ilmkreis @kwa20

kwa20 commented 2 years ago

@Marko-ilmkreis @bernardsilenou If the person really has a divers sex, this should be detected as a duplicate as long as it is not set to either male or female in the existing person. I suspect that the actual problem here is that labs set the sex to other even though it's meant to be unknown. If it would be unknown, actual sexes like male, female and other would also be detected as duplicates. Other will already be mapped in the adapter.

marko-arn commented 2 years ago

@kwa20 @bernardsilenou I think the problem is multiple. It starts when someone fill in the test request for the lab not properly (d (divers) instead leave it blank) or the lab fills it wrong into their system. There is not only one problem and some days we have it very often from one lab and the other day only one or two errors. It would be good to have an option to ignore the sex for the processing of duplicate detection.