Retrospective duplicate detection & resolving [2+3]

[ghost]

@MartinWahnschaffeSymeda We have a problem with duplicate records, where first name, last name repeats itself after picking or matching person. Also when there is a mistake in a letter or word, it saves as a new case, whereas it was only the same case. E.g Taiwo Edito is a case, and another person created Taiwo Edoto, but it is one and the same person due to spelling error. Can we have an algorithm that checks based on 5 key indicator match (first name, last name, date of birth or age, district, region and if possible health facility to match), then call out for duplicates

[ghost]

@MartinWahnschaffeSymeda Please can you include an active/inactive function to all cases so that atleast for now, we can deactivate cases which appear as duplicates after moving case from one lga to another or duplicates. This would go a long way to solve inaccurate case counts due to duplicates. Especially now that we have not yet classified and not a case on the dashboard

[MartinWahnschaffe]

@dta16 Quickfix: there is a delete function in the new version for admins (see #383).

In additon to this we need two things:

1. A way to further reduce duplicated, by making the name/birthdate comparison more fault-tolerant by applying a similarity algorithm (#432)
2. Build an interface that allows searching for duplicates in cases and contacts and merging them together. Merging of cases/contacts has to reflect in the data model -> dont delete the merged case/contact!

[MartinWahnschaffe]

• [x] create a mockup on how to do duplicate detection

[MartinWahnschaffe]

Inspiration:

• 

[ghost]

@MartinWahnschaffeSymeda I like this concept... We should talk about it during our next sprint..next tuesday

[MartinWahnschaffe]

Thoughts:

• query should be based on the existing case query (with filters and sorting)
• it joins with all cases that fullfill the following criteria:
  • were created at a later point in time
  • disease and LGA is equal
  • report date is within same date range (e.g. +/- one month)
  • name is similar: would be great to use the same or a similar algorithm as we are using when creating a case. May be difficult to implement in SQL. See also #757
• I would ignore date of birth and health facility for now
• query does paging (10 per page or so). Put results into a container -> "Load more"
• display either in TreeGrid or a custom component
• create a new view for this. Move filters to a component (CaseCriteria...)
• include archived cases

Review notes:

• date of birth should also show age
• add gender
• make it easy to access info on reporting user, so it can be called to ask
• add column with completeness: determine how to calculate this -> maybe later
• make names bold
• would be great to have a popup to display data of both cases next to each other. Generic solution possible?
• add "pick" button to ignore data of other case
• filter to cases created within a certain time period
• would be good to use additional data of the person to get rid of false positives (e.g. occupation)
• users may need to change the time period for the report date filter
• sorting would be nice

---

• report date for web users should be empty be default (but still required)

[MartinWahnschaffe]

Main todos:

1. Add logic to return a number of case duplicate pairs based on #757 search algorithm. Likely a list of CaseIndexDto pairs.
2. Display pairs in tree grid and do page based loading of additional cases
3. Do merge #1232

Optional todos:

4. Switch pair entries to have the "better" one first

[MateStrysewske]

• [x] Don't use pagination for now; we might not need it
• [x] Add a "Dismiss" button that removes the case in question from the container (will appear again on next reload, but might help with data management)
• [x] Let users open the cases in a new tab
• [x] Re-colorize the lines when sorting

[MateStrysewske]

• [x] Make sure report date is empty by default
• [x] Make similarity reference value configurable (properties)
• [x] Case creation: add checkbox column (or radio buttons ideally)
• [x] Change "Dismiss" to "Hide"
• [x] Take out investigation status, outcome and point of entry from the grid

[MateStrysewske]

• [x] Change Long value in getCasesForDuplicateMerging to actual calculation
• [x] Change query to retrieve CaseIndexDtos by using two queries that return a list (one with all parent elements, one with all child elements)