Enhance Search and Add mappings into concept search

[paynejd]

Right now, there is no way to get concept results when using a map code as search criteria. Currently, you can search mappings using a map code and use that to get to the concept, but that is a multi-step approach that requires users to know where to search.

I would like to be able to search using a code, e.g. "A15.1", and any concepts that actually have that code along with any concepts mapped to that code, with SAME-AS mappings weighted more highly than other types of mappings.

[paynejd]

MVP requirements:

• Just target map codes (for direct/forward mappings)
• Add target map code to full text search
• Also implement an API parameter that only searches the map codes (separate from full-text search)
• Option to not match map codes? Default is to include map code results
• Facets: Consider for map types and target sources if low level of effort, otherwise move this to round 2

Weighting of search results:

• Weight "SAME-AS" and it's variants higher
• All other map types weighted lower and the same
• Rough order:
  • Exact ID/mnemonic
  • Exact external ID
  • Exact name
  • Exact synonym
  • Exact map code
  • Partial ID
  • Partial name/synonym
  • Partial map code
  • Partial external ID (negative score?)

Search results will become noisier:

• Sunny to experiment with enabling ES highlight feature to provide context for search results, especially for map codes and descriptions, because these don't show up in the response
  • https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html

Separate design activity:

• Implement a "Make this search better" button -- e.g. user provides a brief description of what they were looking for and submits (along with their search history) and we can use that optimize

Out of scope:

• Not searching target concept name (e.g. matching a target concept name that is not a synonym of the from-concept)
• Not indexing indirect mappings
• Ability to filter map type

[snyaggarwal]

@paynejd @jamlung-ri This is deployed on QA with many other improvements on Search, explained here

[paynejd]

This is such a big improvement! Searches for “Malaria” and “B50” (and many others) had way better results than before. Bold formatting of the matching search phrase made looking through results much more efficient.

Feedback for discussion:

• Get feedback from Paul on behavior and icon to display more info about how a match was made
• Since we have specific info about what matched
• “Coombs” returns 3 concepts
• Search for “Coombs” inside CIEL:
  • Search result 159606 – can we add the locale for the matching name, and still only highlight the matching portion of the name?
  • coombs tès endirèk [ht] ⇒ Indirect antiglobulin test in serum or plasma [en]
  • Search result 159607 “Direct Coombs Test” does not show up if you change the search criteria to “antiglobulin” – are synonyms included in the search?

[jamlung-ri]

Overall, this appears to be working SO much better!

Thoughts:

• Do we need a "fuzzy matched" formatting? For example, maybe we bold the exact matches, but we bold/italizice fuzzy matches? See screenshot where this could be helpful

• We'll want to work on the language for "Matches to a non-Same-As mapping"

• Does matching on a name mean that OCL won't highlight the matches in its mappings? Example: the name has a highlight, but shouldn't the mappings highlight "male" too?

Bad results:

• Searching for a LOINC code as a mapping for CIEL is returning pretty bad results e.g. searching "1527-1" (I ecpected to see 163703 Glucose, serum 1/2 hr post 75g oral glucose (mg/dL)):

Interesting results (not necessarily bad?):

• Fuzzy matching appears to be happening on fields like ID. Example: Search "0867005" and get these results -
  

• Searching "R53" returns some results that are "Narrower-than" mappings, not "Same-As" -

• Fuzzy matching appears to be happening on mapped codes too. Example: searching "V80.9" returns a result with a mapping for "A80.9"

• Related to the previous bullet, it seems weird that searching "A80.9" doesn't return concepts with mappings to "V80.9". I expected to see "133902 Motor Vehicle Traffic Accident due to Loss of Control"

[snyaggarwal]

This is such a big improvement! Searches for “Malaria” and “B50” (and many others) had way better results than before. Bold formatting of the matching search phrase made looking through results much more efficient.

Feedback for discussion:

• Get feedback from Paul on behavior and icon to display more info about how a match was made
• Since we have specific info about what matched
• “Coombs” returns 3 concepts

• Search for “Coombs” inside CIEL:

  • Search result 159606 – can we add the locale for the matching name, and still only highlight the matching portion of the name?
  • coombs tès endirèk [ht] ⇒ Indirect antiglobulin test in serum or plasma [en]
  • Search result 159607 “Direct Coombs Test” does not show up if you change the search criteria to “antiglobulin” – are synonyms included in the search?

@paynejd

• We cannot add locale as part of the highlight in the table view, because we load less in listing and not the verbose view. The locales are part of the verbose view.
• "antiglobulin" should show 159607 “Direct Coombs Test” in results, looks like a bug. I will check this. We do index/search on synonyms.

[snyaggarwal]

Search result 159607 “Direct Coombs Test” does not show up if you change the search criteria to “antiglobulin” – are synonyms included in the search?

@paynejd this is fixed

[snyaggarwal]

Searching for a LOINC code as a mapping for CIEL is returning pretty bad results e.g. searching "1527-1" (I ecpected to see 163703 Glucose, serum 1/2 hr post 75g oral glucose (mg/dL)):

@jamlung-ri this is fixed.

[snyaggarwal]

@jamlung-ri Replies inline:

Searching "R53" returns some results that are "Narrower-than" mappings, not "Same-As" -

If you search for "R53" Narrower-than results are returned after "Same-As", we do search on other map types, but score them lower than Same-As

Fuzzy matching appears to be happening on mapped codes too. Example: searching "V80.9" returns a result with a mapping for "A80.9"

Yes we do fuzzy match on mapped codes as well. Refer to the document. We may want to discuss which fields to fuzzy match on.

Related to the previous bullet, it seems weird that searching "A80.9" doesn't return concepts with mappings to "V80.9". I expected to see "133902 Motor Vehicle Traffic Accident due to Loss of Control"

The fuzzy match algo works on number of expansions it can perform. Expansion mean "Maximum number of variations created.". We kept this number very low (currently 2), otherwise there will be too many fuzzy matches, especially in "name". So you may see more closer fuzzy matches in "mapped codes" and "ID" vs "name", since "name" have too many characters. We may want to talk about if we want to keep different expansion number for different fields.

[jamlung-ri]

@snyaggarwal We got some good feedback on the OpenMRS/OCL Squad call today. We're definitely getting closer!

One big piece in particular: the "?" icon probably won't be adequate. We need to show a small preview of the matching code. Jon gave similar feedback in this comment.

Other feedback:

• Might be prettier to use "→" instead of "==>" for synonyms - maybe use HTML's " &rArr "
• Shouldn't lowercase the synonym if possible (note the inconsistency in the circled areas of the screenshot)

[snyaggarwal]

@jamlung-ri Both the feedbacks were added on QA. For the second one, had to trigger re-indexing of concepts, which is happening as I write this. So once that done, it should reflect the right case of synonyms.

[paynejd]

Searching for b50 on QA brought up a couple wonkier results: https://app.qa.openconceptlab.org/#/search/?q=b50

1. The first set of results is a partial word match (e.g. "B50-196") -- can we make highlighting smart enough to still highlight "B50" here?
2. "B50" matched a concept ID of "150" -- didn't we decide to disable this type of matching for concept IDs?
3. Seems like the same-as matches (id-testing/6 and clonetest4/11) should be boosted higher than the fuzzy name match of 050

[snyaggarwal]

@paynejd QA concepts are re-indexing.

[snyaggarwal]

@paynejd

1. This is a bug -- Checked in fix.
2. We dint decide (yet!) to change that behavior of fuzzy matching on ID.
3. same-as matches (id-testing/6 and clonetest4/11) are also fuzzy matched and they are scored lower than core attributes of concept.

[paynejd]

@snyaggarwal Thanks for quick response! Adding the second point for discussion to our architecture call...

[snyaggarwal]

@paynejd @jamlung-ri This is deployed on Staging.

[snyaggarwal]

@paynejd @paynejd Updated QA after removing fuzzy search on ID and map codes.

[paynejd]

This is working really well, including the latest change in QA.

We need design input (from @paulsonder?) on the question mark behavior (the "Why this matched?" popup) before this moves forward to production.

[paulsonder]

In general I'm finding the user experience on this a little confusing but it's quite advanced in terms of development now. I've copied out the requirements and I'm thinking about how they apply in V3.

With regards to the pop-up, my initial thought was around proximity and how we could serve the content in a tooltip when the row was hovered instead...

[snyaggarwal]

• [x] Change to a tooltip instead of a pop up
• [x] Display the code/text that was matched in the tooltip, in addition to the type of match (e.g. “SAME-AS”)
• [x] Change to “info” icon instead of “help” icon

[snyaggarwal]

• [x] Change to a tooltip instead of a pop up
• [x] Display the code/text that was matched in the tooltip, in addition to the type of match (e.g. “SAME-AS”)
• [x] Change to “info” icon instead of “help” icon

@paynejd @jamlung-ri this is deployed on QA

[paynejd]

@snyaggarwal the latest change looks great - let's deploy to production!

[snyaggarwal]

@paynejd This is deployed everywhere.

[paynejd]

Wahoo!!!

On Sat, Aug 5, 2023 at 11:38 PM Sny @.***> wrote:

@paynejd https://github.com/paynejd This is deployed everywhere.

— Reply to this email directly, view it on GitHub https://github.com/OpenConceptLab/ocl_issues/issues/1583#issuecomment-1666698278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJCOOO25FK2HSGYHLM4KZDXT4GTRANCNFSM6AAAAAAYGOLRZE . You are receiving this because you were mentioned.Message ID: @.***>