wardle / hermes

A library and microservice implementing the health and care terminology SNOMED CT with support for cross-maps, inference, fast full-text search, autocompletion, compositional grammar and the expression constraint language.
Eclipse Public License 2.0
177 stars 20 forks source link

Search using constraint with refinements times out #46

Closed rlustemberg closed 1 year ago

rlustemberg commented 1 year ago

The Health Ministry from Argentina advises to use the following constraint when searching within generic medications: <763158003: 732943007 =*, [0..0] 774159003=*. I've noticed that using such a constraint on the AR Edition of SNOMED causes the request to time out. Removing the second refinement reduce search time to around 15 secs (docker container on a i7 8core iMac). So it seems to be properly interpreting the ECL. I wonder if this is an indexing issue. Except for this only problem, we are in awe regarding the performance of this terminology server. Orders of magnitude more performance and less footprint. Also the full text search with it's fuzzy fallback is amazing.

wardle commented 1 year ago

Thanks for reporting this. I've fixed the ECL parsing to build a different query when a wildcard is used like this in a refinement when the wildcard is used as the value of an attribute. Instead of trying to realise all concepts (which is what a wildcard expression usually does), because this is in the context of a refinement, we can instead cheat and query for results with that attribute independent of the value. This means that the results will be fast.

However, I have not been able to test this with your full expression as I only have access to the UK edition. I'd be interested to know whether your full expression expands correctly, and whether you have tested this with any other terminology servers. It may be that the handling of a [0...0] cardinality attribute needs additional work.

wardle commented 1 year ago

Re-opening as a cardinality of [0..0] not providing correct results. Looks like an easy fix however for this special case within parse-attribute--expression in which we'll have to turn it into a query to not include results with a count of the specified attribute(s).

wardle commented 1 year ago

Fixes included in v1.2.1032 release

rlustemberg commented 1 year ago

Thank you so much. I'll do some tests and let you know how is it all working

rlustemberg commented 1 year ago

Just tested locally. 45ms total response time. Headers response time 45ms. Processing time seems negligible. Great stuff!

wardle commented 1 year ago

That's great to hear. Thank you.