This PR adds full support for ECL v1.1.1, both syntax and evaluation.
IHTSDO ECL syntax and guide: http://snomed.org/ecl
While the syntax is fully supported, the following restrictions apply to the evaluator:
Concrete domain value refinement is evaluated against SNOMED CT Concepts (new syntax is required to distinguish between Concept and Relationship refinements, will be available as an ECL extension, mainly to support the SG drug model)
Reversed (R) data type refinements are not supported
Grouped data type refinements are not supported (the current concrete domain model does not support groupIds, like SNOMED CT Relationships)
Reversed flag (R) is not supported in attribute group
In order to support all six comparison operators (=, !=, <, >, <=, >=) for numeric values, I had to change indexing of concrete domain members. Previously the member document had only a single value field with String type. We used this field to store all four concrete domain data type values (integer, decimal, boolean, string) by converting them to their String representation. While this was okay for storing/retrieving and searching via exact match queries, this clearly can't support range queries, since Lucene will use all values available when executing range queries (true/false from boolean types, any string value and values produced by BigDecimal.toPlainString() are clearly not lexicographically sortable).
Therefore the following changes have been applied to SNOMED CT index mapping:
Replaced value field with type prefixed ones (decimalValue, integerValue, booleanValue, stringValue)
Range queries require DataType literal in order to construct the right expression (see SnomedRefSetMemberIndexEntry.Expressions.valueRange)
Instead of BigDecimal.toPlainString(), the index will encode these values with OrderedBytes.encodeNumeric() to produce sortable byte arrays, that can be stored directly in the index (both in searchable and docvalue field).
Note: IHTSDO datasets currently have zero concrete domain members, index migration (reindex) is not required.
API changes:
Added ecl query parameter to GET /{path}/concepts endpoint.
Added eclFilter property to POST /{path}/concepts/search endpoint.
Marked escg query parameter deprecated on GET /{path}/concepts endpoint.
kaicode
commented
7 years ago
This PR adds full support for ECL v1.1.1, both syntax and evaluation. IHTSDO ECL syntax and guide: http://snomed.org/ecl
While the syntax is fully supported, the following restrictions apply to the evaluator:
R) data type refinements are not supportedgroupIds, like SNOMED CT Relationships)R) is not supported in attribute groupIn order to support all six comparison operators (
=,!=,<,>,<=,>=) for numeric values, I had to change indexing of concrete domain members. Previously the member document had only a singlevaluefield with String type. We used this field to store all four concrete domain data type values (integer,decimal,boolean,string) by converting them to theirStringrepresentation. While this was okay for storing/retrieving and searching via exact match queries, this clearly can't support range queries, since Lucene will use all values available when executing range queries (true/falsefrom boolean types, any string value and values produced byBigDecimal.toPlainString()are clearly not lexicographically sortable). Therefore the following changes have been applied to SNOMED CT index mapping:valuefield with type prefixed ones (decimalValue,integerValue,booleanValue,stringValue)DataTypeliteral in order to construct the right expression (seeSnomedRefSetMemberIndexEntry.Expressions.valueRange)BigDecimal.toPlainString(), the index will encode these values withOrderedBytes.encodeNumeric()to produce sortable byte arrays, that can be stored directly in the index (both in searchable and docvalue field).Note: IHTSDO datasets currently have zero concrete domain members, index migration (reindex) is not required.
API changes:
eclquery parameter toGET /{path}/conceptsendpoint.eclFilterproperty toPOST /{path}/concepts/searchendpoint.escgquery parameter deprecated onGET /{path}/conceptsendpoint.