search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
209 stars 84 forks source link

Snowstorm ECL return inactive concepts #616

Open liquid36 opened 3 months ago

liquid36 commented 3 months ago

Hi! I'm dealing with a posible issue. When I request snowstorm for an ECL, it return inactive concepts.

I tried it in my local machine and in the Snomed Browser. I imported this version SnomedCT_InternationalRF2_PRODUCTION_20240301T120000Z.zip into MAIN.

ECL: ^446608001 

curl -X 'GET' \
  'http://localhost:8080/MAIN/concepts?activeFilter=true&termActive=true&language=es&ecl=%5E446608001&includeLeafFlag=false&form=inferred&returnIdOnly=true&offset=0&limit=10000' \
  -H 'accept: application/json' \
  -H 'Accept-Language: en'
https://browser.ihtsdotools.org/snowstorm/snomed-ct/MAIN/2024-03-01/concepts?activeFilter=true&termActive=true&language=es&ecl=%5E446608001&includeLeafFlag=false&form=inferred&returnIdOnly=true&offset=0&limit=10000
{
  "items": [ 
   ....
    "243955001",
    "312539002",
    "243079002",
    "77208007",
    "6541001",
    "162119005",
    "314926009",
    "263390006",
    "243931006",
    "368597006",
    "362291003",
    "331610001",
    "20008000",
    "69515008",
    "272689002",
    "367629006",
    "149885003",
    "58477004",
    "36064001",
    "182051002",
    "243946004",
    "174363005",
    "699982007",
    "245793003",
    "280244007",
  ],
  "total": 23600,
  "limit": 10000,
  "offset": 0,
  "searchAfter": "WzI3ODg5MTAwM10=",
  "searchAfterArray": [
    278891003
  ]
}

In the response, i found the concept 362291003 which is inactive in the current branch.

https://browser.ihtsdotools.org/snowstorm/snomed-ct/browser/MAIN/2024-03-01/concepts/362291003?descendantCountForm=inferred

{
  "conceptId" : "362291003",
  "descendantCount" : 0,
  "fsn" : {
    "term" : "Entire prosencephalon (body structure)",
    "lang" : "en"
  },
  "pt" : {
    "term" : "Entire prosencephalon",
    "lang" : "en"
  },
  "active" : false,
  "effectiveTime" : "20240101",
  "released" : true,
  "releasedEffectiveTime" : 20240101,
  "inactivationIndicator" : "AMBIGUOUS",
  "associationTargets" : {
    "POSSIBLY_EQUIVALENT_TO" : [ "263353005", "730443005" ]
  },
  "moduleId" : "900000000000207008",
  "definitionStatus" : "PRIMITIVE",
  "descriptions" : [ {
    "active" : true,
    "moduleId" : "900000000000207008",
    "released" : true,
    "releasedEffectiveTime" : 20170731,
    "descriptionId" : "748214019",
    "term" : "Entire prosencephalon (body structure)",
    "conceptId" : "362291003",
    "typeId" : "900000000000003001",
    "acceptabilityMap" : {
      "900000000000509007" : "PREFERRED",
      "900000000000508004" : "PREFERRED"
    },
    "type" : "FSN",
    "lang" : "en",
    "inactivationIndicator" : "CONCEPT_NON_CURRENT",
    "caseSignificance" : "CASE_INSENSITIVE",
    "effectiveTime" : "20170731"
  }, {
    "active" : true,
    "moduleId" : "900000000000207008",
    "released" : true,
    "releasedEffectiveTime" : 20170731,
    "descriptionId" : "479899012",
    "term" : "Entire prosencephalon",
    "conceptId" : "362291003",
    "typeId" : "900000000000013009",
    "acceptabilityMap" : {
      "900000000000509007" : "PREFERRED",
      "900000000000508004" : "PREFERRED"
    },
    "type" : "SYNONYM",
    "lang" : "en",
    "inactivationIndicator" : "CONCEPT_NON_CURRENT",
    "caseSignificance" : "CASE_INSENSITIVE",
    "effectiveTime" : "20170731"
  } ],
  "annotations" : [ ],
  "classAxioms" : [ {
    "axiomId" : "be80f815-c5a0-4db9-b1fe-9c2e215d7772",
    "moduleId" : "900000000000207008",
    "active" : false,
    "released" : true,
    "definitionStatusId" : "900000000000074008",
    "relationships" : [ {
      "active" : true,
      "moduleId" : "900000000000207008",
      "released" : false,
      "sourceId" : "362291003",
      "destinationId" : "81861006",
      "typeId" : "116680003",
      "type" : {
        "conceptId" : "116680003",
        "active" : true,
        "definitionStatus" : "PRIMITIVE",
        "moduleId" : "900000000000012004",
        "fsn" : {
          "term" : "Is a (attribute)",
          "lang" : "en"
        },
        "pt" : {
          "term" : "Is a",
          "lang" : "en"
        },
        "id" : "116680003"
      },
      "target" : {
        "conceptId" : "81861006",
        "active" : false,
        "definitionStatus" : "PRIMITIVE",
        "moduleId" : "900000000000207008",
        "fsn" : {
          "term" : "Prosencephalon structure (body structure)",
          "lang" : "en"
        },
        "pt" : {
          "term" : "Prosencephalon structure",
          "lang" : "en"
        },
        "id" : "81861006"
      },
      "groupId" : 0,
      "modifier" : "EXISTENTIAL",
      "characteristicType" : "STATED_RELATIONSHIP"
    } ],
    "id" : "be80f815-c5a0-4db9-b1fe-9c2e215d7772",
    "definitionStatus" : "PRIMITIVE",
    "effectiveTime" : 20240101
  } ],
  "gciAxioms" : [ ],
  "relationships" : [ {
    "active" : false,
    "moduleId" : "900000000000207008",
    "released" : true,
    "releasedEffectiveTime" : 20190731,
    "relationshipId" : "3993335023",
    "sourceId" : "362291003",
    "destinationId" : "81861006",
    "typeId" : "116680003",
    "type" : {
      "conceptId" : "116680003",
      "active" : true,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000012004",
      "fsn" : {
        "term" : "Is a (attribute)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Is a",
        "lang" : "en"
      },
      "id" : "116680003"
    },
    "target" : {
      "conceptId" : "81861006",
      "active" : false,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000207008",
      "fsn" : {
        "term" : "Prosencephalon structure (body structure)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Prosencephalon structure",
        "lang" : "en"
      },
      "id" : "81861006"
    },
    "groupId" : 0,
    "modifier" : "EXISTENTIAL",
    "characteristicType" : "STATED_RELATIONSHIP",
    "effectiveTime" : "20190731",
    "id" : "3993335023"
  }, {
    "active" : false,
    "moduleId" : "900000000000207008",
    "released" : true,
    "releasedEffectiveTime" : 20240101,
    "relationshipId" : "179597027",
    "sourceId" : "362291003",
    "destinationId" : "81861006",
    "typeId" : "116680003",
    "type" : {
      "conceptId" : "116680003",
      "active" : true,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000012004",
      "fsn" : {
        "term" : "Is a (attribute)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Is a",
        "lang" : "en"
      },
      "id" : "116680003"
    },
    "target" : {
      "conceptId" : "81861006",
      "active" : false,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000207008",
      "fsn" : {
        "term" : "Prosencephalon structure (body structure)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Prosencephalon structure",
        "lang" : "en"
      },
      "id" : "81861006"
    },
    "groupId" : 0,
    "modifier" : "EXISTENTIAL",
    "characteristicType" : "INFERRED_RELATIONSHIP",
    "effectiveTime" : "20240101",
    "id" : "179597027"
  }, {
    "active" : false,
    "moduleId" : "900000000000207008",
    "released" : true,
    "releasedEffectiveTime" : 20190131,
    "relationshipId" : "721005025",
    "sourceId" : "362291003",
    "destinationId" : "258335003",
    "typeId" : "123005000",
    "type" : {
      "conceptId" : "123005000",
      "active" : false,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000012004",
      "fsn" : {
        "term" : "Part of (attribute)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Part of",
        "lang" : "en"
      },
      "id" : "123005000"
    },
    "target" : {
      "conceptId" : "258335003",
      "active" : true,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000207008",
      "fsn" : {
        "term" : "Entire brain (body structure)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Entire brain",
        "lang" : "en"
      },
      "id" : "258335003"
    },
    "groupId" : 0,
    "modifier" : "EXISTENTIAL",
    "characteristicType" : "ADDITIONAL_RELATIONSHIP",
    "effectiveTime" : "20190131",
    "id" : "721005025"
  }, {
    "active" : false,
    "moduleId" : "900000000000207008",
    "released" : true,
    "releasedEffectiveTime" : 20190131,
    "relationshipId" : "721004026",
    "sourceId" : "362291003",
    "destinationId" : "57991002",
    "typeId" : "123005000",
    "type" : {
      "conceptId" : "123005000",
      "active" : false,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000012004",
      "fsn" : {
        "term" : "Part of (attribute)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Part of",
        "lang" : "en"
      },
      "id" : "123005000"
    },
    "target" : {
      "conceptId" : "57991002",
      "active" : true,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000207008",
      "fsn" : {
        "term" : "Entire embryo (body structure)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Entire embryo",
        "lang" : "en"
      },
      "id" : "57991002"
    },
    "groupId" : 0,
    "modifier" : "EXISTENTIAL",
    "characteristicType" : "ADDITIONAL_RELATIONSHIP",
    "effectiveTime" : "20190131",
    "id" : "721004026"
  }, {
    "active" : false,
    "moduleId" : "900000000000207008",
    "released" : true,
    "releasedEffectiveTime" : 20190131,
    "relationshipId" : "721006029",
    "sourceId" : "362291003",
    "destinationId" : "83418008",
    "typeId" : "123005000",
    "type" : {
      "conceptId" : "123005000",
      "active" : false,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000012004",
      "fsn" : {
        "term" : "Part of (attribute)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Part of",
        "lang" : "en"
      },
      "id" : "123005000"
    },
    "target" : {
      "conceptId" : "83418008",
      "active" : true,
      "definitionStatus" : "PRIMITIVE",
      "moduleId" : "900000000000207008",
      "fsn" : {
        "term" : "Entire fetus (body structure)",
        "lang" : "en"
      },
      "pt" : {
        "term" : "Entire fetus",
        "lang" : "en"
      },
      "id" : "83418008"
    },
    "groupId" : 0,
    "modifier" : "EXISTENTIAL",
    "characteristicType" : "ADDITIONAL_RELATIONSHIP",
    "effectiveTime" : "20190131",
    "id" : "721006029"
  } ],
  "alternateIdentifiers" : [ ],
  "validationResults" : [ ]
}

What am i missing?

kaicode commented 3 months ago

The ECL ^ "Member of" operator selects the current members of a given reference set. Regardless of if the concept itself is active or not. However it will not include concepts referenced in inactive refset member rows.

This is the active member of the SNOMED CT to ICD-10 reference set that references the inactive concept in your example: https://browser.ihtsdotools.org/snowstorm/snomed-ct/MAIN/2024-03-01/members?referenceSet=446608001&referencedComponentId=362291003

The Member of operator results in different behaviour to the hierarchy based ECL operators. When using hierarchy operators only active concepts are returned because inactive concepts have no active relationships, and are therefore not part of the hierarchy. However an inactive concept can still be a member of a reference set.

The inactive concepts within a reference set can be found using the member operator and the concept filter. For example:

^ 446608001 |SNOMED CT to ICD-O simple map reference set|  {{ C active = false }}

This SNOMED CT to ICD-O map contains 82 inactive concepts in the latest (August 2024) International Edition.

There are some reference sets that only contain inactive concepts, for example the inactivation indicator and historical association reference sets.

I hope that helps.

liquid36 commented 3 months ago

I tried the concept filter in the ECL and it worked fine. What is the purpose of activeFilter param?

Thanks.

kaicode commented 3 months ago

You have a good point. The activeFilter is supposed to work, even when running ECL.

It looks like there is a bug: the active filtering is not being applied when using the returnIdOnly parameter.

These two requests, with the same ecl and activeFilter give a different results count:

The returnIdOnly parameter is intended for more advanced users who need better performance. Applying the ECL concept filter {{ C active = false }} is a good workaround.