search

OpenConceptLab / ocl_issues

Issues for all OCL repos. NOTE: Install ZenHub Browser Extension and request access to the OCL Roadmap board to view all issues and to contribute
4 stars 1 forks source link

CIEL v2024-07-26 release #1908

Closed bmamlin closed 1 month ago

bmamlin commented 1 month ago

Staging

Production

bmamlin commented 1 month ago

First version where retired mappings and version creation were embedded into bulk import. Import worked; however, version creation stalled (seeding concepts/mappings were not successful). @snyaggarwal plans to debug tomorrow.

bmamlin commented 1 month ago

@snyaggarwal,

Production version finished processing and when I dowloaded the export to validate, I found there were 11 mappings missing:

{
  "missing_mappings": [
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/110818/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "76360ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "15938005"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/115193/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "170867ABBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "109962001"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/155666/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "40699ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "15253005"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/156095/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "40760ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "87715008"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/156479/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "BROADER-THAN", "owner_type": "Organization", "external_id": "41714ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "73465006"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/156709/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "40909ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "60968001"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/156721/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "41503ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "37224001"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/157387/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "41145ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "64710004"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/161440/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/Regenstrief/sources/LOINC/", "owner": "CIEL", "map_type": "BROADER-THAN", "owner_type": "Organization", "external_id": "146666ABBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "20407-3"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/161442/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/Regenstrief/sources/LOINC/", "owner": "CIEL", "map_type": "BROADER-THAN", "owner_type": "Organization", "external_id": "146682ABBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "22702-5"},
    {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/163613/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "280183ABBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "315107005"}
  ],
}

When you debug the import, maybe it will show why these didn't make it into the version. Let me know if these can be added to the version and a new export generated or if I need to delete the version, re-import these 11 mappings, and then create the version again.

snyaggarwal commented 1 month ago

@bmamlin I checked these 11 mappings they all are existing mappings with different map_type. If you take the example of first: {"retired": false, "from_concept_url": "/orgs/CIEL/sources/CIEL/concepts/110818/", "type": "Mapping", "source": "CIEL", "to_source_url": "/orgs/IHTSDO/sources/SNOMED-CT/", "owner": "CIEL", "map_type": "NARROWER-THAN", "owner_type": "Organization", "external_id": "76360ABBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "to_concept_code": "15938005"}, There is an existing mapping between 110818 and 15938005 which "SAME-AS". Now the import file is trying to update (not create) it to "NARROWER-THAN". In order to update the mapping the importer needs to find the mapping. Importers finds the mapping by from/to and map_type and since map_type is "NARROWER-THAN" it cant find the mapping and tries to create it, but fails with the error "There can be only one mapping between two concepts". This happens because there is no "id" of mapping in the import file.

I would suggest you will first have to retire these existing 11 mappings and then run just these 11 mappings import. Let me know if I can help in this

bmamlin commented 1 month ago

these 11 mappings they all are existing mappings with different map_type

Ok. @askanter must have changed the types of these mappings. I'll delete the existing version, retire these mappings, import them with their new map types, and then – assuming everything matches as expected – release the version.

askanter commented 1 month ago

Yes, thanks. There are plenty of examples of SAME-AS maps which are being changed to something else.

bmamlin commented 1 month ago

I ended up just manually changing each of these mappings in OCL directly and am now generating a new export of HEAD for validation on Production.

bmamlin commented 1 month ago

I'm seeing extra mappings when validating (what I got from Andy & what is on OCL are not matching exactly yet), so the retired mappings included in the bulk import didn't work as expected, my diff script is getting confused by the extra entries, or something else is going on. I'm going to have to find some time to debug.

Sorry, @askanter, but this is going to delay the release of CIEL v2024-07-26 until I can sort out the discrepancies.

bmamlin commented 1 month ago

Ok. I went through every resource in the diff. It looks like my approach of taking extra mappings from OCL, setting "retired": true, and slapping them at the top of the bulk import didn't work. For example:

{"smart":"dfdf56f98e8d0b35867f0257bddc60f8","standard":"f1fb93a67cc80c48c3d0a2c82cb4a54c"},"external_id":"280183ABBBBBBBBBBBBBBBBBBBBBBBBBBBBB","retired":true,"map_type":"SAME-AS","source":"CIEL","owner":"CIEL","owner_type":"Organization","from_concept_code":"163613","from_concept_name":null,"from_concept_url":"/orgs/CIEL/sources/CIEL/concepts/163613/","to_concept_code":"315107005","to_concept_name":null,"to_concept_url":null,"from_source_owner":"CIEL","from_source_owner_type":"Organization","from_source_url":"/orgs/CIEL/sources/CIEL/","from_source_name":"CIEL","to_source_owner":"IHTSDO","to_source_owner_type":"Organization","to_source_url":"/orgs/IHTSDO/sources/SNOMED-CT/","to_source_name":"SNOMED-CT","url":"/orgs/CIEL/sources/CIEL/mappings/313797/","version":"7933325","id":"313797","versioned_object_id":313797,"versioned_object_url":"/orgs/CIEL/sources/CIEL/mappings/313797/","is_latest_version":true,"update_comment":null,"version_url":"/orgs/CIEL/sources/CIEL/mappings/313797/7933325/","uuid":"7933325","version_created_on":"2024-06-28T00:28:12.053721Z","from_source_version":null,"to_source_version":null,"from_concept_name_resolved":"Cryptococcus Sp Ag presence in serum test","type":"Mapping","sort_weight":null,"version_updated_on":"2024-06-28T00:28:12.063441Z","version_updated_by":"ocladmin","latest_source_version":null,"created_on":"2024-06-28T00:28:12.053721Z","updated_on":"2024-06-28T00:28:12.063441Z","created_by":"ocladmin","updated_by":"ocladmin","public_can_view":true}

I probably need to pull out a portion of the properties instead of just changing the retired property from false to true.

In any case, there were 29 differences: I fixed 11 of them by changing the map types manually in OCL and, for the remaining 18, I manually extracted the list and ran my patch script to retired them. So, hopefully, things should be aligned again. And this was all due to unsuccessfully switching to retiring within the bulk import. Mea culpa.

I've triggered a new export of HEAD on both Staging & Production and will validate tomorrow (probably after my clinic).

@snyaggarwal, any idea what might have caused the versions to stall? Because it happened on both Staging & Production, I wouldn't be surprised if it was a side effect of triggering the version from a bulk import.

snyaggarwal commented 1 month ago

@bmamlin I couldn't find any relevant app failures on this, need to look at the infra failures (memory and such). Triggering a version creation from bulk import is a common practice. We have many tests against it. I think it failed due to OOM because the version creation and export all run in sync in the bulk importer.

bmamlin commented 1 month ago

CIEL v2024-07-26 validated & released on both Staging & Production.