search

OHDSI / OMOP-Standardized-Vocabularies

This repository is not longer active. It used to have the only purpose of creating releases of the Standardized Vocabularies, i.e. the content, not those of the Pallas Vocabulary Build System itself. As of 17-July-2018, vocabulary releases are also processed by Pallas. Please visit https://github.com/OHDSI/Vocabulary-v5.0/releases.
13 stars 6 forks source link

Drop in ancestor_concept when upgrading from 22-DEC-17 to 17-MAR-19 #33

Closed mark-velez closed 5 years ago

mark-velez commented 5 years ago

The All of Us Research Program has been using v5.0 22-DEC-17 and we are planning to upgrade to v5.0 17-MAR-19. We are finding that there is a drastic drop in concept_ancestor records as indicated by the query results below.

SELECT 
  'v5.0 22-DEC-17' d, 
  COUNT(1) n 
FROM `v1.concept_ancestor` 
UNION ALL
SELECT 
  'v5.0 17-MAR-19' d, 
  COUNT(1) n 
FROM `v2.concept_ancestor`
d n
v5.0 22-DEC-17 122927519
v5.0 17-MAR-19 56570086

Below is a sample of concept_ancestor records that are in v5.0 22-DEC-17 but not in v5.0 17-MAR-19 image as determined by something like the query below.

SELECT c1.concept_id a_concept_id, 
       c1.vocabulary_id a_vocabulary_id, 
       c1.concept_name a_concept_name,
       c2.concept_id d_concept_id, 
       c2.vocabulary_id d_vocabulary_id, 
       c2.concept_name d_concept_name
FROM `v1.concept_ancestor` a1
JOIN `v1.concept` c1
 ON a1.ancestor_concept_id = c1.concept_id
JOIN `v1.concept` c2
 ON v1.descendant_concept_id = c2.concept_id
WHERE NOT EXISTS
(SELECT 1 FROM `v2.concept_ancestor` a2
 WHERE a1.ancestor_concept_id = a2.ancestor_concept_id
  AND a1.descendant_concept_id = a2.descendant_concept_id)

It looks like a lot of the records missing from v5.0 17-MAR-19 are where the ancestor is NDFRT and the descendant is RxNorm Extension.

dataset vocabulary_id vocabulary_name vocabulary_version
v5.0 22-DEC-17 RxNorm RxNorm (NLM) RxNorm Full 20170807
v5.0 17-MAR-19 RxNorm RxNorm (NLM) RxNorm 20190204
v5.0 22-DEC-17 RxNorm Extension RxNorm Extension (OMOP) RxNorm Extension 15-OCT-17
v5.0 17-MAR-19 RxNorm Extension RxNorm Extension (OMOP) RxNorm Extension 2019-03-11

Should this change be expected? We are currently investigating consequences this may have for our downstream processes. I appreciate any help!

aostropolets commented 5 years ago

Yes, NDFRT was deprecated by the source(RxNorm). They are replacing it with MedFRT which is still to be OMOPed

mark-velez commented 5 years ago

Hi @aostropolets! Thank you for the quick response and useful information. For anyone who might stumble upon this ticket, I found this announcement from NLM that NDF-RT will be removed from RxNorm beginning 2019.

I don't anticipate other issues, but please forgive me as I complete my investigation of unaccounted for drops in concept_ancestor this week. I will close this ticket afterwards.

mark-velez commented 5 years ago

We found other drops, but nothing alarming. Thanks @aostropolets.