search

National-COVID-Cohort-Collaborative / Data-Ingestion-and-Harmonization

Data Ingestion and Harmonization
41 stars 12 forks source link

No qualifier concept ids defined in the ValueSet Mapping document #55

Closed stephanieshong closed 2 years ago

stephanieshong commented 4 years ago

Could not find any valueSet mapping for the qualifier concept. Please validate if this is correct: case when lower(trim(result_qual)) = 'positive' then 45884084 when lower(trim(result_qual)) = 'negative' then 45878583 when lower(trim(result_qual)) = 'normal' then 45884153 when lower(trim(result_qual)) = 'abnormal' then 45878745 when lower(trim(result_qual)) = 'low' then 45881666 when lower(trim(result_qual)) = 'high' then 45876384 when lower(trim(result_qual)) = 'borderline' then 45880922 when lower(trim(result_qual)) = 'elevatied' then 3445772 --ssh add issue number when lower(trim(result_qual)) = 'un' then 45880480 else 45877393 end as QUALIFIER_CONCEPT_ID --null/un/elevated/boderline/ot/low/high/normal/negative/positive/abnormal

kmkostka commented 4 years ago

These are fine.

Elevated using Concept ID: 3445772 is a non-standard vocabulary (Nebraska Lexicon). That source concept maps to the Concept ID: 4328749 (https://athena.ohdsi.org/search-terms/terms/4328749) where High replaces Elevated (https://athena.ohdsi.org/search-terms/terms/40287622) as of 01/01/2013.

stephanieshong commented 4 years ago

Thank you. @kmkostka Does this mean other values are ok?

stephanieshong commented 3 years ago
LAB_RESULT_CM RESULT_QUAL ABNORMAL Abnormal See notes Gap
LAB_RESULT_CM RESULT_QUAL BORDERLINE Borderline See notes Gap
LAB_RESULT_CM RESULT_QUAL ELEVATED Elevated See notes Gap
LAB_RESULT_CM RESULT_QUAL HIGH High See notes Gap
LAB_RESULT_CM RESULT_QUAL LOW Low See notes Gap
LAB_RESULT_CM RESULT_QUAL NEGATIVE Negative See notes Gap
LAB_RESULT_CM RESULT_QUAL NI No  Information See notes Gap
LAB_RESULT_CM RESULT_QUAL NORMAL Normal See notes Gap
LAB_RESULT_CM RESULT_QUAL OT Other See notes Gap
LAB_RESULT_CM RESULT_QUAL POSITIVE Positive See notes Gap
LAB_RESULT_CM RESULT_QUAL UN Unknown See notes Gap
LAB_RESULT_CM RESULT_QUAL UNDETECTABLE Undetectable See notes Gap
LAB_RESULT_CM RESULT_QUAL UNDETERMINED Undetermined See notes Gap

What concept id should we use for undetectable and undetermined?

cyaghmour commented 3 years ago

Hi,

In her review, Kristen suggested using concept_id zero.

Thanks, Charles

From: Stephanie Hong notifications@github.com Sent: Monday, July 13, 2020 11:54 AM To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization Data-Ingestion-and-Harmonization@noreply.github.com Cc: Charles Yaghmour cyaghmour@samvit-solutions.com; Assign assign@noreply.github.com Subject: Re: [National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization] No qualifier concept ids defined in the ValueSet Mapping document (#55)

LAB_RESULT_CM RESULT_QUAL ABNORMAL Abnormal See notes Gap LAB_RESULT_CM RESULT_QUAL BORDERLINE Borderline See notes Gap LAB_RESULT_CM RESULT_QUAL ELEVATED Elevated See notes Gap LAB_RESULT_CM RESULT_QUAL HIGH High See notes Gap LAB_RESULT_CM RESULT_QUAL LOW Low See notes Gap LAB_RESULT_CM RESULT_QUAL NEGATIVE Negative See notes Gap LAB_RESULT_CM RESULT_QUAL NI No Information See notes Gap LAB_RESULT_CM RESULT_QUAL NORMAL Normal See notes Gap LAB_RESULT_CM RESULT_QUAL OT Other See notes Gap LAB_RESULT_CM RESULT_QUAL POSITIVE Positive See notes Gap LAB_RESULT_CM RESULT_QUAL UN Unknown See notes Gap LAB_RESULT_CM RESULT_QUAL UNDETECTABLE Undetectable See notes Gap LAB_RESULT_CM RESULT_QUAL UNDETERMINED Undetermined See notes Gap

What concept id should we use for undetectable and undetermined?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization/issues/55#issuecomment-657641915, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGFG2JR3M6P4TALP7W6Y4J3R3MUZTANCNFSM4OJS4TQQ.

stephanieshong commented 3 years ago

@cyaghmour - could you double confirm, and validate the following numbers?

when lower(trim(result_qual)) = 'un' then 45880480 else 45877393<--- default value ( all other cases INVALID, Not Detected, See Comments, TNP, QNS)

RESULT_QUAL ABNORMAL Abnormal 45878745
RESULT_QUAL BORDERLINE Borderline 45880922
RESULT_QUAL ELEVATED Elevated 4328749
RESULT_QUAL HIGH High 45876384
RESULT_QUAL LOW Low 45881666
RESULT_QUAL NEGATIVE Negative 45878583
RESULT_QUAL NI No  Information 46237210
RESULT_QUAL NORMAL Normal 45884153
RESULT_QUAL OT Other 46237210
RESULT_QUAL POSITIVE Positive 45884084
RESULT_QUAL UN Unknown 45877986
RESULT_QUAL UNDETECTABLE Undetectable 0
RESULT_QUAL UNDETERMINED Undetermined 45880649
RESULT_QUAL pos pos 45884084
RESULT_QUAL neg neg 45878583
RESULT_QUAL presumptive positive presumptive positive 45884084
RESULT_QUAL presumptive negative presumptive negative 45878583
RESULT_QUAL detected detected 45884084
RESULT_QUAL not detected not detected 45878583
RESULT_QUAL inconclusive inconclusive 45877990
stephanieshong commented 3 years ago

https://github.com/National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization/issues/55

Could you double confirm concept Id that I populaed on this page?

Stephanie

From: cyaghmour notifications@github.com Sent: Monday, July 13, 2020 2:04 PM To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization Data-Ingestion-and-Harmonization@noreply.github.com Cc: Stephanie Hong shong59@jh.edu; Author author@noreply.github.com Subject: Re: [National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization] No qualifier concept ids defined in the ValueSet Mapping document (#55)

  External Email - Use Caution

Hi,

In her review, Kristen suggested using concept_id zero.

Thanks, Charles

From: Stephanie Hong notifications@github.com<mailto:notifications@github.com> Sent: Monday, July 13, 2020 11:54 AM To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization Data-Ingestion-and-Harmonization@noreply.github.com<mailto:Data-Ingestion-and-Harmonization@noreply.github.com> Cc: Charles Yaghmour cyaghmour@samvit-solutions.com<mailto:cyaghmour@samvit-solutions.com>; Assign assign@noreply.github.com<mailto:assign@noreply.github.com> Subject: Re: [National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization] No qualifier concept ids defined in the ValueSet Mapping document (#55)

LAB_RESULT_CM RESULT_QUAL ABNORMAL Abnormal See notes Gap LAB_RESULT_CM RESULT_QUAL BORDERLINE Borderline See notes Gap LAB_RESULT_CM RESULT_QUAL ELEVATED Elevated See notes Gap LAB_RESULT_CM RESULT_QUAL HIGH High See notes Gap LAB_RESULT_CM RESULT_QUAL LOW Low See notes Gap LAB_RESULT_CM RESULT_QUAL NEGATIVE Negative See notes Gap LAB_RESULT_CM RESULT_QUAL NI No Information See notes Gap LAB_RESULT_CM RESULT_QUAL NORMAL Normal See notes Gap LAB_RESULT_CM RESULT_QUAL OT Other See notes Gap LAB_RESULT_CM RESULT_QUAL POSITIVE Positive See notes Gap LAB_RESULT_CM RESULT_QUAL UN Unknown See notes Gap LAB_RESULT_CM RESULT_QUAL UNDETECTABLE Undetectable See notes Gap LAB_RESULT_CM RESULT_QUAL UNDETERMINED Undetermined See notes Gap

What concept id should we use for undetectable and undetermined?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization/issues/55#issuecomment-657641915, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGFG2JR3M6P4TALP7W6Y4J3R3MUZTANCNFSM4OJS4TQQ.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNational-COVID-Cohort-Collaborative%2FData-Ingestion-and-Harmonization%2Fissues%2F55%23issuecomment-657707919&data=02%7C01%7Cshong59%40jh.edu%7Ce47be29cf9f64d078e0808d827571fd1%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302602434480105&sdata=SOtc9s6bNyeHpedv07v9CU8dd5KDHL4GDBig5t5H4yk%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOVLW5ZXFDLJVSFEMGQL72TR3NEAVANCNFSM4OJS4TQQ&data=02%7C01%7Cshong59%40jh.edu%7Ce47be29cf9f64d078e0808d827571fd1%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302602434480105&sdata=4OFUoOQVmWC6WNPpEPum%2BNL4BONKRYlXcK7oxCZsnbY%3D&reserved=0.

stephanieshong commented 3 years ago

Hi Charles, In your valueSet mapping document you usually provide all the following column I have listed below. Could you do the same thing for the following list of qualitative codes: I cannot populate the table with missing information.

CDM_TBL

CDM_TBL_COLUMN_NAME

SRC_CODE

SRC_CD_DESCRIPTION

TARGET_CONCEPT_ID

TARGET_CONCEPT_NAME

TARGET_DOMAIN_ID

TARGET_VOCABULARY_ID

TARGET_CONCEPT_CLASS_ID

TARGET_STANDARD_CONCEPT

TARGET_CONCEPT_CODE

LAB_RESULT_CM

RESULT_QUAL

ABNORMAL

Abnormal

45878745

LAB_RESULT_CM

RESULT_QUAL

BORDERLINE

Borderline

45880922

LAB_RESULT_CM

RESULT_QUAL

ELEVATED

Elevated

4328749

LAB_RESULT_CM

RESULT_QUAL

HIGH

High

45876384

LAB_RESULT_CM

RESULT_QUAL

LOW

Low

45881666

LAB_RESULT_CM

RESULT_QUAL

NEGATIVE

Negative

45878583

LAB_RESULT_CM

RESULT_QUAL

NI

No Information

46237210

LAB_RESULT_CM

RESULT_QUAL

NORMAL

Normal

45884153

LAB_RESULT_CM

RESULT_QUAL

OT

Other

46237210

LAB_RESULT_CM

RESULT_QUAL

POSITIVE

Positive

45884084

LAB_RESULT_CM

RESULT_QUAL

UN

Unknown

45877986

LAB_RESULT_CM

RESULT_QUAL

UNDETECTABLE

Undetectable

0

LAB_RESULT_CM

RESULT_QUAL

UNDETERMINED

Undetermined

0

Stephanie From: cyaghmour notifications@github.com Sent: Monday, July 13, 2020 2:04 PM To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization Data-Ingestion-and-Harmonization@noreply.github.com Cc: Stephanie Hong shong59@jh.edu; Author author@noreply.github.com Subject: Re: [National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization] No qualifier concept ids defined in the ValueSet Mapping document (#55)

  External Email - Use Caution

Hi,

In her review, Kristen suggested using concept_id zero.

Thanks, Charles

From: Stephanie Hong notifications@github.com<mailto:notifications@github.com> Sent: Monday, July 13, 2020 11:54 AM To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization Data-Ingestion-and-Harmonization@noreply.github.com<mailto:Data-Ingestion-and-Harmonization@noreply.github.com> Cc: Charles Yaghmour cyaghmour@samvit-solutions.com<mailto:cyaghmour@samvit-solutions.com>; Assign assign@noreply.github.com<mailto:assign@noreply.github.com> Subject: Re: [National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization] No qualifier concept ids defined in the ValueSet Mapping document (#55)

LAB_RESULT_CM RESULT_QUAL ABNORMAL Abnormal See notes Gap LAB_RESULT_CM RESULT_QUAL BORDERLINE Borderline See notes Gap LAB_RESULT_CM RESULT_QUAL ELEVATED Elevated See notes Gap LAB_RESULT_CM RESULT_QUAL HIGH High See notes Gap LAB_RESULT_CM RESULT_QUAL LOW Low See notes Gap LAB_RESULT_CM RESULT_QUAL NEGATIVE Negative See notes Gap LAB_RESULT_CM RESULT_QUAL NI No Information See notes Gap LAB_RESULT_CM RESULT_QUAL NORMAL Normal See notes Gap LAB_RESULT_CM RESULT_QUAL OT Other See notes Gap LAB_RESULT_CM RESULT_QUAL POSITIVE Positive See notes Gap LAB_RESULT_CM RESULT_QUAL UN Unknown See notes Gap LAB_RESULT_CM RESULT_QUAL UNDETECTABLE Undetectable See notes Gap LAB_RESULT_CM RESULT_QUAL UNDETERMINED Undetermined See notes Gap

What concept id should we use for undetectable and undetermined?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization/issues/55#issuecomment-657641915, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGFG2JR3M6P4TALP7W6Y4J3R3MUZTANCNFSM4OJS4TQQ.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNational-COVID-Cohort-Collaborative%2FData-Ingestion-and-Harmonization%2Fissues%2F55%23issuecomment-657707919&data=02%7C01%7Cshong59%40jh.edu%7Ce47be29cf9f64d078e0808d827571fd1%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302602434480105&sdata=SOtc9s6bNyeHpedv07v9CU8dd5KDHL4GDBig5t5H4yk%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOVLW5ZXFDLJVSFEMGQL72TR3NEAVANCNFSM4OJS4TQQ&data=02%7C01%7Cshong59%40jh.edu%7Ce47be29cf9f64d078e0808d827571fd1%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302602434480105&sdata=4OFUoOQVmWC6WNPpEPum%2BNL4BONKRYlXcK7oxCZsnbY%3D&reserved=0.

cyaghmour commented 3 years ago

Hi @stephanieshong ,

I am no longer the custodian of the mapping spreasheet so I can't update it. Please check with Davera. However, I have attached a stand alone file with the information you requested.

Thanks Charles lab_results_cm_results_qual-only.xlsx

stephanieshong commented 3 years ago
@DaveraGabriel @cyaghmour @kmkostka @cukarthik RESULT_QUAL UNDETECTABLE Undetectable 0

For Undetectable could we use unknown instead of 0?

stephanieshong commented 3 years ago

@DaveraGabrielhttps://github.com/DaveraGabriel @cyaghmourhttps://github.com/cyaghmour @kmkostkahttps://github.com/kmkostka @cukarthikhttps://github.com/cukarthik RESULT_QUAL

UNDETECTABLE

Undetectable

0

For Undetectable could we use unknown ( 45877393 unknown concept _id ) instead of 0 for concept id? 0 seems that we did not map to anything. Stephanie

From: cyaghmour notifications@github.com Sent: Monday, July 13, 2020 2:59 PM To: National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization Data-Ingestion-and-Harmonization@noreply.github.com Cc: Stephanie Hong shong59@jh.edu; Mention mention@noreply.github.com Subject: Re: [National-COVID-Cohort-Collaborative/Data-Ingestion-and-Harmonization] No qualifier concept ids defined in the ValueSet Mapping document (#55)

  External Email - Use Caution

Hi @stephanieshonghttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fstephanieshong&data=02%7C01%7Cshong59%40jh.edu%7C6a4731837a094571fb9908d8275ed0c4%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302636082465561&sdata=0XpjUqYg1NfRcHfcmzkVTO1itZgtbpwEfmt2imLb7JA%3D&reserved=0 ,

I am no longer the custodian of the mapping spreasheet so I can't update it. Please check with Davera. However, I have attached a stand alone file with the information you requested.

Thanks Charles lab_results_cm_results_qual-only.xlsxhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNational-COVID-Cohort-Collaborative%2FData-Ingestion-and-Harmonization%2Ffiles%2F4914544%2Flab_results_cm_results_qual-only.xlsx&data=02%7C01%7Cshong59%40jh.edu%7C6a4731837a094571fb9908d8275ed0c4%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302636082475559&sdata=dKPSgZ8t3rPSZidv3GKoL60Btz8vengb5pYveqdGNqs%3D&reserved=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNational-COVID-Cohort-Collaborative%2FData-Ingestion-and-Harmonization%2Fissues%2F55%23issuecomment-657734198&data=02%7C01%7Cshong59%40jh.edu%7C6a4731837a094571fb9908d8275ed0c4%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302636082475559&sdata=OsFcSgnC%2FX3kRwy08Y8ACVSbSPvOnKyVJIDL2bJIh5g%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOVLW53XAPLMBCQY7IL6ETTR3NKPPANCNFSM4OJS4TQQ&data=02%7C01%7Cshong59%40jh.edu%7C6a4731837a094571fb9908d8275ed0c4%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637302636082485553&sdata=XKdLAIYIfBWFJTQFyzLK%2FNVzUzCAECdocTBnOoEW5jw%3D&reserved=0.

tannerzhang commented 3 years ago

'pos' 45884084 'neg' 45878583 'presumptive positive' 45884084 'presumptive negative'45878583 'detected' 45884084 'not detected' 45878583 'inconclusive' 45877990

The mapping process is not case-sensitive and the spaces in the qualitative result fields are trimmed.

kmkostka commented 3 years ago

@stephanieshong The concept you picked is related to a LOINC code that is supposed to be used as an answer for Race values.

Which is to say: you could map it to this value but there’s a chance most end users would not know to use this VALUE_AS_A_CONCEPT when they were specifying a query. An end user would be looking to pull LOINC codes that are expected to be results for a lab test.

stephanieshong commented 3 years ago

@kmkostka - Can you be more specific? We are not using, 45877393, anywhere.

kmkostka commented 3 years ago

@stephanieshong You tagged me above and asked: “For Undetectable could we use unknown ( 45877393 unknown concept _id ) instead of 0 for concept id?”

My response is: this code is probably not appropriate.

stephanieshong commented 3 years ago

'un' map to 0 'unknown' map to 0 undetectable should be map to negative 45878583 ? @kmkostka - is this correct?

kmkostka commented 3 years ago

@stephanieshong Yes for ‘un’ and ‘unknown’. Not sure Undetected is clinically synonymous with Negative. At least in HIV research, it definitely isn’t. I’m not sure what the prevailing clinical wisdom is for COVID. @cgchute or @cukarthik, thoughts?

DaveraGabriel commented 3 years ago

My apologies for not uploading this work prior to just now. Last week, I took a stab at mapping these nulls that were produced as a result of running the DQD for one site. Had planned to discus in our mapping validation meeting scheduled for tomorrow. The urgency of the N3C soft launch has preempted this delay.
fkDomain Check Failure Concept Codes_DG_Comments6JUL2020.xlsx

@stephanieshong @kmkostka

stephanieshong commented 2 years ago

Other, No Information, and Unknowns are now mapped to 0 . (OT, NI, and UN are mapped to 0)