Weekly Monday Morning Error Report 9/16/2024

[chrisroederucdenver]

Check and summarize errors on last run in CCDA_OMOP_by_Pthon

• [ ] pull main and master branches from github and foundry, check they are synched. (We should be working in Foundry on the master branch, so this should just be a matter of pulling from both, merging master into main and pushing main back to github). Merging main to master should not do anything if we haven't made changes on main (github).
  • [ ] merge and push as necessary
• [ ] check status of github actions on main branch. Are they all good?
  • [ ] Run data_driven_parse.py
  • [ ] Run bin/process.sh (layer_datasets.py and the duckdb load)
  • [ ] Run bin/compare_output.sh
  • [ ] count errors from the comparison
  • [ ] Check the logs/load_omop.log
  • [ ] Scan logs/logdomain_.log
  • [ ] Update any correct files
• [ ] if not, which files failed? List here.

[chrisroederucdenver]

2024-09-30 from Foundry Workspace

Summary

• 72 for (trying to) mapping the values set for Encounter Activity code, EncounterType to make our way to OMOP visit_concept_id

  • visit needs visit_occurrence_id
  • visit needs visit_type_concept_id
  • pretty sure visit_concept_id is wrong (there's an issue for this)

• 83 involves the provider_id and person_id issues

• 109 should get us to the race, ethnictiy and gender concepts.

• 28 has a comment specifically addressing value_as_concept_id (set to 0 for no matching concept)

  • Measurement needs value_as_concept_id

Details

• PR with minor updates created, waiting for Adam
• Duckdb load failures files/sections
  • Person needs race, ethnicity and gender, person_ids
  • Failed to load Person from eHX_Terry.xml__Person.csv
    • race_concept_id NULL
  • Failed to load Person from CCDA_CCD_b1_InPatient_v2.xml__Person.csv
    • person_id PK
  • Failed to load Person from healtheconnectak-ccd-20210226.2.xml__Person.csv
    • race_concept_id NULL
  • Failed to load Person from C-CDA_R2-1_CCD_modified.xml__Person.csv
    • person_id PK
  • Failed to load Person from anna_flux.xml__Person.csv
    • race_concept_id NULL
  • Failed to load Person from bennis_shauna_ccda.xml__Person.csv
    • race_concept_id NULL
  • Failed to load Person from CCD-Sample.xml__Person.csv
    • person_id PK
  • Visit NEEDS provider_id and visit_concept_id
    • Failed to load Visit from CCDA_CCD_b1_Ambulatory_v2.xml__Visit.csv HAS OUTPUT
    • non-integer provider_id
    • Failed to load Visit from CCDA_CCD_b1_InPatient_v2.xml__Visit.csv HAS OUTPUT
    • non-integer provider_id
    • Failed to load Visit from 170.314b2_AmbulatoryToC.xml__Visit.csv HAS OUTPUT
    • NULL visit_concept_id
    • Failed to load Visit from ToC_CCDA_CCD_CompGuideSample_FullXML.xml__Visit.csv HAS OUTPUT
    • visit_occurrence_id not found
  • Other files don't create visit rows: anna_flux.xml, bennis_shauna_ccda.xml, C-CDA_R-21_CCD.xml and the modified version, CCD-Sample.xml, eHX_Terry.xml, healtheconnectak-ccd-2021-226.2.xml, Patient_502.xml
  • Measurement NEEDS VALUE_AS_CONCEPT
  • Failed to load Measurement from 170.314b2_AmbulatoryToC.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from Patient-502.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from C-CDA_R2-1_CCD_modified.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from CCDA_CCD_b1_Ambulatory_v2.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from ToC_CCDA_CCD_CompGuideSample_FullXML.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from CCDA_CCD_b1_InPatient_v2.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from bennis_shauna_ccda.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from C-CDA_R2-1_CCD.xml__Measurement.csv
    • value_as_concept_id not found
  • Failed to load Measurement from CCD-Sample.xml__Measurement.csv
    • value_as_concept_id not found
  • no output from anna_flux.xml, eHX_Terry.xml, healtheconnectak-CCD-20210225.2.xml
  • Observations OK
  • no output from most, C-CDA_R2-1_CCD_modified.xml and Patient-502.xml have output and load.

[AdamLeeIT]

@chrisroederucdenver does the metadata.py default': (0,'default') resolve the race_concept_id NULL problems?

    'race_concept_id':{
        'config_type': 'DERIVED',
        'FUNCTION': VT.map_hl7_to_omop_concept_id,
        'argument_names': {
            'concept_code': 'race_concept_code',
            'vocabulary_oid': 'race_concept_codeSystem',
            **'default': (0,'default')**
        }

[chrisroederucdenver]

"does" or "should"? The first question is answered by looking at the output, which shows the problem, and the code that was run to get that, which has the default config in it. Since it's not running right, it doesn't seem to resolve the issue. You should be able to check this for yourself. Give it a shot, happy to review and discuss.

The second seems reasonable, I have to read the code to verify.

from value_transformations.py `def _map_to_omop_concept_row(vocabulary_oid, concept_code, default): """ """ try: concept_id_df = concept_df[(concept_df['oid'] == vocabulary_oid) & (concept_df['concept_code'] == concept_code)]

    if len(concept_id_df) < 1:
       logger.error(f"no concept for \"{vocabulary_oid}\" \"{concept_code}\" ")
       return default

    if len(concept_id_df) > 1:
       logger.warning(f"more than one  concept for \"{vocabulary_oid}\" \"{concept_code}\", chose the first")
       return default

    return concept_id_df
except IndexError as e:
    logger.warning(f"no concept for \"{vocabulary_oid}\" \"{concept_code}\" type:{type(e)}")
    return default

def map_hl7_to_omop_concept_id(args_dict): """ expects: vocabulary_oid, concept_code """ return int(_map_to_omop_concept_row(args_dict['vocabulary_oid'], args_dict['concept_code'], args_dict['default'])\ ['concept_id'].iloc[0]) `

It looks like that was the intent. Not sure why it's not working. Still a problem.

[chrisroederucdenver]

116 fixed some bugs in the above code and will make next week's analysis look better with regard to race and ethnicity.