Data processing and quality checks

[jshoughtaling]

Implement the quality report tooling, based on the quality reports package and built into a deployable image.

See below for a proposal of what those reports may eventually include.

[jshoughtaling]

Proposed Structure

Section 1: Metadata and Comparison with Prior Deliveries

• Metadata about delivery

  • Date and time delivered
  • Packet size
  • Number of prior deliveries
  • Feedback from most recent delivery

• Table with counts & percentages, with cells colored from red to green

  • # files delivered
  • # standard data quality check pass/fail & severity
  • # potential issues with PHI per modality
  • # chorus-specific quality checks with severity
  • # CHoRUS-specific characteristics
  • …

• Target # of patients to be contributed from site

  • # and % of target patients with any data overall and by modality (+delta from previous submission)
  • # and % of target patients with complete data overall and by modality (+delta from previous submission)
  • # and % of target patients with complete and quality approved data overall and by modality (+delta from previous submission)
  • # and % of target patients with complete and quality approved and deidentified data overall and by modality (+delta from previous submission)
  • # and % of target patients who conform with the general CHoRUS cohort definition

• Table with count differential from prior delivery

  • Tables will be created as csv files stored alongside the delivery directory for reference and calc of diff

Section 2: PHI / Deidentification Checks Across Modalities

• Table with findings from OMOP-Based PHI Checks
• Table with findings from Imaging PHI Checks
• Table with findings from Waveform PHI Checks
• Table with findings from Note PHI checks

Section 3: OMOP Data Quality

• Format similar to report proposed by Claire, with drilldown on failed checks and additional context
• Integrate CHoRUS-specific checks in the same format

Section 4: CHoRUS Characterization

• # and % of concepts of interest and delta from last time
• Patients that fall into select phenotypes of interest
• Visit grouping and richness

Section 5: Comparison with Consortium

• Re-package some of the Ares network overviews and place into the report, highlighting the respective datasite
  • How many events per domain
  • Unmapped concepts
  • Number of releases and subjects