Complete PanelApp parser rewrite

[tskir]

Part 1

• [x] Remove old & partially refactored modules
• [x] Merge existing code into one module
• [x] Unify argument parsing
• [x] Unify Spark session init
• [x] Optimise imports
• [x] Fix formatting errors and separators prior to phenotype splitting
• [x] Reimplement handling of records with no meaningful phenotypes
• [x] Remove extra information prior to ontology extraction
• [x] Reimplement extraction of ontology information

Part 2

• [x] Separate extraction for OMIM and other ontologies
• [x] Reimplement post splitting phenotype cleanup
• [x] Construct final ontology identifiers for OnToma querying

Part 3

• [x] Investigate how the publications are represented in the API
• [x] Implement PubMed ID parsing and cleanup
• [x] Collect and join literature references
• [x] Save output as a single JSON file

Part 4

• [x] Produce final ontology identifiers
• [x] Populate final evidence string structure
• [x] Optimise imports
• [x] Consistently use null values instead of empty strings
• [x] Fix validation issues
• [x] Internal code review
• [x] Update the locked environment
• [x] Test run for the updated pipeline
• [x] Write changelog
• [x] Submit pull request

Address comments, iteration 1A

• [x] Investigate and resolve evidence string duplication issues
• [x] Fix & improve PMID parsing
• [x] Add examples of PMID matching patterns

Address comments, iteration 1B

• [x] Comment on the use of a temporary directory
• [x] Irene's comments on discussion points
• [x] Discuss pre-splitting cleanup
• [x] Fix filtering by panel version
• [x] Output phenotype cleanup for debugging purposes
• [x] Check and amend phenotype cleanup regular expressions
• [x] Output PMID cleanup for debugging purposes
• [x] Check and amend PMID cleanup regular expressions

Address comments, iteration 1C

• [x] Look into removing duplicate entries from cohortPhenotypes
• [x] Reimplement using the panel names as fallback
• [x] Compile the latest debugging data for phenotypes and PMIDs
• [x] Describe the changes and resubmit the PR for review

Remaining

• [x] Compare old and new evidence strings
• [x] Extract data reporting to a separate issue
• [x] Investigate and fix missing confidence column
• [x] Address comments, iteration 2
• [x] Finalise and merge the PR

[tskir]

List of data issues to report back

• Multiple formats are used to specify OMIM identifiers:
  • 103580
  • MIM 615553
  • MIM# 615249 (that's a tab character used as a separator)
  • OMIM 610549
  • OMIM #612073
  • OMIM:612186
  • OMIM: 617236
  • OMIM #617767
  • OMIM# 618658
• Multiple formats for specifying other ontologies:
  • ORPHA90636
  • Orphanet:453499
  • OrphaNet: ORPHA404454
• Multiple formats are used when both the string and the OMIM identifier are present:
  • #107480:Townes-Brocks syndrome
  • #601869: Deafness, autosomal recessive 15
  • Cenani-Lenz syndactyly syndrome 212780
  • Cenani-Lenz syndactyly syndrome, 212780
  • Centronuclear myopathy 5 ( 615959)
  • Cerebellar ataxia, areflexia, pes cavus, optic atrophy and sensorineural hearing loss (CAPOS, #601338)
  • {Amyotrophic lateral sclerosis, susceptibility to, 13}, 183190
• Typos and formatting errors in specific disease names
  • • "Usher syndrome, type 1D, 601067;Deafness, autosomal recessive; 12, 601386;Usher syndrome, type 1D/F digenic, 601067" — Disease name is using a semicolon, which breaks phenotype splitting
  • Combined oxidative phosphorylation deficiency 6, 300816Cowchock syndrome, 310490 — Two diseases are glued to each other
  • 17,20-lyase deficiency, isolated 202110 and 17-alpha-hydroxylase/17,20-lyase deficiency 202110 — Two diseases in one, separated by “and”
• Additional unstructured information present in phenotype strings
  • Meier-Gorlin syndrome with craniosynostosis (from PMID 27374770)
  • Jervell and Lange-Nielsen syndrome [Congenital sensorineural hearing loss, Prolonged QT interval on EKG, SyncopeTorsades de pointes, Sudden cardiac death, Caused by mutation in the potassium voltage-gated channel, KQT-like subfamily, member 1 gene (KCNQ1), Caused by mutation in the potassium voltage-gated channel, Isk-related subfamily, member 1 gene (KCNE1)]
  • 22q11.2 deletion syndrome, Orphanet:567 (includes developmental delay)
• Phenotype names starting with or containing ? with unclear meaning:
  • 616553 ?Dyskeratosis congenita 6 and 7
  • 3- ?-hydroxysterol ?5-oxidoreductase/isomerase deficiency (Disorders of bile acid biosynthesis)
• Very different ways of describing the ontologies:
  • Epileptic encephalopathy, intellectual disability, no OMIM# yet
• Inconsistent separators (; with and without leading/trailing spaces)

Different ways of specifying the publications in the API

98(6):1193-207. doi: 10.1016/j.ajhg.2016.05.004. PubMed PMID: 27259053, PubMed Central PMCID: PMC4908191.
doi:10.​1007/​s12265-016-9673-5
DOI: https://doi.org/10.1016/j.xhgg.2021.100033
https://doi.org/10.1101/797787
https://doi-org.ezproxy.library.qmul.ac.uk/10.1093/brain/awaa085
PMID: 26933893
PMID: 27078007 (full text not available to confirm findings).
Aldahmesh (2012) Genet Med 14(12):955-962, PMID: 22935719
12702164
15985586 (two siblings)
16060907 (Camilot et al., 2005 report subclinical hypothyroid subjects with heterozygous substitutions
25674101 - review from the same authors as PMID:23972370

2194867118000911 — but displayed as two separate ones: https://panelapp.genomicsengland.co.uk/panels/81/gene/FAM20C/#!details