Test data model

[MattWellie]

Fixes

• We don't have a good way of generating highly customisable test data in MT format which can run through the whole AIP process
• Each unit test does just enough to prove itself valid, without using realistic data
• One fully featured data engine could satisfy E2E runs of AIP, and unit testing, and everything in between

Proposed Changes

• Creates reanalysis.data_model, which contains a model for each component piece of annotation data; gnomad genomes/exomes, cadd, spliceAI, clinvar, transcript_consequences, some global fields, and entries (variant calls)
• Other than variant coordinates, alleles, and gene ID, all of these fields will populate with generic data (should satisfy all AIP logic without being 'interesting')
• We can override all fields at any granularity to produce any variant data we want (without any need to simulate real data, or run VEP)
• Fully featured (?) - this will mock all the annotation, as many variants and samples as you like, phased genotypes are interpreted correctly... the whole shebang.
• Not currently capable of making coffee, but I'm working on it.

Intention:

"I want to create a variant which AIP should classify as Cat3 & Cat5, which will satisfy a Biallelic MOI"

- Create a variant with high SpliceAI scores
- Create one or more transcript consequences which satisfy Cat3 (missense+, lof:HC or Clinvar+), and have a geneID assc. With a biallelic MOI
- Create as many samples as you like, each with a corresponding genotype
- Write out as a MT
- Write out as a VCF

Corner case:

• I want a pair of variants in the same gene (set the gene ID), which are Clinvar Pathogenic (set the Clinvar details) and should be picked up as Cat. 1.
• The gene should only function with a Biallelic MOI (create fake PanelApp data, or set the gene ID to something autosomal & biallelic)
• I want 3 samples
  • 1 is HomVar for one variant, WT for the other - should be reported
  • 1 has a compound het - should be reported
  • 1 is het for both variants, but they are in phase, so we won't expect it in the report

All of this should be doable in ~20 lines of code, using model_go_brrrr.py as a template.

Voila! You have some appropriately formatted data for running through the whole process 🧙

nb. doesn't currently do the pedigree, or panelapp data.

Checklist

• [ ] Related Issue created
• [ ] Tests covering new change
• [x] Linting checks pass

[MattWellie]

The three obvious(?) ways to generate a full MT from the annotation state:

• MT.from_parts(): explicitly state all rows, columns, entries, and use the relevant constructor
• MT.from_rows_table(): translate the HT to a MT without columns, then add columns later?
• MT from import_vcf(), then add annotations through a row-wise join: easiest, least customisable

[MattWellie]

https://hail.is/docs/0.2/hail.MatrixTable.html#hail.MatrixTable.from_parts may not actually exist in the codebase..

[MattWellie]

https://github.com/hail-is/hail/blob/9f7ede5e455af7111101b55923a2601361c434f5/hail/python/hail/table.py#L3295

This looks much more promising