Add pipeline for benchmarking SVs

[rickymagner]

This issue serves as a discussion hub for development around a pipeline for benchmarking SVs. Development currently is happening on the rm_benchmark_svs branch. There you can find the following useful documents:

• A README which contains documentation on the current state of the pipeline and visualization tools included.
• A Roadmap which contains checklists of items to do in the near future before the first major release.

If you have comments, issues using the pipeline, etc. please add them to this thread. I will post updates here during development as well.

[samuelklee]

Thanks again for the demo just now, @rickymagner! Things are looking great.

I've just shared the kage-lite-dev-1 Terra workspace with you. This currently hosts WDLs for running KAGE, PanGenie, and a leave-one-out evaluation that calculates weighted genotype concordance of imputed short-read genotypes with the panel genotypes for the left-out sample (stratified by length and genomic context, although I also have some offline versions that stratify by AF as well).

The workspace bucket contains some panel VCFs that may be interesting to throw through your prototype:

• 32 HGSVC2 LR assemblies: gs://fc-947cb896-7721-4f4d-8747-902d3a3394ee/20200825_pav-all-samples-freeze3-final.vcf.gz
• 3202 SR 1kG + high coverage trios SR samples (chr1; in the bucket, you'll also see versions with some population annotations stripped out, subset to AF >= 0.001, symbolic alleles dropped, etc.): gs://fc-947cb896-7721-4f4d-8747-902d3a3394ee/1kGP_high_coverage_Illumina.chr1.filtered.SNV_INDEL_SV_phased_panel.vcf.gz

And now some runs that contain genotyped VCFs from KAGE and PanGenie. For these, you'll find VCFs in corresponding shards for each left-out sample:

• call-KAGEPlusGLIMPSECase/../[SAMPLE_NAME].kage.glimpse.vcf.gz for the result of KAGE genotyping + GLIMPSE imputation
• call-KAGEPlusGLIMPSECase/../[SAMPLE_NAME].kage.vcf.gz for the result of KAGE genotyping
• call-call-PanGenieCase/../[SAMPLE_NAME]_genotyping.vcf.gz for the result of PanGenie

You'll also find corresponding plots in the call-CalculateMetrics* folders (these are the 1D plots I've been showing in recent meetings).

Here are the runs:

• 11 left-out samples vs. the HGSVC2 LR panel over chr1+2
• 1 left-out sample vs. the 1kG+trios SR panel over 10Mbp of chr1

As we discussed, it would be great to see the panels compared against each other, as well as the various combinations of panel vs. case. Might also be interesting to check each against your current gold standard set as well (although I may need to actually run that particular sample to generate the case VCFs!)

Finally, here's a (very rough first cut) of the sort of benchmarking heatmap that might be useful: I was planning on adding some annotations to each cell, but hovering tooltips would be even better!

[samuelklee]

@rickymagner just a heads up, I'm cleaning up the workspace and deleting a bunch of runs, including the two I linked above. Here are some more recent runs that might be useful for you:

HGSVC2 32 panel samples, 10 cases, chr1: 18e46b03-f34f-4e22-9a10-01e4ec6f6b8d 1kg 3202 panel samples, AF>=0.001, 10 cases, chr1: 9d99a3c8-f3fe-4f29-8b66-0eccecb04770 1kg 127 panel samples, AF >= 0.02, 10 cases, chr1: ecaf79f2-9785-436e-a603-989864c54444 HGSVC2 32 panel samples, 1 HG00731 case, chr1-22+chrX: 721c121d-932f-4ee4-9f71-6ed1591b3f74

[rickymagner]

Completed in #159