ACNV evaluation

[samuelklee]

• [x] get /dsde/working/slee/acs-eval/scripts/acs.sh running on gsa queue instead of using ParallelShell jobrunner
• [x] try running normal-normal pairs
• [ ] run GATK CNV, get normal/tumor HetPulldowns, and run GATK ACNV:
  • [x] STAD samples
  • [ ] amaro's CLL samples
  • [x] ignaty et al.'s samples
  • [x] LUAD (46/622, a few repeat cases w/ different normals---note queue script currently overwrites pulldown and result in such cases)
  • [x] THCA (52/511, a few repeat cases w/ different normals---note queue script currently overwrites pulldown and result in such cases)
  • [x] PRAD (50/51, one sample failing to complete)
• [x] CRSP validation (on purity series only) with revised segments
  • [x] change output of ACSModeledSegment segment files to be compatible with ModeledSegment files (unlog segment mean---i.e., transform log_2 segment-mean posterior samples appropriately---and change column headers in SegmentTableColumns, may need to use blank values for calls, depending on what the CRSP validation needs?) -- PR #327.
  • [x] check thresholds/evaluate procedure for similar-segment merging? (minor refactoring may be necessary)
  • [x] Run purity samples through GATK CNV and ACNV (incl. conversion to GATK CNV seg file)
  • [x] Run samples through CRSP validation script
• [ ] concordance with amaro's samples
  • [ ] ...
• [x] concordance with ignaty et al.'s samples
  • [x] scripts for fetching annotations and making concordance tables, plots, histograms
  • [x] concordance on MAF < 0.46 cut: As expected, MAE was only lower for ACS because of balanced segments (ACS pegs to 0.5, not sure if HAPSEG does as well). After cut, ACNV is more concordant with HAPSEG than is ACS on both MAE and RMSE. Interestingly, this is primarily due to ACNV finding more segments (which might arise from oversegmentation or from true events) that are almost all discordant with HAPSEG balanced segments (and hence, look like outliers, leading to high RMSE); however, for segments where all three are roughly concordant, ACNV's estimates of the MAF tend to be slightly more concordant with those from ACS than with those from HAPSEG.
  • [x] histograms of per-sample metrics: I've added total squared error (TSE), which is a better measure of concordance when focusing on unbalanced segments (where RMSE is misleading because you don't get penalized for finding a greater number of discordant SNPs). Especially in terms of TSE, ACNV is far more concordant with HAPSEG than is ACS on both a per-sample basis and overall.
  • [x] PRAD: Over 50 samples, TSE concordance with HAPSEG for unbalanced segments is a factor of ~4 better than ACS. MAE and RMSE are also a factor of 2 smaller.
  • [x] THCA: Over 52 samples, TSE concordance with HAPSEG for unbalanced segments is a factor of 10 better than ACS!
  • [x] LUAD: Still more HAPSEG-concordant than ACS, but metrics look a bit worse in comparison to those from PRAD and THCA (as LUAD is more active). However, TSE is dominated by a few samples (5715 and 3615, where HAPSEG misses an event with a large number of SNPs that both ACNV and ACS find; also note that the latter appears 4 times in the cohort due to multiple normals...)
• [x] spot check against ABSOLUTE?

Iteration evaluation results and TODOs for alpha:

• [x] #331 ignoring CR segments reduces oversegmentation and significantly improves concordance in conjunction with #258
  • [x] add flag to make CR segments optional---in #351
• [x] #258 segmenting on per-SNP maxLL MAF (assuming no allelic bias) saves some discordant segments that contribute significantly to RMSE
  • [x] already merged!
• [x] #333 filter on CR calls in segment union---in #351. merging CR neutral segments reduces oversegmentation---although not as much as SNP-only, concordance with HAPSEG improves.
• [x] #332 iterate on #258 by finding maxLL allelic bias with per-SNP maxLL MAF segmentation, using that to improve per-SNP maxLL MAF, and repeating until segmentation converges

[samuelklee]

@davidbenjamin @LeeTL1220 feel free to edit and expand.

[samuelklee]

/dsde/working/slee/acs-eval/scripts/plotting.py might be useful for generating plots, at least until the R/java plotting is in master. be sure to use the Python-3.4 dotkit if you want to run it on the server.

EDIT: made plots for about ~40 samples in /dsde/working/slee/acs-eval/out_case_stad_pd250_acs/acs_plots if you guys want to take a look

[samuelklee]

ran STAD normal-normal sanity check in /dsde/working/slee/acs-eval/out_case_stad_pd250_acs_normal_normal

[LeeTL1220]

Regarding Ignat et al's data: HAPSEG comparison is probably most valuable and easiest to defend.

[LeeTL1220]

@samuelklee Looks like the purity series bam files were removed. Going to try to get those back.

[samuelklee]

PRAD concordance plots/histograms now in /dsde/working/slee/acnv-eval. ACNV shows better RMSE concordance with HAPSEG than ACS (suggesting that we are less susceptible to outliers/oversegmentation); ACS shows better MAE concordance than ACNV, but they are both < 0.01. I think a case can be made to segment only on SNPs and to ignore the noisy coverage input from CNV.

[LeeTL1220]

Moved CN LOH evaluation into a separate issue.

[samuelklee]

experimented with merging CR segments that were called copy-neutral on PRAD. note that oversegmentation remains (both in largely neutral regions that remain broken up by short amps/deletions, as well as in amps and dels), but concordance with HAPSEG improves slightly for the most part. however, runtime does increase because of similar-segment MCMC iterations. i think that using sdundo could help here. going to experiment with LUAD now.

[samuelklee]

Did not get to CLL samples and left ABSOLUTE spot checking to CGA, but I think we are generally satisfied with ACNV performance. I will continue running some minor evaluations before alpha but will go ahead and close this issue.