search

Illumina / canvas

Canvas - Copy number variant (CNV) calling from DNA sequencing data
Other
121 stars 20 forks source link

SmallPedigree-WGS System.ArgumentOutOfRangeException: Index was out of range. #119

Open curtisjamison opened 5 years ago

curtisjamison commented 5 years ago

Hi -

We encountered an error running canvas Germline WGS. The bam seems to be okay as both the strelka and Manta VCF both have data that has passed filter, and the ploidy file has been used multiple time without incident. (reference 1055 is GCHr37) Any help/advice would be greatly appreciated.

Cheers, Curt

Calling parameters: 2019-04-23T12:42:58+00:00,Running Canvas Germline-WGS 1.39.0.1598+master 2019-04-23T12:42:58+00:00,Command-line arguments: Germline-WGS -b ../sorted1055.bam --sample-b-allele-vcf ../sift.vcf.gz -o canvasOut -r /mnt/data/refData/1055/1055.kmer.fa -g /mnt/data/refData/1055 -n default -f /mnt/data/refData/1055/filter13.bed --ploidy-vcf /mnt/data/refData/1055/female-ploidy.vcf

$ cat CanvasError.txt 2019-04-23T13:17:04+00:00,ERROR: Job CNV.vcf.gz failed with exit code 134. Job logs: /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/CNV.vcf.gz.stdout /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/CNV.vcf.gz.stderr Job error message: Unhandled Exception: System.InvalidOperationException: Sequence contains no elements at System.Linq.Enumerable.First[TSource](IEnumerable1 source) at CanvasCommon.CanvasSegmentWriter.WriteVariants(IEnumerable1 segmentsOfAllSamples, List1 ploidies, GenomeMetadata genome, BgzipOrS treamWriter writer, Nullable1 denovoQualityThreshold) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasCommon\CanvasSegm entWriter.cs:line 124 at CanvasCommon.CanvasSegmentWriter.WriteSegments(String outVcfPath, List1 segments, Nullable1 diploidCoverage, String wholeGenomeFa staDirectory, String sampleName, List1 extraHeaders, PloidyInfo ploidy, Int32 qualityThreshold, Boolean isPedigreeInfoSupplied, Nullable 1 denovoQualityThreshold, Nullable1 sizeThreshold) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasCommon\CanvasSegmen tWriter.cs:line 283 at CanvasDiploidCaller.CanvasDiploidCaller.CallVariants(String variantFrequencyFile, String inFile, String outFile, String ploidyVcfPa th, String referenceFolder, String sampleName, String truthDataPath) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDip loidCaller\CanvasDiploidCaller.cs:line 290 at CanvasDiploidCaller.Program.Main(String[] args) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDiploidCaller\Prog ram.cs:line 98 2019-04-23T13:17:04+00:00,ERROR: Canvas workflow error: Isas.Framework.WorkManagement.JobFailedException: Job CNV.vcf.gz failed with exit code 134 at Isas.Framework.WorkManagement.JobLaunching.BasicJobLauncher.RunSystemProcessCheckExitAsync(JobInfo job, Action1 stdoutCallback, Ac tion1 stderrCallback) at Isas.Framework.WorkManagement.JobLaunching.RuntimeTrackingJobLauncher.LaunchJobAsync(JobInfo jobInfo, Action1 stdoutCallback, Acti on1 stderrCallback) at Isas.Framework.WorkManagement.IJobLauncherExtensions.LaunchJob(IJobLauncher jobLauncher, JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IWorkDoerJobExtensions.<>c__DisplayClass0_01.b0(WorkResources sr, IJobLauncher jobLauncher ) at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<>cDisplayClass11_01.<RunWithResources>b__3(WorkRes ources resources) --- End of stack trace from previous location where exception was thrown --- at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<RunWithResources>b__11_1[T](Task1 resourceRequestTas k, Task1 functionTask) at Isas.Framework.WorkManagement.AsTaskExtensions.Await[T](Task1 task) at Canvas.CanvasRunner.RunGermlineCalling(IFileLocation partitionedPath, CanvasCallset callset, String ploidyVcfPath, IFileLocation ca nvasSnvPath) at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.<>cDisplayClass0_0.b0() at Isas.Framework.Checkpointing.Internals.SerializingCheckpointRunner.<>cDisplayClass8_0`1.b0(Checkpoint checkpoint ) at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func`2 run) at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.RunCheckpoint(ICheckpointRunner runner, String name, Action a) at Canvas.CanvasRunner.CallSampleInternal(CanvasCallset callset) at Canvas.CanvasRunner.CallSample(CanvasCallset callset) at Canvas.ModeLauncher.Launch()

eroller commented 5 years ago

Germline-WGS is deprecated. Please run SmallPedigree-WGS, even for a single sample.

curtisjamison commented 5 years ago

Hmm. Okay, thanks. Will switch to that.

curtisjamison commented 5 years ago

The SmallPedigree-WGS workflow also failed:

2019-04-24T21:16:29+00:00,ERROR: Job default.partitioned failed with exit code 134. Job logs: /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/default.partitioned.stdout /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/default.partitioned.stderr Job error message: Unhandled Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index at CanvasCommon.Utilities.Quartiles(List1 x) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasCommon\Utilities.cs:line 382 at CanvasPartition.HiddenMarkovModelsRunner.Run(List1 segmentation, Boolean isPerSample) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasPartition\HiddenMarkovModelsRunner.cs:line 45 at CanvasPartition.CanvasPartition.Main(String[] args) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasPartition\CanvasPartition.cs:line 170 2019-04-24T21:16:29+00:00,ERROR: Canvas workflow error: Isas.Framework.WorkManagement.JobFailedException: Job default.partitioned failed with exit code 134 at Isas.Framework.WorkManagement.JobLaunching.BasicJobLauncher.RunSystemProcessCheckExitAsync(JobInfo job, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.JobLaunching.RuntimeTrackingJobLauncher.LaunchJobAsync(JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IJobLauncherExtensions.LaunchJob(IJobLauncher jobLauncher, JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IWorkDoerJobExtensions.<>cDisplayClass0_01.<DoWork>b__0(WorkResources sr, IJobLauncher jobLauncher) at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<>c__DisplayClass11_01.b3(WorkResources resources) --- End of stack trace from previous location where exception was thrown --- at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.b11_1[T](Task1 resourceRequestTask, Task1 functionTask) at Isas.Framework.WorkManagement.AsTaskExtensions.Await[T](Task1 task) at Canvas.CanvasRunner.InvokeCanvasPartitionMultisample(SmallPedigreeCallset callsets, List1 cleanedPaths, String canvasBedPath, String commonCnvsBed) at Isas.Framework.Checkpointing.Internals.SerializingCheckpointRunner.<>c__DisplayClass8_0`1.b0(Checkpoint checkpoint) at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func`2 run) at Canvas.CanvasRunner.CallSampleInternal(SmallPedigreeCallset callset) at Canvas.CanvasRunner.CallPedigree(SmallPedigreeCallset callset) at Canvas.ModeLauncher.Launch()

curtisjamison commented 5 years ago

Hi Guys -- any word on having the same failure with small-pedigree?

eroller commented 5 years ago

This error is happening because there don't appear to be any coverage values to operate on to perform segmentation.

Can you check the input files to the partition command that is failing? I believe it is the .cleaned file that contains the coverage values. Is that file empty? We have seen weird behavior when there was a mismatch between the chromosome names in the bam vs. the reference genome. Make sure they are all the same convention (e.g. hg19 vs GRCh37).

curtisjamison commented 5 years ago

Hi Eric - We've set up a pipeline of Isaac/Strelka/Canvas/Manta. The BAM is directly from Isaac and all using the same GRCh37 reference. So far we've run 500+ files through the pipeline with only this one failure.

Looking at the input files, I see no real coverage values in the default.cleaned.

ll default.cleaned -rw-r--r-- 1 root root 20 Apr 24 21:16 default.cleaned more default.cleaned 

eroller commented 5 years ago

Is this a reproducible error or maybe something strange going on with the filesystem?

If this is reproducible then the coverage data must be getting dropped somewhere. Does the .binned file have the expected coverage data. For WGS it should be hundreds of MB in size.

curtisjamison commented 5 years ago

Hi Eric - This is very reproducible -- it has happened on two different virtual machines and on multiple retries.

The bin job only runs for a minute: Job default_0.binned duration: 00:01:14.1 and the .binned file is empty, but the chromosome files are populated. ll canvasOut3/TempCNV total 3165264 drwxr-xr-x 2 root root 4096 May 3 21:22 ./ drwxr-xr-x 6 root root 4096 May 3 21:02 ../ -rw-r--r-- 1 root root 20 May 3 21:22 default_0.binned -rw-r--r-- 1 root root 280406991 May 3 21:03 default_0_chr1.dat -rw-r--r-- 1 root root 152476636 May 3 21:04 default_0_chr10.dat -rw-r--r-- 1 root root 151882376 May 3 21:04 default_0_chr11.dat -rw-r--r-- 1 root root 150583427 May 3 21:04 default_0_chr12.dat -rw-r--r-- 1 root root 129566158 May 3 21:05 default_0_chr13.dat -rw-r--r-- 1 root root 120768278 May 3 21:05 default_0_chr14.dat -rw-r--r-- 1 root root 115347859 May 3 21:05 default_0_chr15.dat -rw-r--r-- 1 root root 101649143 May 3 21:06 default_0_chr16.dat -rw-r--r-- 1 root root 91344657 May 3 21:06 default_0_chr17.dat -rw-r--r-- 1 root root 87836947 May 3 21:06 default_0_chr18.dat -rw-r--r-- 1 root root 66520151 May 3 21:21 default_0_chr19.dat -rw-r--r-- 1 root root 273599337 May 3 21:03 default_0_chr2.dat -rw-r--r-- 1 root root 70903753 May 3 21:06 default_0_chr20.dat -rw-r--r-- 1 root root 54146177 May 3 21:21 default_0_chr21.dat -rw-r--r-- 1 root root 57717682 May 3 21:21 default_0_chr22.dat -rw-r--r-- 1 root root 222775276 May 3 21:03 default_0_chr3.dat -rw-r--r-- 1 root root 215048603 May 3 21:03 default_0_chr4.dat -rw-r--r-- 1 root root 203529710 May 3 21:03 default_0_chr5.dat -rw-r--r-- 1 root root 192504493 May 3 21:04 default_0_chr6.dat -rw-r--r-- 1 root root 179031038 May 3 21:04 default_0_chr7.dat -rw-r--r-- 1 root root 164659567 May 3 21:05 default_0_chr8.dat -rw-r--r-- 1 root root 158865152 May 3 21:05 default_0_chr9.dat

eroller commented 5 years ago

So something odd is happening when merging the bin data from each of the chromosomes. Can you check the Logging directory to see if there is anything in stderr or stdout for the CanvasBin invocations?

curtisjamison commented 5 years ago

Is the bin count supposed to have a minus sign in front?

04/24/2019 08:57:09 Deserialization complete -2147483648 binSize 04/24/2019 08:57:31 Launch BinCountsForChromosome jobs... 04/24/2019 08:57:39 Completed BinCountsForChromosome jobs. 04/24/2019 08:57:39 Output binned counts: 04/24/2019 08:57:39 Output complete

eroller commented 5 years ago

-2147483648 is int.MinValue and is probably the result of overflow in the bin calculation. I haven't seen this before and I haven't been able to figure out how it could happen. If you are able to share the binned data I could dig deeper to see what is happening.

curtisjamison commented 5 years ago

Thanks for looking into this deeper. I can share the binned data. Please let me know which folder to tar up and I'll get onto a share drive for you.

eroller commented 5 years ago

I should only need the .dat files. Can you also send the last command line for CanvasBin.dll . It should contain each of the .dat files along with some other parameters.

eroller commented 5 years ago

@osowiecki That might be unrelated to this issue. Can you create a separate issue and include additional log messages including the contents of any files mentioned in the error log.

osowiecki commented 5 years ago

@osowiecki That might be unrelated to this issue. Can you create a separate issue and include additional log messages including the contents of any files mentioned in the error log.

It must be related to https://github.com/Illumina/canvas/issues/92 And you can find my extended post there.