ERROR: Exception caught in WorkDoerFactory

[ssri28]

Hi Team,

I am getting error in running Canvas germline caller.

ERROR: ERROR: Exception caught in WorkDoerFactory. Cancelling all jobs. Exception: One or more errors occurred. (One or more errors occurred. (Job tabix-index-tmp-ballele.bedgraph.gz.1 failed with exit code 139))

cmd: /home/ec2-user/dotnet/dotnet /home/ec2-user/data/Canvas-1.40.0.1613+master_x64/Canvas.dll SmallPedigree-WGS -b L2.sorted.rmdup.bam --sample-b-allele-vcf L2.HC.vcf -n MSC --ploidy-vcf canvas_ploidy.vcf -o MSC_cnv -r /home/ec2-user/data/genome/hg19/hg38.fa -g /home/ec2-user/data/genome/hg19/ -f exclude_intervals.bed

Kindly help me to fix the bug.

thanks a lot

[eroller]

run chmod +x on the tabix executable

[ssri28]

Hi, Thanks for the reply. chmod +x on the tabix executable did not work.

[ssri28]

Hi, Thanks for the reply. chmod +x on the tabix executable did not work.

Still i am getting same error

On Mon, 4 Nov 2019 at 23:41, Eric Roller notifications@github.com wrote:

run chmod +x on the tabix executable

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Illumina/canvas/issues/134?email_source=notifications&email_token=AKXA3TTSOCTQZSVZSAB654TQSBQUDA5CNFSM4JIQYGN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAGBVI#issuecomment-549478613, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKXA3TQ3T6KWHFJVZO7UMNLQSBQUDANCNFSM4JIQYGNQ .

[eroller]

The std out should show the command that is failing. Can you make sure this is the same tabix executable you updated permissions on. There may be more than one tabix executable. You can also just try: chmod -R 777 /home/ec2-user/data/Canvas-1.40.0.1613+master_x64

[ssri28]

Hi eroller,

I even tried it..

Error i am getting

ERROR: Exception caught in WorkDoerFactory. Cancelling all jobs. Exception: Type specified in JSON 'CanvasCommon.CommandLineParsing.OptionProcessing.SuccessfulParsingResult1[[Canvas.CommandLineParsing.SmallPedigreeInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null' is not compatible with 'CanvasCommon.CommandLineParsing.OptionProcessing.IParsingResult1[[Canvas.CommandLineParsing.GermlineWgsInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null'. Path '$type', line 2, position 157. Newtonsoft.Json.JsonSerializationException: Type specified in JSON 'CanvasCommon.CommandLineParsing.OptionProcessing.SuccessfulParsingResult1[[Canvas.CommandLineParsing.SmallPedigreeInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null' is not compatible with 'CanvasCommon.CommandLineParsing.OptionProcessing.IParsingResult1[[Canvas.CommandLineParsing.GermlineWgsInput, Canvas, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null]], CanvasCommon, Version=1.40.0.1613, Culture=neutral, PublicKeyToken=null'. Path '$type', line 2, position 157. at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolveTypeName(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, String qualifiedTypeName) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) at Isas.Framework.Checkpointing.Internals.CheckpointJsonSerializer.Deserialize[T](IFileLocation path) at Isas.Framework.Checkpointing.Internals.CheckpointJsonSerializer.Load[T](String checkpoint) at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func2 run) at Canvas.CommandLineParsing.ModeParser1.Parse(MainParser main, ILogger logger, ISettings settings, ICheckpointRunner checkpointRunner, IWorkDoer workDoer, String[] args, TextWriter standardWriter, TextWriter errorWriter) at Canvas.CommandLineParsing.MainParser.<>cDisplayClass20_2.b2(ICheckpointRunner checkpointer) at Isas.Framework.Checkpointing.CheckpointerFactory.<>cDisplayClass1_1.b1(CheckpointManager checkpointManager) at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>cDisplayClass0_02.<SupplyArgument>b__1(TFill arg) at Isas.Framework.Checkpointing.Internals.CheckpointManagerFactory.<>c__DisplayClass4_01.b0() at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>cDisplayClass0_02.<SupplyArgument>b__0() at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>c__DisplayClass0_02.b1(TFill arg) at Isas.Framework.Utilities.TaskTrackerFactory.<>cDisplayClass2_01.<SupplyArgument>b__0() at Isas.Framework.Utilities.IArgumentSupplierExtensions.<>c__DisplayClass0_02.b0() at Isas.Framework.WorkManagement.JobLaunching.JobLauncherFactory.RunWithJobLauncher(ILogger logger, ISettings settings, IDirectoryLocation loggingDir, Action1 logCommand, CancellationToken cancellationToken, Action1 function) at Isas.Framework.WorkManagement.JobLaunching.JobLauncherFactory.RunWithJobLauncher(ILogger logger, ISettings settings, IDirectoryLocation analysisFolder, CancellationToken cancellationToken, Action1 function) at Isas.Framework.WorkManagement.ResourceManagement.WorkResourceManagerFactory.RunWithResourceManager(ILogger logger, ISettings settings, CancellationToken cancellationToken, Action1 function) at Isas.Framework.WorkManagement.WorkDoerFactory.RunWithWorkDoer(ILogger logger, ISettings settings, IDirectoryLocation analysisFolder, CancellationTokenSource cancellationTokenSource, Action1 function) at Canvas.CommandLineParsing.MainParser.<>c__DisplayClass20_0.<Run>b__0(ILogger logger) at Isas.Framework.FrameworkFactory.IsasFrameworkFactory.RunWithLogger(IFileLocation workflowLog, IFileLocation workflowError, Action1 function) at Canvas.CommandLineParsing.MainParser.Run(String[] args, TextWriter standardOutput, TextWriter standardError) at Canvas.Program.Main(String[] args)

[eroller]

Try again using an empty output directory

[ssri28]

Hi Eroller,

Thank for the reply..

i tried giving empty folder.

Again i am getting errors: Detailed Error Report : 2019-11-06T22:15:43+00:00,ERROR: Job CNV.vcf.gz failed with exit code 134. Job logs: /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stdout /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stderr Job error message: Unhandled Exception: System.InvalidOperationException: Sequence contains no elements at System.Linq.Enumerable.Average(IEnumerable1 source) at CanvasDiploidCaller.CanvasDiploidCaller.CallVariants(String variantFrequencyFile, String inFile, String outFile, String ploidyVcfPath, String referenceFolder, String sampleName, String truthDataPath) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDiploidCaller\CanvasDiploidCaller.cs:line 298 at CanvasDiploidCaller.Program.Main(String[] args) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDiploidCaller\Program.cs:line 98 2019-11-06T22:15:43+00:00,ERROR: Canvas workflow error: Isas.Framework.WorkManagement.JobFailedException: Job CNV.vcf.gz failed with exit code 134 at Isas.Framework.WorkManagement.JobLaunching.BasicJobLauncher.<RunSystemProcessCheckExitAsync>d__7.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Isas.Framework.WorkManagement.JobLaunching.RuntimeTrackingJobLauncher.<LaunchJobAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Isas.Framework.WorkManagement.IJobLauncherExtensions.LaunchJob(IJobLauncher jobLauncher, JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IWorkDoerJobExtensions.<>c__DisplayClass0_01.b0(WorkResources sr, IJobLauncher jobLauncher) at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<>cDisplayClass11_01.<RunWithResources>b__3(WorkResources resources) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<<RunWithResources>b__11_1>d1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Isas.Framework.WorkManagement.AsTaskExtensions.Await[T](Task1 task) at Canvas.CanvasRunner.RunGermlineCalling(IFileLocation partitionedPath, CanvasCallset callset, String ploidyVcfPath, IFileLocation canvasSnvPath) at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.<>c__DisplayClass0_0.<RunCheckpoint>b__0() at Isas.Framework.Checkpointing.Internals.SerializingCheckpointRunner.<>c__DisplayClass8_01.b0(Checkpoint checkpoint) at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func`2 run) at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.RunCheckpoint(ICheckpointRunner runner, String name, Action a) at Canvas.CanvasRunner.d38.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Canvas.CanvasRunner.CallSample(CanvasCallset callset) at Canvas.ModeLauncher.Launch()

[eroller]

So far I have seen three different error messages, with only one command line indicating you are running SmallPedigree-WGS workflow. However, the most recent error indicates you are running Germline-WGS? You should be running SmallPedigree-WGS as Germline-WGS has been deprecated.

Please attach these files in the future: CanvasLog.txt, CanvasError.txt and any log files mentioned in CanvasError.txt, for example looking at the above error you should also include: /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stdout /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stderr

[ssri28]

Hi Eroller,

Thanks for the rapid response..

i am running now SmallPedigree-WGS

I am attaching the error log.. kindly check it Canvas_error.zip

[ssri28]

Hi Eroller,

Thanks for the rapid response..

i am running now SmallPedigree-WGS

I am attaching the error log.. kindly check it

On Thu, 7 Nov 2019 at 04:11, Eric Roller notifications@github.com wrote:

So far I have seen three different error messages, with only one command line indicating you are running SmallPedigree-WGS workflow. However, the most recent error indicates you are running Germline-WGS? You should be running SmallPedigree-WGS as Germline-WGS has been deprecated.

Please attach these files in the future: CanvasLog.txt, CanvasError.txt and any log files mentioned in CanvasError.txt, for example looking at the above error you should also include: /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stdout /home/ec2-user/data/human_exome/ES_MSC_cnv_t/Logging/CNV.vcf.gz.stderr

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Illumina/canvas/issues/134?email_source=notifications&email_token=AKXA3TWE3MEJ4UFXXJJZCXTQSNB2JA5CNFSM4JIQYGN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIHZSI#issuecomment-550534345, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKXA3TS3R3JCGCK5L52M3GLQSNB2JANCNFSM4JIQYGNQ .

[eroller]

Failed to open file /home/ec2-user/data/human_exome/ES_MSC_cnv2/TempCNV/ballele.bedgraph.gz

Can you open that file? What are the contents? Is it a valid gz file?

[ssri28]

Hi Eroller,

Its a empty file.

[eroller]

Can you send me the Logging directory inside the output folder?

[ssri28]

Hi Eroller, i am sending you the logging directory files Logging.zip

[eroller]

It is empty? I don't see any files in there.

[ssri28]

Hi Eroller,

Please check this attachment

Thanks Logging.zip

[eroller]

It looks like all the b-allele VCF entries are being skipped. Sites must have PASS filter with GT field and GQ. Can you show me some lines of the VCF?

[ssri28]

Hi Eroller, I am attaching few lines of the vcf file ES_MSC_HM2FCDSXX_L2.HC_sample.vcf.zip

[eroller]

yes, try adding PASS to the FILTER column

[ssri28]

Hi Eroller, thanks for the reply, shall i add "PASS" to all Filter columns in the VCF file?

[eroller]

typically the FILTER is applied based on the quality of the variant calls (e.g. by using a threshold on GQ). You should consult the documentation for the small variant caller that generated the VCF on how to apply filters.

[ssri28]

Hi Eroller, Thank you for your suggestion.. now it is working for me.