nf-cmgg / structural

A bioinformatics best-practice analysis pipeline for calling structural variants (SVs), copy number variants (CNVs) and repeat region expansions (RREs) from short DNA reads
https://nf-cmgg.github.io/structural/
MIT License
19 stars 2 forks source link

Fix `manta` errors #38

Closed nvnieuwk closed 1 year ago

nvnieuwk commented 1 year ago

Description of the bug

When running manta on an HPC, the following error occurs:


[2023-04-26T17:36:31.661233Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Initiating pyFlow run
[2023-04-26T17:36:31.661876Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] pyFlowClientWorkflowClass: MantaWorkflow
[2023-04-26T17:36:31.661892Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] pyFlowVersion: 1.1.20
[2023-04-26T17:36:31.661903Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] pythonVersion: 2.7.15.final.0
[2023-04-26T17:36:31.661914Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] WorkingDir: '/vscmnt/gent_kyukon_scratch/_kyukon_scratch_gent/vo/000/gvo00082/vsc44804/work-structural/53/73332a2c679df04e2df65e4f53d647'
[2023-04-26T17:36:31.661925Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] ProcessCmdLine: 'manta/runWorkflow.py -m local -j 12'
[2023-04-26T17:36:31.661935Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] mode: local
[2023-04-26T17:36:31.661945Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] nCores: 12
[2023-04-26T17:36:31.661954Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] memMb: 94883
[2023-04-26T17:36:31.661964Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] dataDir: /vscmnt/gent_kyukon_scratch/_kyukon_scratch_gent/vo/000/gvo00082/vsc44804/work-structural/53/73332a2c679df04e2df65e4f53d647/manta/workspace/pyflow.data
[2023-04-26T17:36:31.661975Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] isDryRun: False
[2023-04-26T17:36:31.661985Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] isContinue: False
[2023-04-26T17:36:31.661995Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] isForceContinue: True
[2023-04-26T17:36:31.662005Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [RunParameters] mailTo: ''
[2023-04-26T17:36:31.662530Z] [node3214.victini.os] [2649321_1] [TaskRunner:masterWorkflow] Starting task specification for master workflow
[2023-04-26T17:36:31.662721Z] [node3214.victini.os] [2649321_1] [MantaWorkflow] Initiating Manta workflow version: 1.6.0
[2023-04-26T17:36:31.663546Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getAlignmentStats_makeTmpDir' to master workflow
[2023-04-26T17:36:31.664448Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getAlignmentStats_makeTmpDir' from master workflow
[2023-04-26T17:36:31.664919Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getAlignmentStats_generateStats_000' to master workflow
[2023-04-26T17:36:31.665661Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getAlignmentStats_mergeStats' to master workflow
[2023-04-26T17:36:31.666299Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getAlignmentStats_removeTmpDir' to master workflow
[2023-04-26T17:36:31.667368Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'summarizeStats' to master workflow
[2023-04-26T17:36:31.668151Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getChromDepth_makeTmpDir' to master workflow
[2023-04-26T17:36:31.668619Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getChromDepth_sample000_makeTmpDir' to master workflow
[2023-04-26T17:36:31.669868Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getChromDepth_makeTmpDir' from master workflow
[2023-04-26T17:36:31.671954Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getChromDepth_sample000_estimateChromDepth_022_chrX' to master workflow
[2023-04-26T17:36:31.675020Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getChromDepth_sample000_catChromDepth' to master workflow
[2023-04-26T17:36:31.675512Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getChromDepth_mergeChromDepth' to master workflow
[2023-04-26T17:36:31.675864Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'getChromDepth_removeTmpDir' to master workflow
[2023-04-26T17:36:31.677351Z] [node3214.victini.os] [2649321_1] [TaskRunner:getAlignmentStats_makeTmpDir] Task initiated on local node
[2023-04-26T17:36:31.679054Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'makeGraphTmpDir' to master workflow
[2023-04-26T17:36:31.680753Z] [node3214.victini.os] [2649321_1] [TaskRunner:getChromDepth_makeTmpDir] Task initiated on local node
[2023-04-26T17:36:31.684658Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'makeLocusGraph_chromId_022_chrX-253744-255091_0000_to_chromId_022_chrX-276323-303356_0000' to master workflow
[2023-04-26T17:36:31.688432Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'makeLocusGraph_chromId_022_chrX-276323-303356_0000_to_chromId_022_chrX-333934-386955_0000' to master workflow
[2023-04-26T17:36:31.690414Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'makeLocusGraph_chromId_022_chrX-333934-386955_0000_to_chromId_022_chrX-333934-386955_0000' to master workflow
[2023-04-26T17:36:31.692362Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] Adding command task 'makeLocusGraph_chromId_022_chrX-333934-386955_0000_to_chromId_022_chrX-333934-386955_0000' to master workflow
[2023-04-26T17:36:31.858902Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getAlignmentStats_makeTmpDir' launched from master workflow
[2023-04-26T17:36:31.859196Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getChromDepth_makeTmpDir' launched from master workflow
[2023-04-26T17:36:31.860726Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getChromDepth_sample000_makeTmpDir' from master workflow
[2023-04-26T17:36:31.861748Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getAlignmentStats_generateStats_000' from master workflow
[2023-04-26T17:36:31.870277Z] [node3214.victini.os] [2649321_1] [TaskRunner:getChromDepth_sample000_makeTmpDir] Task initiated on local node
[2023-04-26T17:36:31.873655Z] [node3214.victini.os] [2649321_1] [TaskRunner:getAlignmentStats_generateStats_000] Task initiated on local node
[2023-04-26T17:36:32.027083Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getChromDepth_sample000_makeTmpDir' launched from master workflow
[2023-04-26T17:36:32.028787Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getChromDepth_sample000_estimateChromDepth_022_chrX' from master workflow
[2023-04-26T17:36:32.040267Z] [node3214.victini.os] [2649321_1] [TaskRunner:getChromDepth_sample000_estimateChromDepth_022_chrX] Task initiated on local node
[2023-04-26T17:36:37.996402Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getAlignmentStats_generateStats_000' launched from master workflow
[2023-04-26T17:36:37.998083Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getAlignmentStats_mergeStats' from master workflow
[2023-04-26T17:36:38.006407Z] [node3214.victini.os] [2649321_1] [TaskRunner:getAlignmentStats_mergeStats] Task initiated on local node
[2023-04-26T17:36:38.212908Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getAlignmentStats_mergeStats' launched from master workflow
[2023-04-26T17:36:38.214380Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'summarizeStats' from master workflow
[2023-04-26T17:36:38.215292Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getAlignmentStats_removeTmpDir' from master workflow
[2023-04-26T17:36:38.227261Z] [node3214.victini.os] [2649321_1] [TaskRunner:summarizeStats] Task initiated on local node
[2023-04-26T17:36:38.228651Z] [node3214.victini.os] [2649321_1] [TaskRunner:getAlignmentStats_removeTmpDir] Task initiated on local node
[2023-04-26T17:36:38.380700Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getAlignmentStats_removeTmpDir' launched from master workflow
[2023-04-26T17:36:38.397495Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'summarizeStats' launched from master workflow
[2023-04-26T17:36:42.069870Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getChromDepth_sample000_estimateChromDepth_022_chrX' launched from master workflow
[2023-04-26T17:36:42.070787Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getChromDepth_sample000_catChromDepth' from master workflow
[2023-04-26T17:36:42.082634Z] [node3214.victini.os] [2649321_1] [TaskRunner:getChromDepth_sample000_catChromDepth] Task initiated on local node
[2023-04-26T17:36:42.336180Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getChromDepth_sample000_catChromDepth' launched from master workflow
[2023-04-26T17:36:42.337454Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getChromDepth_mergeChromDepth' from master workflow
[2023-04-26T17:36:42.349626Z] [node3214.victini.os] [2649321_1] [TaskRunner:getChromDepth_mergeChromDepth] Task initiated on local node
[2023-04-26T17:36:42.603686Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getChromDepth_mergeChromDepth' launched from master workflow
[2023-04-26T17:36:42.604793Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'getChromDepth_removeTmpDir' from master workflow
[2023-04-26T17:36:42.605613Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'makeGraphTmpDir' from master workflow
[2023-04-26T17:36:42.618055Z] [node3214.victini.os] [2649321_1] [TaskRunner:getChromDepth_removeTmpDir] Task initiated on local node
[2023-04-26T17:36:42.624799Z] [node3214.victini.os] [2649321_1] [TaskRunner:makeGraphTmpDir] Task initiated on local node
[2023-04-26T17:36:42.772107Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'makeGraphTmpDir' launched from master workflow
[2023-04-26T17:36:42.773262Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'makeLocusGraph_chromId_022_chrX-253744-255091_0000_to_chromId_022_chrX-276323-303356_0000' from master workflow
[2023-04-26T17:36:42.774037Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'makeLocusGraph_chromId_022_chrX-333934-386955_0000_to_chromId_022_chrX-333934-386955_0000' from master workflow
[2023-04-26T17:36:42.774802Z] [node3214.victini.os] [2649321_1] [TaskManager] Launching command task: 'makeLocusGraph_chromId_022_chrX-276323-303356_0000_to_chromId_022_chrX-333934-386955_0000' from master workflow
[2023-04-26T17:36:42.787991Z] [node3214.victini.os] [2649321_1] [TaskRunner:makeLocusGraph_chromId_022_chrX-276323-303356_0000_to_chromId_022_chrX-333934-386955_0000] Task initiated on local node
[2023-04-26T17:36:42.788838Z] [node3214.victini.os] [2649321_1] [TaskRunner:makeLocusGraph_chromId_022_chrX-333934-386955_0000_to_chromId_022_chrX-333934-386955_0000] Task initiated on local node
[2023-04-26T17:36:42.791145Z] [node3214.victini.os] [2649321_1] [TaskRunner:makeLocusGraph_chromId_022_chrX-253744-255091_0000_to_chromId_022_chrX-276323-303356_0000] Task initiated on local node
[2023-04-26T17:36:42.792098Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'getChromDepth_removeTmpDir' launched from master workflow
[2023-04-26T17:36:43.207232Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'makeLocusGraph_chromId_022_chrX-333934-386955_0000_to_chromId_022_chrX-333934-386955_0000' launched from master workflow
[2023-04-26T17:36:43.322213Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'makeLocusGraph_chromId_022_chrX-276323-303356_0000_to_chromId_022_chrX-333934-386955_0000' launched from master workflow
[2023-04-26T17:36:43.386734Z] [node3214.victini.os] [2649321_1] [TaskManager] Completed command task: 'makeLocusGraph_chromId_022_chrX-253744-255091_0000_to_chromId_022_chrX-276323-303356_0000' launched from master workflow
[2023-04-26T17:36:48.387914Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR] Failed to complete master workflow, error code: 1
[2023-04-26T17:36:48.388127Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR] errorMessage:
[2023-04-26T17:36:48.388150Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR] Unhandled Exception in TaskRunner-Thread-masterWorkflow
[2023-04-26T17:36:48.388166Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR] Traceback (most recent call last):
[2023-04-26T17:36:48.388181Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/pyflow/pyflow.py", line 1069, in run
[2023-04-26T17:36:48.388203Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     (retval, retmsg) = self._run()
[2023-04-26T17:36:48.388217Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/pyflow/pyflow.py", line 1121, in _run
[2023-04-26T17:36:48.388231Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     self.workflow.workflow()
[2023-04-26T17:36:48.388245Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/mantaWorkflow.py", line 853, in workflow
[2023-04-26T17:36:48.388260Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     graphTasks = runLocusGraph(self,dependencies=graphTaskDependencies)
[2023-04-26T17:36:48.388276Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/mantaWorkflow.py", line 286, in runLocusGraph
[2023-04-26T17:36:48.388293Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     graphTasks.add(self.addTask(graphTask,graphCmd,dependencies=dirTask,memMb=self.params.estimateMemMb))
[2023-04-26T17:36:48.388309Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/pyflow/pyflow.py", line 3707, in addTask
[2023-04-26T17:36:48.388323Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     self._addTaskCore(self._getNamespace(), label, payload, dependencies)
[2023-04-26T17:36:48.388337Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/pyflow/pyflow.py", line 4087, in _addTaskCore
[2023-04-26T17:36:48.388351Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     self._tdag.addTask(namespace, label, payload, dependencies)
[2023-04-26T17:36:48.388367Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/pyflow/pyflow.py", line 537, in wrapped
[2023-04-26T17:36:48.388381Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     return f(self, *args, **kw)
[2023-04-26T17:36:48.388394Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]   File "/usr/local/share/manta-1.6.0-1/lib/python/pyflow/pyflow.py", line 2546, in addTask
[2023-04-26T17:36:48.388409Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR]     raise Exception("Task: '%s' is already in TaskDAG" % (fullLabel))
[2023-04-26T17:36:48.388424Z] [node3214.victini.os] [2649321_1] [WorkflowRunner] [ERROR] Exception: Task: 'makeLocusGraph_chromId_022_chrX-333934-386955_0000_to_chromId_022_chrX-333934-386955_00

Command used and terminal output

No response

Relevant files

No response

System information

No response

nvnieuwk commented 1 year ago

Issues lied with a wrongly compiled version in the conda version. I fixed this and made a docker container from this on https://quay.io/repository/cmgg/manta?tab=tags&tag=latest