search

uclahs-cds / metapipeline-DNA

Nextflow pipeline to convert BAM to FASTQ, align, perform QC, assess targeted coverage, call gSNP, call sSNV, call mtSNV, call SVs, call sCNA, and perform subclonal reconstruction
GNU General Public License v2.0
4 stars 0 forks source link

Metapipeline job silently failed during recalibrate BAM - identify_outputs #189

Open nwiltsie opened 7 months ago

nwiltsie commented 7 months ago

Describe the issue

@WuSelina's job silently failed while reporting success:

Submission Script output: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/output/P004_P006_paired_metaDNA.log:

[fa/549b6a] process > create_config_metapipeline_... [100%] 2 of 2 ✔
[0b/b645b4] process > call_metapipeline_DNA (2)      [100%] 2 of 2 ✔
[40/9e021f] process > check_process_status (2)       [100%] 2 of 2 ✔

Completed at: 05-Apr-2024 12:56:33
Duration    : 3d 17h 13m 14s
CPU hours   : 89.2
Succeeded   : 8

Step output: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a/.command.log:

[16/a88725] process > recalibrate_BAM:create_YAML... [100%] 1 of 1 ✔
[-        ] process > recalibrate_BAM:run_recalib... -
[03/ff7666] process > recalibrate_BAM:run_recalib... [  0%] 0 of 1
[-        ] process > call_gSNP:create_YAML_call_... -
[-        ] process > call_gSNP:run_call_gSNP        -
[-        ] process > call_sSNV:create_YAML_call_... -
[-        ] process > call_sSNV:run_call_sSNV        -
[-        ] process > call_mtSNV:create_YAML_call... -
[-        ] process > call_mtSNV:run_call_mtSNV      -
[-        ] process > call_gSV:create_YAML_call_gSV  -
[-        ] process > call_gSV:run_call_gSV          -
[-        ] process > call_sSV:create_YAML_call_sSV  -
[-        ] process > call_sSV:run_call_sSV          -
[-        ] process > call_sCNA:create_YAML_call_... -
[-        ] process > call_sCNA:run_call_sCNA        -
ERROR ~ Unexpected error [InvocationTargetException]

 -- Check script '/hot/software/pipeline/metapipeline-DNA/Nextflow/release/6.0.0-rc.2/module/recalibrate_BAM/./identify_outputs.nf' at line: 25 or see '.nextflow.log' file for more details

executor >  local (2)
[16/a88725] process > recalibrate_BAM:create_YAML... [100%] 1 of 1 ✔
[-        ] process > recalibrate_BAM:run_recalib... -
[03/ff7666] process > recalibrate_BAM:run_recalib... [100%] 1 of 1 ✔
[-        ] process > call_gSNP:create_YAML_call_... -
[-        ] process > call_gSNP:run_call_gSNP        -
[-        ] process > call_sSNV:create_YAML_call_... -
[-        ] process > call_sSNV:run_call_sSNV        -
[-        ] process > call_mtSNV:create_YAML_call... -
[-        ] process > call_mtSNV:run_call_mtSNV      -
[-        ] process > call_gSV:create_YAML_call_gSV  -
[-        ] process > call_gSV:run_call_gSV          -
[-        ] process > call_sSV:create_YAML_call_sSV  -
[-        ] process > call_sSV:run_call_sSV          -
[-        ] process > call_sCNA:create_YAML_call_... -
[-        ] process > call_sCNA:run_call_sCNA        -
ERROR ~ Unexpected error [InvocationTargetException]

 -- Check script '/hot/software/pipeline/metapipeline-DNA/Nextflow/release/6.0.0-rc.2/module/recalibrate_BAM/./identify_outputs.nf' at line: 25 or see '.nextflow.log' file for more details

Step .nextflow.log file: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a/.nextflow.log:

~> TaskHandler[id: 2; name: recalibrate_BAM:run_recalibrate_BAM_delete_all (1); status: RUNNING; exit: -; error: -; workDir: /scratch/110525/03/ff76662bfcb03f435899f47ba4b4dd]
Apr-05 12:54:59.923 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: recalibrate_BAM:run_recalibrate_BAM_delete_all (1); status: COMPLETED; exit: 0; error: -; workDir: /scratch/110525/03/ff76662bfcb03f435899f47ba4b4dd]
Apr-05 12:55:00.004 [Actor Thread 71] ERROR nextflow.extension.OperatorImpl - @unknown
java.lang.reflect.InvocationTargetException: null
>---at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>---at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
... snip ...
>---at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>---at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert file_found.size() == 1
       |          |
       []         0
>---at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:432)
>---at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:670)
>---at Script_c0b80539.identify_file(Script_c0b80539:20)
>---... 76 common frames omitted
Apr-05 12:55:00.038 [Actor Thread 71] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.reflect.InvocationTargetException

/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/output /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/input/batch2_aligned_BAMs_input_P004_P006.csv

nwiltsie commented 7 months ago

So there are really two issues here:

  1. The pipeline died for some unknown reason
  2. The wrapper job reported success despite the failure
$ sacct -j 110520,110525 --parsable --format=JobID,JobName,State,Start,End | xsv table -d'|'
JobID          JobName                                                                     State      Start                End
110520         P004_P006_paired_metaDNA                                                    COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110520.batch   batch                                                                       COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110520.extern  extern                                                                      COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110525         wgs_call_metapipeline_DNA_STGHKGFH000004_bb_ff891798967bb1bbcebd760873cd7a  FAILED     2024-04-01T23:19:59  2024-04-05T12:55:01
110525.batch   batch                                                                       FAILED     2024-04-01T23:19:59  2024-04-05T12:55:01
110525.extern  extern                                                                      COMPLETED  2024-04-01T23:19:59  2024-04-05T12:55:01
WuSelina commented 7 months ago

I ran metapipeline of the same patient (STGHKGFH000004) in 'paired' mode using only the first 1000 lines in the tumor/normal BAMs (excluding the header). I have set the pipeilne_work_dir = leading_work_dir instead of /scratch to save logs/output that might otherwise be lost for this run. Perhaps we can see some errors logs/outputs by setting it up this way while keeping the inputs small. Update: It ran successfully.

Here is the path to the metapipeline run: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/debug_recalibrate-BAM