Closed philsteinberg closed 1 year ago
OK I'm just going to pick 1 issue to reply to since all the DPClust multi-region errors are the same. I've seen this before but I don't remember what I did...
Have you tried running multi-region DPClust on the test samples in pipeline-call-SRC?
Could you try using the tip from Yash here: https://github.com/uclahs-cds/tool-SRC-util/issues/39 and set the work_dir to somewhere in /hot/
? That way we can look at the allDirichletProcessInfo.txt
files and do some debugging from there. It seems like the pipeline dies at the running DPClust step
@lydiayliu The pipeline-call-SRC README paths to the test input are not linked/valid anymore. I only found these test files from #3
ILHNLNEV000001-N002-A01-F
Config: /hot/software/pipeline/pipeline-call-SRC/Nextflow/development/unreleased/yashpatel-add-phylowgs/ILHNLNEV000001-N002-A01-F.config
YAML: /hot/software/pipeline/pipeline-call-SRC/Nextflow/development/unreleased/yashpatel-add-phylowgs/ILHNLNEV000001-N002-A01-F.yaml
However, the yaml directory only contains the ILHNLNEV000001-N002-A01-F of the patient, so I cannot use this for multi-region mode.
I changed my seed_51404.config to include a new working directory path:
algorithm: "DPClust"
options: "--seed 51404 --min_frac_muts_cluster -1"
dpclust_multisample = true
work_dir = "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/"
Error log pretty much the same: /hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/logs/ILHNLNEV000001_51404_Strelka2-Battenberg-DPClust-mr.log
Error executing process > 'workflow_dpclust:call_RunDP_DPClust'
Caused by:
Process `workflow_dpclust:call_RunDP_DPClust` terminated with an error exit status (1)
Command executed:
set -euo pipefail
printf "ILHNLNEV000001 ILHNLNEV000001-T002-P02-F ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt 0.534937500000003 male NA NA
ILHNLNEV000001 ILHNLNEV000001-T001-P01-F ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt 0.562956521739133 male NA NA
ILHNLNEV000001 ILHNLNEV000001-T004-L02-F ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt 0.56375 male NA NA
ILHNLNEV000001 ILHNLNEV000001-T003-L01-F ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt 0.635411764705883 male NA NA\n" > ILHNLNEV000001_project_file_intermediate.txt
printf "sample\tsubsample\tdatafile\tcellularity\tsex\tcnadatafile\tindeldatafile\n" > ILHNLNEV000001_project_file.txt
while IFS=$' ' read -r -a arr
do
printf "${arr[0]} ${arr[1]} `realpath ${arr[2]}` ${arr[3]} ${arr[4]} ${arr[5]} ${arr[6]}
" >> ILHNLNEV000001_project_file.txt
done < ILHNLNEV000001_project_file_intermediate.txt
Rscript /hot/user/yashpatel/pipeline-call-SRC/pipeline-call-SRC/./module/dpclust_pipeline.R -d / -o ./ -r 1 -i ILHNLNEV000001_project_file.txt -k --seed 51404 --min_frac_muts_cluster -1
Command exit status:
1
Command output:
[1] ""
[1] "Running: ILHNLNEV000001"
[1] "Working dir: ./"
[1] "Analysis type: nd_dp"
[1] "Datafiles:"
[1] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/43/5859671f3ac6a40f7fe87518ec839a/ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt"
[2] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/a1/5a8ce60fb4abb078ed083176db72a7/ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt"
[3] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/d5/a5b6e28c9820c9366191e33d05000f/ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt"
[4] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/96/215c615cd3a8fbcc2de4269dbfcf20/ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt"
[1] ""
[1] "Loading data..."
Command error:
acf214f6fcba: Already exists
5f952968c778: Already exists
313a98bcd182: Already exists
0fa8b30a6199: Already exists
c082b8b01591: Already exists
177ca9de5b2d: Already exists
5a3b71fb959c: Already exists
a7c3f2278282: Already exists
874ae8d94f7e: Already exists
11a68d1beffd: Already exists
e9ad119fed45: Already exists
0deb46237183: Pulling fs layer
0deb46237183: Verifying Checksum
0deb46237183: Download complete
0deb46237183: Pull complete
Digest: sha256:586905a6f63308c3691632f5a9b115989be3a06924907c0aa7588f2dc1624423
Status: Downloaded newer image for ghcr.io/uclahs-cds/docker-dpclust:75f5d7e
Bioconductor version 3.12 (BiocManager 1.30.10), ?BiocManager::install for help
Bioconductor version '3.12' is out-of-date; the current release version '3.16'
is available with R version '4.2'; see https://bioconductor.org/install
[1] ""
[1] "Running: ILHNLNEV000001"
[1] "Working dir: ./"
[1] "Analysis type: nd_dp"
[1] "Datafiles:"
[1] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/43/5859671f3ac6a40f7fe87518ec839a/ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt"
[2] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/a1/5a8ce60fb4abb078ed083176db72a7/ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt"
[3] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/d5/a5b6e28c9820c9366191e33d05000f/ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt"
[4] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/96/215c615cd3a8fbcc2de4269dbfcf20/ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt"
[1] ""
The following objects are masked _by_ .GlobalEnv:
assign_sampled_muts, generate_cluster_ordering, keep_temp_files,
min_frac_muts_cluster, min_muts_cluster, mut.assignment.type,
no.iters, no.iters.burn.in, num_muts_sample, species
The following objects are masked _by_ .GlobalEnv:
cellularity, datafiles, is.male, mutphasingfiles, samplename,
subsamples
The following object is masked _by_ .GlobalEnv:
seed
[1] "Loading data..."
Error in chromosome[, s] <- list_of_tables[[s]][, Chromosome] :
number of items to replace is not a multiple of replacement length
Calls: RunDP -> load.data -> load.data.inner
Execution halted
Work dir:
/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/97/7f760b3dd430526313c5b5c645fcd4
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
executor > local (18)
[b1/c52933] process > run_validate_PipeVal (11) [100%] 12 of 12 ✔
[0d/5e58ac] process > create_inputs_SRCutil (1) [100%] 1 of 1 ✔
[- ] process > workflow_pyclonevi:fit_mode... -
[- ] process > workflow_pyclonevi:write_re... -
[- ] process > workflow_phylowgs:call_mult... -
[- ] process > workflow_phylowgs:write_res... -
[- ] process > workflow_phylowgs:index_dat... -
[96/215c61] process > workflow_dpclust:generate_i... [100%] 4 of 4 ✔
[97/7f760b] process > workflow_dpclust:call_RunDP... [100%] 1 of 1, failed: 1 ✘
[- ] process > workflow_pyclone:run_analys... -
[- ] process > workflow_fastclone:run_solv... -
Error executing process > 'workflow_dpclust:call_RunDP_DPClust'
Caused by:
Process `workflow_dpclust:call_RunDP_DPClust` terminated with an error exit status (1)
Command executed:
set -euo pipefail
printf "ILHNLNEV000001 ILHNLNEV000001-T002-P02-F ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt 0.534937500000003 male NA NA
ILHNLNEV000001 ILHNLNEV000001-T001-P01-F ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt 0.562956521739133 male NA NA
ILHNLNEV000001 ILHNLNEV000001-T004-L02-F ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt 0.56375 male NA NA
ILHNLNEV000001 ILHNLNEV000001-T003-L01-F ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt 0.635411764705883 male NA NA\n" > ILHNLNEV000001_project_file_intermediate.txt
printf "sample\tsubsample\tdatafile\tcellularity\tsex\tcnadatafile\tindeldatafile\n" > ILHNLNEV000001_project_file.txt
while IFS=$' ' read -r -a arr
do
printf "${arr[0]} ${arr[1]} `realpath ${arr[2]}` ${arr[3]} ${arr[4]} ${arr[5]} ${arr[6]}
" >> ILHNLNEV000001_project_file.txt
done < ILHNLNEV000001_project_file_intermediate.txt
Rscript /hot/user/yashpatel/pipeline-call-SRC/pipeline-call-SRC/./module/dpclust_pipeline.R -d / -o ./ -r 1 -i ILHNLNEV000001_project_file.txt -k --seed 51404 --min_frac_muts_cluster -1
Command exit status:
1
Command output:
[1] ""
[1] "Running: ILHNLNEV000001"
[1] "Working dir: ./"
[1] "Analysis type: nd_dp"
[1] "Datafiles:"
[1] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/43/5859671f3ac6a40f7fe87518ec839a/ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt"
[2] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/a1/5a8ce60fb4abb078ed083176db72a7/ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt"
[3] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/d5/a5b6e28c9820c9366191e33d05000f/ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt"
[4] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/96/215c615cd3a8fbcc2de4269dbfcf20/ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt"
[1] ""
[1] "Loading data..."
Command error:
acf214f6fcba: Already exists
5f952968c778: Already exists
313a98bcd182: Already exists
0fa8b30a6199: Already exists
c082b8b01591: Already exists
177ca9de5b2d: Already exists
5a3b71fb959c: Already exists
a7c3f2278282: Already exists
874ae8d94f7e: Already exists
11a68d1beffd: Already exists
e9ad119fed45: Already exists
0deb46237183: Pulling fs layer
0deb46237183: Verifying Checksum
0deb46237183: Download complete
0deb46237183: Pull complete
Digest: sha256:586905a6f63308c3691632f5a9b115989be3a06924907c0aa7588f2dc1624423
Status: Downloaded newer image for ghcr.io/uclahs-cds/docker-dpclust:75f5d7e
Bioconductor version 3.12 (BiocManager 1.30.10), ?BiocManager::install for help
Bioconductor version '3.12' is out-of-date; the current release version '3.16'
is available with R version '4.2'; see https://bioconductor.org/install
[1] ""
[1] "Running: ILHNLNEV000001"
[1] "Working dir: ./"
[1] "Analysis type: nd_dp"
[1] "Datafiles:"
[1] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/43/5859671f3ac6a40f7fe87518ec839a/ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt"
[2] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/a1/5a8ce60fb4abb078ed083176db72a7/ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt"
[3] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/d5/a5b6e28c9820c9366191e33d05000f/ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt"
[4] "/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/96/215c615cd3a8fbcc2de4269dbfcf20/ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt"
[1] ""
The following objects are masked _by_ .GlobalEnv:
assign_sampled_muts, generate_cluster_ordering, keep_temp_files,
min_frac_muts_cluster, min_muts_cluster, mut.assignment.type,
no.iters, no.iters.burn.in, num_muts_sample, species
The following objects are masked _by_ .GlobalEnv:
cellularity, datafiles, is.male, mutphasingfiles, samplename,
subsamples
The following object is masked _by_ .GlobalEnv:
seed
[1] "Loading data..."
Error in chromosome[, s] <- list_of_tables[[s]][, Chromosome] :
number of items to replace is not a multiple of replacement length
Calls: RunDP -> load.data -> load.data.inner
Execution halted
Work dir:
/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/97/7f760b3dd430526313c5b5c645fcd4
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Try the test sample here: https://github.com/uclahs-cds/pipeline-call-SRC/pull/11, it seemed to have worked with HATCHET and ms MuTect2.
Right so now you see in your error log that the "Work dir" is the debug directory you set (in your old logs it should be something in /scratch/
Work dir:
/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/97/7f760b3dd430526313c5b5c645fcd4
Now you can go to the directory and see the temporary files
yiyangliu@ip-0A12520B:/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/97/7f760b3dd430526313c5b5c645fcd4$ ls
ILHNLNEV000001_DPoutput_2000iters_1000burnin_seed51404 ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt
ILHNLNEV000001_project_file_intermediate.txt ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt
ILHNLNEV000001_project_file.txt ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt
ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt
If you do ls -a
, you can see the hidden log files as well, the one you want is .command.log
. This is handy for meta-pipelines where the first layer of errors are hidden. Not so useful now since what we want are the intermediate allDirichletProcessInfo.txt
files that are causing the issue, but this is good to know.
Ok so I think the problem is pretty clear, a quick wc -l
revelas that each of the allDirichletProcessInfo.txt
files have differing number of lines. This doesn't work for DPClust since dpclust requires in multi-region SRC that all samples have the SAME set of SNVs.
yiyangliu@ip-0A12520B:/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug/97/7f760b3dd430526313c5b5c645fcd4$ wc -l *Info.txt
4999 ILHNLNEV000001-T001-P01-F_allDirichletProcessInfo.txt
4886 ILHNLNEV000001-T002-P02-F_allDirichletProcessInfo.txt
4896 ILHNLNEV000001-T003-L01-F_allDirichletProcessInfo.txt
4768 ILHNLNEV000001-T004-L02-F_allDirichletProcessInfo.txt
19549 total
The allDirichletProcessInfo.txt
is not something that tool-SRC-util
produces but is the direct product of the loci.txt
and alleleCounts.txt
files that tool-SRC-util
produces. For multi-region DPClust SNV pre-processing, for now only SNVs that are called in all samples can be used. I'm surprised that I never noticed this... (since PyClone-VI also requires SNVs to be shared between all samples, did we not just use the same system for DPClust?) Please open an issue in that repo and fix this for DPClust (and ideally double check for PyClone-VI). I think you can handle it from here!
Btw you can also dig around your debug
folder for more fun stuff, since all the intermediate files from the run can be found here.
For example, you can clearly see that the intermediate loci.txt
files produces have differing number of lines between the samples:
yiyangliu@ip-0A12520B:/hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/debug$ wc -l */*/*loci.txt
4885 43/5859671f3ac6a40f7fe87518ec839a/ILHNLNEV000001-T002-P02-F-loci.txt
4895 96/215c615cd3a8fbcc2de4269dbfcf20/ILHNLNEV000001-T003-L01-F-loci.txt
4998 a1/5a8ce60fb4abb078ed083176db72a7/ILHNLNEV000001-T001-P01-F-loci.txt
4767 d5/a5b6e28c9820c9366191e33d05000f/ILHNLNEV000001-T004-L02-F-loci.txt
19545 total
Running Strelka2-Battenberg-DPClust single-region mode worked fine. Same issue for SomaticSniper/Mutect2-Battenberg-DPClust-mr. Could this be do to the naming conventions in multi-region mode?
error log:
example error log: /hot/project/method/AlgorithmEvaluation/BNCH-000082-SRCRNDSeed/pipeline-call-src/run-strelka2-battenberg-dpclust/logs/ILHNLNEV000001_51404_Strelka2-Battenberg-DPClust-mr.log
error message: