search

caleblareau / mgatk

mgatk: mitochondrial genome analysis toolkit
http://caleblareau.github.io/mgatk
MIT License
101 stars 27 forks source link

Error with bcall mode #38

Closed Arkosen closed 3 years ago

Arkosen commented 3 years ago

I want to perform genotyping for scATAC-seq data generated using 10x multiomics protocol. The bam file was generate using cellranger-arc.

The command I am running is as follows :

mgatk bcall -i atac_possorted_bam.bam -g hg38 -bt CB -b filtered.barcodes.txt -ns 100 -c 8 -n sample1

the error msg is as follows:

Thu Jun 03 17:04:43 PDT 2021: Genotyping samples with 8 threads Error in checkGrep(grep(".A.txt", files)) : Improper folder specification; file missing / extra file present. See documentation Calls: importMito -> checkGrep Execution halted

I was hoping you could help me with this as I am not sure what I am doing wrong

Thanks in advance

caleblareau commented 3 years ago

Can you send an ls -lRh of the output folder?

caleblareau commented 3 years ago

mmm no there should be a mgatk_out folder where you executed things

Arkosen commented 3 years ago

.: total 12K drwxrwxr-x 2 asen asen 4.0K Jun 3 17:33 final drwxrwxr-x 4 asen asen 4.0K Jun 3 17:35 logs -rw-rw-r-- 1 asen asen 0 Jun 3 17:37 output.txt drwxrwxr-x 3 asen asen 4.0K Jun 3 17:35 qc

./final: total 124K -rw-rw-r-- 1 asen asen 119K Jun 3 17:33 chrM_refAllele.txt

./logs: total 40K -rw-rw-r-- 1 asen asen 409 Jun 3 17:35 base.mgatk.log drwxrwxr-x 2 asen asen 4.0K Jun 3 17:34 filterlogs -rw-rw-r-- 1 asen asen 512 Jun 3 17:34 MCG001D.parameters.txt -rw-rw-r-- 1 asen asen 2.2K Jun 3 17:35 MCG001D.snakemake_gather.log -rw-rw-r-- 1 asen asen 20K Jun 3 17:35 MCG001D.snakemake_scatter.log drwxrwxr-x 2 asen asen 4.0K Jun 3 17:34 rmdupslogs

./logs/filterlogs: total 0 -rw-rw-r-- 1 asen asen 18 Jun 3 17:34 AACTAGCTCATCCTAT-1.filter.log -rw-rw-r-- 1 asen asen 19 Jun 3 17:34 ACTCAGTAGGAGTAAT-1.filter.log -rw-rw-r-- 1 asen asen 17 Jun 3 17:34 ATCCAACAGCGGCTGT-1.filter.log -rw-rw-r-- 1 asen asen 18 Jun 3 17:34 CGGACAACATTAAGCT-1.filter.log -rw-rw-r-- 1 asen asen 19 Jun 3 17:34 GAACCTTAGGCCGGAA-1.filter.log -rw-rw-r-- 1 asen asen 18 Jun 3 17:34 GAGCTTAGTGCATCGG-1.filter.log -rw-rw-r-- 1 asen asen 17 Jun 3 17:34 GCTTGTTGTGTGTCCC-1.filter.log -rw-rw-r-- 1 asen asen 18 Jun 3 17:34 GGATGTAAGCGGTTAT-1.filter.log

./logs/rmdupslogs: total 0

./qc: total 4.0K drwxrwxr-x 2 asen asen 4.0K Jun 3 17:34 quality

./qc/quality: total 0

Arkosen commented 3 years ago

My apologies. I absentmindedly deleted the 'mgatk_out' folder. This is 'ls -LRh' output for the correct folder. Does this help?

caleblareau commented 3 years ago

yes thanks-- can you now cat mgatk_out/logs/*log

Arkosen commented 3 years ago

Thu Jun 03 17:34:32 PDT 2021: Starting analysis with mgatk Thu Jun 03 17:34:32 PDT 2021: mgatk will process 1127 samples Thu Jun 03 17:34:34 PDT 2021: Processing samples with 8 threads Thu Jun 03 17:35:37 PDT 2021: mgatk successfully processed the supplied .bam files Thu Jun 03 17:35:51 PDT 2021: Successfully created final output files Thu Jun 03 17:35:53 PDT 2021: Intermediate files successfully removed. Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job counts: count jobs 1 all 1 make_depth_table 1 make_final_sparse_matrices 3

[Thu Jun 3 17:35:37 2021] rule make_final_sparse_matrices: output: mgatk/test/final/MCG001D.A.txt.gz, mgatk/test/final/MCG001D.C.txt.gz, mgatk/test/final/MCG001D.G.txt.gz, mgatk/test/final/MCG001D.T.txt.gz, mgatk/test/final/MCG001D.coverage.txt.gz jobid: 2

Job counts: count jobs 1 make_final_sparse_matrices 1 [Thu Jun 3 17:35:38 2021] Error in rule make_final_sparse_matrices:  jobid: 0  output: mgatk/test/final/MCG001D.A.txt.gz, mgatk/test/final/MCG001D.C.txt.gz, mgatk/test/final/MCG001D.G.txt.gz, mgatk/test/final/MCG001D.T.txt.gz, mgatk/test/final/MCG001D.coverage.txt.gz  RuleException: AttributeError in line 54 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Gather: 'InputFiles' object has no attribute 'As' File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 2357, in run_wrapper File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Gather", line 54, in rule_make_final_sparse_matrices File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init__.py", line 574, in _callback File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/concurrent/futures/thread.py", line 52, in run File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 560, in cached_or_run File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 2390, in run_wrapper Exiting because a job execution failed. Look above for error message Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /iblm/netapp/home/asen/.snakemake/log/2021-06-03T173537.658498.snakemake.log Building DAG of jobs... Using shell: /bin/bash Provided cores: 8 Rules claiming more threads will be scaled down. Job counts: count jobs 1 all 1 make_sample_list 1127 process_one_sample 1129

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/ACTCAGTAGGAGTAAT-1.bam.txt output: mgatk/test/temp/ready_bam/ACTCAGTAGGAGTAAT-1.qc.bam, mgatk/test/temp/ready_bam/ACTCAGTAGGAGTAAT-1.qc.bam.bai, mgatk/test/qc/depth/ACTCAGTAGGAGTAAT-1.depth.txt, mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.A.txt, mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.C.txt, mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.G.txt, mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.T.txt, mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.coverage.txt jobid: 120 wildcards: sample=ACTCAGTAGGAGTAAT-1

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/GAACCTTAGGCCGGAA-1.bam.txt output: mgatk/test/temp/ready_bam/GAACCTTAGGCCGGAA-1.qc.bam, mgatk/test/temp/ready_bam/GAACCTTAGGCCGGAA-1.qc.bam.bai, mgatk/test/qc/depth/GAACCTTAGGCCGGAA-1.depth.txt, mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.A.txt, mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.C.txt, mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.G.txt, mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.T.txt, mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.coverage.txt jobid: 561 wildcards: sample=GAACCTTAGGCCGGAA-1

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/AACTAGCTCATCCTAT-1.bam.txt output: mgatk/test/temp/ready_bam/AACTAGCTCATCCTAT-1.qc.bam, mgatk/test/temp/ready_bam/AACTAGCTCATCCTAT-1.qc.bam.bai, mgatk/test/qc/depth/AACTAGCTCATCCTAT-1.depth.txt, mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.A.txt, mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.C.txt, mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.G.txt, mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.T.txt, mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.coverage.txt jobid: 15 wildcards: sample=AACTAGCTCATCCTAT-1

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/CGGACAACATTAAGCT-1.bam.txt output: mgatk/test/temp/ready_bam/CGGACAACATTAAGCT-1.qc.bam, mgatk/test/temp/ready_bam/CGGACAACATTAAGCT-1.qc.bam.bai, mgatk/test/qc/depth/CGGACAACATTAAGCT-1.depth.txt, mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.A.txt, mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.C.txt, mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.G.txt, mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.T.txt, mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.coverage.txt jobid: 456 wildcards: sample=CGGACAACATTAAGCT-1

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/ATCCAACAGCGGCTGT-1.bam.txt output: mgatk/test/temp/ready_bam/ATCCAACAGCGGCTGT-1.qc.bam, mgatk/test/temp/ready_bam/ATCCAACAGCGGCTGT-1.qc.bam.bai, mgatk/test/qc/depth/ATCCAACAGCGGCTGT-1.depth.txt, mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.A.txt, mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.C.txt, mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.G.txt, mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.T.txt, mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.coverage.txt jobid: 217 wildcards: sample=ATCCAACAGCGGCTGT-1

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/GAGCTTAGTGCATCGG-1.bam.txt output: mgatk/test/temp/ready_bam/GAGCTTAGTGCATCGG-1.qc.bam, mgatk/test/temp/ready_bam/GAGCTTAGTGCATCGG-1.qc.bam.bai, mgatk/test/qc/depth/GAGCTTAGTGCATCGG-1.depth.txt, mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.A.txt, mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.C.txt, mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.G.txt, mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.T.txt, mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.coverage.txt jobid: 591 wildcards: sample=GAGCTTAGTGCATCGG-1

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/GCTTGTTGTGTGTCCC-1.bam.txt output: mgatk/test/temp/ready_bam/GCTTGTTGTGTGTCCC-1.qc.bam, mgatk/test/temp/ready_bam/GCTTGTTGTGTGTCCC-1.qc.bam.bai, mgatk/test/qc/depth/GCTTGTTGTGTGTCCC-1.depth.txt, mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.A.txt, mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.C.txt, mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.G.txt, mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.T.txt, mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.coverage.txt jobid: 711 wildcards: sample=GCTTGTTGTGTGTCCC-1

[Thu Jun 3 17:34:41 2021] rule process_one_sample: input: mgatk/test/.internal/samples/GGATGTAAGCGGTTAT-1.bam.txt output: mgatk/test/temp/ready_bam/GGATGTAAGCGGTTAT-1.qc.bam, mgatk/test/temp/ready_bam/GGATGTAAGCGGTTAT-1.qc.bam.bai, mgatk/test/qc/depth/GGATGTAAGCGGTTAT-1.depth.txt, mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.A.txt, mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.C.txt, mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.G.txt, mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.T.txt, mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.coverage.txt jobid: 726 wildcards: sample=GGATGTAAGCGGTTAT-1

Job counts: count jobs 1 process_one_sample 1 Job counts: count jobs 1 process_one_sample 1 Job counts: count jobs 1 process_one_sample 1 Job counts: count jobs 1 process_one_sample 1 Job counts: count jobs 1 process_one_sample 1 Job counts: count jobs 1 process_one_sample 1 Job counts: count jobs 1 process_one_sample 1 Job counts: count jobs 1 process_one_sample 1 Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/GCTTGTTGTGTGTCCC-1.qc.bam' MissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/GCTTGTTGTGTGTCCC-1.qc.bam mgatk/test/temp/ready_bam/GCTTGTTGTGTGTCCC-1.qc.bam.bai mgatk/test/qc/depth/GCTTGTTGTGTGTCCC-1.depth.txt mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.A.txt mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.C.txt mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.G.txt mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.T.txt mgatk/test/temp/sparse_matrices/GCTTGTTGTGTGTCCC-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_success Exiting because a job execution failed. Look above for error message Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/GAGCTTAGTGCATCGG-1.qc.bam' Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/ATCCAACAGCGGCTGT-1.qc.bam' Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/GGATGTAAGCGGTTAT-1.qc.bam' Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/GAACCTTAGGCCGGAA-1.qc.bam' Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/AACTAGCTCATCCTAT-1.qc.bam' Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/ACTCAGTAGGAGTAAT-1.qc.bam' Traceback (most recent call last): File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/python/oneSample.py", line 82, in pysam.index(outputbam) File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/pysam/utils.py", line 56, in call retval, stderr, stdout = _pysam_dispatch( File "pysam/libcutils.pyx", line 293, in pysam.libcutils._pysam_dispatch OSError: No such file or directory: 'mgatk/test/temp/ready_bam/CGGACAACATTAAGCT-1.qc.bam' MissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/GAGCTTAGTGCATCGG-1.qc.bam mgatk/test/temp/ready_bam/GAGCTTAGTGCATCGG-1.qc.bam.bai mgatk/test/qc/depth/GAGCTTAGTGCATCGG-1.depth.txt mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.A.txt mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.C.txt mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.G.txt mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.T.txt mgatk/test/temp/sparse_matrices/GAGCTTAGTGCATCGG-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_success MissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/AACTAGCTCATCCTAT-1.qc.bam mgatk/test/temp/ready_bam/AACTAGCTCATCCTAT-1.qc.bam.bai mgatk/test/qc/depth/AACTAGCTCATCCTAT-1.depth.txt mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.A.txt mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.C.txt mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.G.txt mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.T.txt mgatk/test/temp/sparse_matrices/AACTAGCTCATCCTAT-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_success MissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/GAACCTTAGGCCGGAA-1.qc.bam mgatk/test/temp/ready_bam/GAACCTTAGGCCGGAA-1.qc.bam.bai mgatk/test/qc/depth/GAACCTTAGGCCGGAA-1.depth.txt mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.A.txt mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.C.txt mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.G.txt mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.T.txt mgatk/test/temp/sparse_matrices/GAACCTTAGGCCGGAA-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_success MissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/GGATGTAAGCGGTTAT-1.qc.bam mgatk/test/temp/ready_bam/GGATGTAAGCGGTTAT-1.qc.bam.bai mgatk/test/qc/depth/GGATGTAAGCGGTTAT-1.depth.txt mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.A.txt mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.C.txt mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.G.txt mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.T.txt mgatk/test/temp/sparse_matrices/GGATGTAAGCGGTTAT-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_success MissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/CGGACAACATTAAGCT-1.qc.bam mgatk/test/temp/ready_bam/CGGACAACATTAAGCT-1.qc.bam.bai mgatk/test/qc/depth/CGGACAACATTAAGCT-1.depth.txt mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.A.txt mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.C.txt mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.G.txt mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.T.txt mgatk/test/temp/sparse_matrices/CGGACAACATTAAGCT-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_successMissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/ACTCAGTAGGAGTAAT-1.qc.bam mgatk/test/temp/ready_bam/ACTCAGTAGGAGTAAT-1.qc.bam.bai mgatk/test/qc/depth/ACTCAGTAGGAGTAAT-1.depth.txt mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.A.txt mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.C.txt mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.G.txt mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.T.txt mgatk/test/temp/sparse_matrices/ACTCAGTAGGAGTAAT-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_success

MissingOutputException in line 21 of /iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/mgatk/bin/snake/Snakefile.Scatter: Job Missing files after 5 seconds: mgatk/test/temp/ready_bam/ATCCAACAGCGGCTGT-1.qc.bam mgatk/test/temp/ready_bam/ATCCAACAGCGGCTGT-1.qc.bam.bai mgatk/test/qc/depth/ATCCAACAGCGGCTGT-1.depth.txt mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.A.txt mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.C.txt mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.G.txt mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.T.txt mgatk/test/temp/sparse_matrices/ATCCAACAGCGGCTGT-1.coverage.txt This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 589, in handle_job_success File "/iblm/netapp/home/asen/myconda/envs/mgatk/lib/python3.9/site-packages/snakemake/executors/init.py", line 252, in handle_job_success Exiting because a job execution failed. Look above for error message Exiting because a job execution failed. Look above for error message Exiting because a job execution failed. Look above for error message Exiting because a job execution failed. Look above for error message Exiting because a job execution failed. Look above for error message Exiting because a job execution failed. Look above for error message Exiting because a job execution failed. Look above for error message Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /iblm/netapp/home/asen/.snakemake/log/2021-06-03T173435.844488.snakemake.log

caleblareau commented 3 years ago

I implemented some changes in v0.6.3 that I believe should solve this-- please comment back on this thread if not!