search

caleblareau / mgatk

mgatk: mitochondrial genome analysis toolkit
http://caleblareau.github.io/mgatk
MIT License
98 stars 25 forks source link

mgatk tenx mode missing depth and sparse_matrices files #63

Open triciapark opened 1 year ago

triciapark commented 1 year ago

Hello, I've been following the wiki to try to run mgatk tenx (v0.6.1) on 10x scRNA data using the following command:

mgatk tenx -bt CB -g mm10 --snake-stdout -c 12 -ub UB -i results/22_6/outs/possorted_genome_bam.bam -b results/22_6/outs/filtered_feature_bc_matrix/barcodes.tsv -o mgatk_results/sample6 -n sample6

But it seems like there are some missing files. This is the log file:

Config file mgatk_results/sample6/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 12
Rules claiming more threads will be scaled down.
Job stats:
job                           count    min threads    max threads
--------------------------  -------  -------------  -------------
all                               1              1              1
call_variants                     1              1              1
make_depth_table                  1              1              1
make_final_sparse_matrices        1              1              1
process_one_slice                12              1              1
total                            16              1              1

Select jobs to execute...

[Thu Jul 21 11:36:25 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.12.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.12.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.coverage.txt
    jobid: 3
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.12.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.A.txt, mgatk_results/sample6/qc/depth/barcodes.12.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.12.coverage.txt
    wildcards: sample=barcodes.12
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:25 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.7.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.7.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.coverage.txt
    jobid: 5
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.7.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.coverage.txt, mgatk_results/sample6/qc/depth/barcodes.7.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.7.G.txt
    wildcards: sample=barcodes.7
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:25 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.11.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.11.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.coverage.txt
    jobid: 7
    reason: Missing output files: mgatk_results/sample6/qc/depth/barcodes.11.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.coverage.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.11.G.txt
    wildcards: sample=barcodes.11
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:25 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.10.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.10.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.coverage.txt
    jobid: 9
    reason: Missing output files: mgatk_results/sample6/qc/depth/barcodes.10.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.coverage.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.10.G.txt
    wildcards: sample=barcodes.10
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.5.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.5.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.coverage.txt
    jobid: 11
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.5.coverage.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.5.T.txt, mgatk_results/sample6/qc/depth/barcodes.5.depth.txt
    wildcards: sample=barcodes.5
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.1.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.1.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.coverage.txt
    jobid: 13
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.1.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.A.txt, mgatk_results/sample6/qc/depth/barcodes.1.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.1.coverage.txt
    wildcards: sample=barcodes.1
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.3.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.3.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.coverage.txt
    jobid: 2
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.3.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.G.txt, mgatk_results/sample6/qc/depth/barcodes.3.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.3.coverage.txt
    wildcards: sample=barcodes.3
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.4.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.4.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.coverage.txt
    jobid: 4
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.4.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.coverage.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.4.T.txt, mgatk_results/sample6/qc/depth/barcodes.4.depth.txt
    wildcards: sample=barcodes.4
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.6.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.6.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.coverage.txt
    jobid: 6
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.6.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.6.coverage.txt, mgatk_results/sample6/qc/depth/barcodes.6.depth.txt
    wildcards: sample=barcodes.6
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.9.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.9.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.coverage.txt
    jobid: 8
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.9.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.coverage.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.9.G.txt, mgatk_results/sample6/qc/depth/barcodes.9.depth.txt
    wildcards: sample=barcodes.9
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.8.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.8.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.coverage.txt
    jobid: 10
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.8.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.A.txt, mgatk_results/sample6/qc/depth/barcodes.8.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.8.coverage.txt
    wildcards: sample=barcodes.8
    resources: tmpdir=/tmp

[Thu Jul 21 11:36:26 2022]
rule process_one_slice:
    input: mgatk_results/sample6/.internal/samples/barcodes.2.bam.txt
    output: mgatk_results/sample6/qc/depth/barcodes.2.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.C.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.T.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.coverage.txt
    jobid: 12
    reason: Missing output files: mgatk_results/sample6/temp/sparse_matrices/barcodes.2.coverage.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.C.txt, mgatk_results/sample6/qc/depth/barcodes.2.depth.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.G.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.A.txt, mgatk_results/sample6/temp/sparse_matrices/barcodes.2.T.txt
    wildcards: sample=barcodes.2
    resources: tmpdir=/tmp

Terminating processes on user request, this might take some time.
Complete log: .snakemake/log/2022-07-21T113623.063765.snakemake.log

And ls -lRh of the output folder shows this:

.:
total 5.0K
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:14 fasta
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:14 final
drwxr-xr-x 4 parkt reznik 4.0K Jul 21 11:36 logs
drwxr-xr-x 3 parkt reznik 4.0K Jul 21 11:38 qc
drwxr-xr-x 7 parkt reznik 4.0K Jul 21 11:38 temp

./fasta:
total 33K
-rw-r--r-- 1 parkt reznik 17K Jul 21 11:14 chrM.fasta
-rw-r--r-- 1 parkt reznik  19 Jul 21 11:14 chrM.fasta.fai

./final:
total 128K
-rw-r--r-- 1 parkt reznik 117K Jul 21 11:14 chrM_refAllele.txt

./logs:
total 3.0K
-rw-r--r-- 1 parkt reznik  124 Jul 21 11:36 base.mgatk.log
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:37 filterlogs
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:36 rmdupslogs
-rw-r--r-- 1 parkt reznik  530 Jul 21 11:36 sample6.parameters.txt

./logs/filterlogs:
total 6.0K
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.10.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.11.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.12.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.1.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.2.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.3.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.4.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.5.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.6.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.7.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.8.filter.log
-rw-r--r-- 1 parkt reznik 26 Jul 21 11:37 barcodes.9.filter.log

./logs/rmdupslogs:
total 0

./qc:
total 1.0K
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:36 quality

./qc/quality:
total 0

./temp:
total 5.0K
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:36 barcoded_bams
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:14 barcode_files
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:36 quality
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:36 ready_bam
drwxr-xr-x 2 parkt reznik 4.0K Jul 21 11:37 temp_bam

./temp/barcoded_bams:
total 163M
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.10.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.10.bam.bai
-rw-r--r-- 1 parkt reznik 13M Jul 21 11:36 barcodes.11.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.11.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.12.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.12.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.1.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.1.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.2.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.2.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.3.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.3.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.4.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.4.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.5.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.5.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.6.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.6.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.7.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.7.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.8.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.8.bam.bai
-rw-r--r-- 1 parkt reznik 14M Jul 21 11:36 barcodes.9.bam
-rw-r--r-- 1 parkt reznik 616 Jul 21 11:36 barcodes.9.bam.bai

./temp/barcode_files:
total 384K
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.10.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.11.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.12.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.1.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.2.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.3.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.4.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.5.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.6.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.7.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.8.txt
-rw-r--r-- 1 parkt reznik 26K Jul 21 11:14 barcodes.9.txt

./temp/quality:
total 0

./temp/ready_bam:
total 0

./temp/temp_bam:
total 111M
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.10.temp0.bam
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.10.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.10.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.5M Jul 21 11:37 barcodes.11.temp0.bam
-rw-r--r-- 1 parkt reznik 4.5M Jul 21 11:37 barcodes.11.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.11.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.12.temp0.bam
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.12.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.12.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.1.temp0.bam
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.1.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.1.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.8M Jul 21 11:37 barcodes.2.temp0.bam
-rw-r--r-- 1 parkt reznik 4.8M Jul 21 11:37 barcodes.2.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.2.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.8M Jul 21 11:37 barcodes.3.temp0.bam
-rw-r--r-- 1 parkt reznik 4.8M Jul 21 11:37 barcodes.3.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.3.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.4.temp0.bam
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.4.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.4.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.5.temp0.bam
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.5.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.5.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.6.temp0.bam
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.6.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.6.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.7.temp0.bam
-rw-r--r-- 1 parkt reznik 4.7M Jul 21 11:37 barcodes.7.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.7.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.5M Jul 21 11:37 barcodes.8.temp0.bam
-rw-r--r-- 1 parkt reznik 4.5M Jul 21 11:37 barcodes.8.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.8.temp1.bam.bai
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.9.temp0.bam
-rw-r--r-- 1 parkt reznik 4.6M Jul 21 11:37 barcodes.9.temp1.bam
-rw-r--r-- 1 parkt reznik  616 Jul 21 11:37 barcodes.9.temp1.bam.bai

Additional Information: I was able to run mgatk on the example code with the call mgatk tenx -i test_barcode.bam -b test_barcodes.txt -o mgatk_results/test -n test -bt CB -c 2 I am using the following versions: Python 3.9.7 R 4.0.0 Snakemake 7.8.5

teresa1997 commented 1 year ago

Hi, we have a similar problem. I'm running mgatk in tenx mode

mgatk tenx -n 10 -i /Output_CellRanger/AUGU_possorted_genome_bam.bam -n AUGU -o /Output_CellRanger/output_mgatk -b /Output_CellRanger/AUGU_barcodes.txt -bt CB

But as @triciapark we have some missing files.

Here the log 

Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job stats:
job                           count    min threads    max threads
--------------------------  -------  -------------  -------------
all                               1              1              1
call_variants                     1              1              1
make_depth_table                  1              1              1
make_final_sparse_matrices        1              1              1
process_one_slice                10              1              1
total                            14              1              1

Select jobs to execute...

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.5.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.5.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.coverage.txt
    jobid: 5
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.T.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.5.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.5.coverage.txt
    wildcards: sample=barcodes.5
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.6.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.6.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.coverage.txt
    jobid: 10
    reason: Missing output files: /Output_CellRanger/output_mgatk/qc/depth/barcodes.6.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.coverage.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.6.T.txt
    wildcards: sample=barcodes.6
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.3.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.3.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.coverage.txt
    jobid: 7
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.coverage.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.T.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.3.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.3.G.txt
    wildcards: sample=barcodes.3
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.1.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.1.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.coverage.txt
    jobid: 4
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.A.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.1.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.coverage.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.1.G.txt
    wildcards: sample=barcodes.1
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.4.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.4.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.coverage.txt
    jobid: 3
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.4.coverage.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.4.depth.txt
    wildcards: sample=barcodes.4
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.2.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.2.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.coverage.txt
    jobid: 9
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.C.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.2.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.coverage.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.2.G.txt
    wildcards: sample=barcodes.2
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.9.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.9.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.coverage.txt
    jobid: 6
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.T.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.9.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.9.coverage.txt
    wildcards: sample=barcodes.9
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.8.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.8.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.coverage.txt
    jobid: 2
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.coverage.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.T.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.8.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.8.C.txt
    wildcards: sample=barcodes.8
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.10.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.10.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.coverage.txt
    jobid: 11
    reason: Missing output files: /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.coverage.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.10.G.txt, /Output_CellRanger/output_mgatk/qc/depth/barcodes.10.depth.txt
    wildcards: sample=barcodes.10
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 10:57:08 2023]
rule process_one_slice:
    input: /Output_CellRanger/output_mgatk/.internal/samples/barcodes.7.bam.txt
    output: /Output_CellRanger/output_mgatk/qc/depth/barcodes.7.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.coverage.txt
    jobid: 8
    reason: Missing output files: 
/Output_CellRanger/output_mgatk/qc/depth/barcodes.7.depth.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.G.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.T.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.C.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.A.txt, /Output_CellRanger/output_mgatk/temp/sparse_matrices/barcodes.7.coverage.txt
    wildcards: sample=barcodes.7
    resources: tmpdir=/scratch/tmp

Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Building DAG of jobs...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Config file /Output_CellRanger/output_mgatk/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Building DAG of jobs...
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Error in checkGrep(grep(".A.txt", files)) : 
  Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep
Execution halted

This is the listo of the output files:

.:
total 30K
drwxr-sr-x 2 cvt   3 Jul  3 09:45 final
drwxr-sr-x 4 cvt   7 Jul  3 09:46 logs
drwxr-sr-x 4 cvt   4 Jul  3 09:46 qc

./final:
total 114K
-rw-r--r-- 1 cvt 119K Jul  4 10:55 chrM_refAllele.txt

./logs:
total 9.0K
-rw-r--r-- 1 cvt  778 Jul  4 10:57 AUGU.parameters.txt
-rw-r--r-- 1 cvt    0 Jul  4 10:57 AUGU.snakemake_tenx.log
-rw-r--r-- 1 cvt  4.9K Jul  4 10:57 base.mgatk.log
drwxr-sr-x 2 cvt    2 Jul  3 09:46 filterlogs
drwxr-sr-x 2 cvt    2 Jul  3 09:46 rmdupslogs

./logs/filterlogs:
total 0

./logs/rmdupslogs:
total 0

./qc:
total 1.0K
drwxr-sr-x 2 cvt 2 Jul  3 09:46 depth
drwxr-sr-x 2 cvt 2 Jul  3 09:46 quality

./qc/depth:
total 0

./qc/quality:
total 0

I also tried with the test file 

mgatk bcall -i test_barcode.bam -n bc1 -o bc1dmem -bt CB -b test_barcodes.txt -z`

but I have the same error 

`[W::hts_idx_load3] The index file is older than the data file: bc1dmem/temp/barcoded_bams/GCCTAGGCAGTTCGGC-1.bam.bai
[W::hts_idx_load3] The index file is older than the data file: bc1dmem/temp/barcoded_bams/CTAACTTAGAGCCACA-1.bam.bai
[W::hts_idx_load3] The index file is older than the data file: bc1dmem/temp/barcoded_bams/CACCACTAGGAGGCGA-1.bam.bai
Config file bc1dmem/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line.
Config file bc1dmem/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Building DAG of jobs...
Using shell: /usr/bin/bash
Using shell: /usr/bin/bash
Provided cores: 10
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Rules claiming more threads will be scaled down.
Job stats:
job                   count    min threads    max threads
------------------  -------  -------------  -------------
all                       1              1              1
make_sample_list          1              1              1
process_one_sample        6              1              1
total                     8              1              1

Job stats:
job                           count    min threads    max threads
--------------------------  -------  -------------  -------------
all                               1              1              1
make_depth_table                  1              1              1
make_final_sparse_matrices        1              1              1
total                             3              1              1

Select jobs to execute...
Select jobs to execute...

[Tue Jul  4 13:20:36 2023]
rule make_depth_table:
    output: bc1dmem/final/bc1.depthTable.txt
    jobid: 1
    reason: Missing output files: bc1dmem/final/bc1.depthTable.txt
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 13:20:36 2023]
rule process_one_sample:
    input: bc1dmem/.internal/samples/CTAACTTAGAGCCACA-1.bam.txt
    output: bc1dmem/temp/ready_bam/CTAACTTAGAGCCACA-1.qc.bam, bc1dmem/temp/ready_bam/CTAACTTAGAGCCACA-1.qc.bam.bai, bc1dmem/qc/depth/CTAACTTAGAGCCACA-1.depth.txt, bc1dmem/temp/sparse_matrices/CTAACTTAGAGCCACA-1.A.txt, bc1dmem/temp/sparse_matrices/CTAACTTAGAGCCACA-1.C.txt, bc1dmem/temp/sparse_matrices/CTAACTTAGAGCCACA-1.G.txt, bc1dmem/temp/sparse_matrices/CTAACTTAGAGCCACA-1.T.txt, bc1dmem/temp/sparse_matrices/CTAACTTAGAGCCACA-1.coverage.txt
    jobid: 6
    reason: Missing output files: bc1dmem/qc/depth/CTAACTTAGAGCCACA-1.depth.txt
    wildcards: sample=CTAACTTAGAGCCACA-1
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 13:20:36 2023]
rule process_one_sample:
    input: bc1dmem/.internal/samples/barcodes.3.bam.txt
    output: bc1dmem/temp/ready_bam/barcodes.3.qc.bam, bc1dmem/temp/ready_bam/barcodes.3.qc.bam.bai, bc1dmem/qc/depth/barcodes.3.depth.txt, bc1dmem/temp/sparse_matrices/barcodes.3.A.txt, bc1dmem/temp/sparse_matrices/barcodes.3.C.txt, bc1dmem/temp/sparse_matrices/barcodes.3.G.txt, bc1dmem/temp/sparse_matrices/barcodes.3.T.txt, bc1dmem/temp/sparse_matrices/barcodes.3.coverage.txt
    jobid: 4
    reason: Missing output files: bc1dmem/qc/depth/barcodes.3.depth.txt
    wildcards: sample=barcodes.3
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 13:20:36 2023]
rule process_one_sample:
    input: bc1dmem/.internal/samples/CACCACTAGGAGGCGA-1.bam.txt
    output: bc1dmem/temp/ready_bam/CACCACTAGGAGGCGA-1.qc.bam, bc1dmem/temp/ready_bam/CACCACTAGGAGGCGA-1.qc.bam.bai, bc1dmem/qc/depth/CACCACTAGGAGGCGA-1.depth.txt, bc1dmem/temp/sparse_matrices/CACCACTAGGAGGCGA-1.A.txt, bc1dmem/temp/sparse_matrices/CACCACTAGGAGGCGA-1.C.txt, bc1dmem/temp/sparse_matrices/CACCACTAGGAGGCGA-1.G.txt, bc1dmem/temp/sparse_matrices/CACCACTAGGAGGCGA-1.T.txt, bc1dmem/temp/sparse_matrices/CACCACTAGGAGGCGA-1.coverage.txt
    jobid: 5
    reason: Missing output files: bc1dmem/qc/depth/CACCACTAGGAGGCGA-1.depth.txt
    wildcards: sample=CACCACTAGGAGGCGA-1
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 13:20:36 2023]
rule process_one_sample:
    input: bc1dmem/.internal/samples/barcodes.2.bam.txt
    output: bc1dmem/temp/ready_bam/barcodes.2.qc.bam, bc1dmem/temp/ready_bam/barcodes.2.qc.bam.bai, bc1dmem/qc/depth/barcodes.2.depth.txt, bc1dmem/temp/sparse_matrices/barcodes.2.A.txt, bc1dmem/temp/sparse_matrices/barcodes.2.C.txt, bc1dmem/temp/sparse_matrices/barcodes.2.G.txt, bc1dmem/temp/sparse_matrices/barcodes.2.T.txt, bc1dmem/temp/sparse_matrices/barcodes.2.coverage.txt
    jobid: 7
    reason: Missing output files: bc1dmem/qc/depth/barcodes.2.depth.txt
    wildcards: sample=barcodes.2
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 13:20:36 2023]
rule process_one_sample:
    input: bc1dmem/.internal/samples/barcodes.1.bam.txt
    output: bc1dmem/temp/ready_bam/barcodes.1.qc.bam, bc1dmem/temp/ready_bam/barcodes.1.qc.bam.bai, bc1dmem/qc/depth/barcodes.1.depth.txt, bc1dmem/temp/sparse_matrices/barcodes.1.A.txt, bc1dmem/temp/sparse_matrices/barcodes.1.C.txt, bc1dmem/temp/sparse_matrices/barcodes.1.G.txt, bc1dmem/temp/sparse_matrices/barcodes.1.T.txt, bc1dmem/temp/sparse_matrices/barcodes.1.coverage.txt
    jobid: 2
    reason: Missing output files: bc1dmem/qc/depth/barcodes.1.depth.txt
    wildcards: sample=barcodes.1
    resources: tmpdir=/scratch/tmp

[Tue Jul  4 13:20:36 2023]
rule process_one_sample:
    input: bc1dmem/.internal/samples/GCCTAGGCAGTTCGGC-1.bam.txt
    output: bc1dmem/temp/ready_bam/GCCTAGGCAGTTCGGC-1.qc.bam, bc1dmem/temp/ready_bam/GCCTAGGCAGTTCGGC-1.qc.bam.bai, bc1dmem/qc/depth/GCCTAGGCAGTTCGGC-1.depth.txt, bc1dmem/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.A.txt, bc1dmem/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.C.txt, bc1dmem/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.G.txt, bc1dmem/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.T.txt, bc1dmem/temp/sparse_matrices/GCCTAGGCAGTTCGGC-1.coverage.txt
    jobid: 3
    reason: Missing output files: bc1dmem/qc/depth/GCCTAGGCAGTTCGGC-1.depth.txt
    wildcards: sample=GCCTAGGCAGTTCGGC-1
    resources: tmpdir=/scratch/tmp

Config file bc1dmem/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line.
Config file bc1dmem/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Config file bc1dmem/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Config file bc1dmem/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Building DAG of jobs...
Building DAG of jobs...
Building DAG of jobs...
Config file bc1dmem/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Config file bc1dmem/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Config file bc1dmem/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Using shell: /usr/bin/bash
Select jobs to execute...
Provided cores: 10
Rules claiming more threads will be scaled down.
Select jobs to execute...
[Tue Jul  4 13:20:39 2023]
Finished job 1.
1 of 3 steps (33%) done
Select jobs to execute...

[Tue Jul  4 13:20:39 2023]
rule make_final_sparse_matrices:
    output: bc1dmem/final/bc1.A.txt.gz, bc1dmem/final/bc1.C.txt.gz, bc1dmem/final/bc1.G.txt.gz, bc1dmem/final/bc1.T.txt.gz, bc1dmem/final/bc1.coverage.txt.gz
    jobid: 2
    reason: Missing output files: bc1dmem/final/bc1.G.txt.gz, bc1dmem/final/bc1.A.txt.gz, bc1dmem/final/bc1.C.txt.gz, bc1dmem/final/bc1.T.txt.gz, bc1dmem/final/bc1.coverage.txt.gz
    resources: tmpdir=/scratch/tmp

[W::hts_idx_load3] The index file is older than the data file: bc1dmem/temp/barcoded_bams/GCCTAGGCAGTTCGGC-1.bam.bai
[W::hts_idx_load3] The index file is older than the data file: bc1dmem/temp/barcoded_bams/CACCACTAGGAGGCGA-1.bam.bai
[W::hts_idx_load3] The index file is older than the data file: bc1dmem/temp/barcoded_bams/CTAACTTAGAGCCACA-1.bam.bai
Config file bc1dmem/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
gzip: bc1dmem/final/bc1.A.txt: No such file or directory
gzip: bc1dmem/final/bc1.C.txt: No such file or directory
gzip: bc1dmem/final/bc1.G.txt: No such file or directory
gzip: bc1dmem/final/bc1.T.txt: No such file or directory
gzip: bc1dmem/final/bc1.coverage.txt: No such file or directory
Waiting at most 5 seconds for missing files.
Error in checkGrep(grep(".A.txt", files)) : 
  Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep
Execution halted

I am using the following versions: Python 3.11.4 R 4.2.0 Snakemake 7.30.1

caleblareau commented 11 months ago

Hi all, apologies for the delayed response-- these seem like two separate issues.

@triciapark it looks like you are erroring out in the remove duplicates step-- do you have java in your environment?

@teresa1997 your issue looks to be where the final R script is not running that collates the data. My guess is that you are missing R packages (GenomicRanges, data.table, SummarizedExperiment, Matrix)

teresa1997 commented 11 months ago

Hi @caleblareau thanks for the answer. The packages are installed. Any other guess?

FionaMoon commented 10 months ago

Hi @caleblareau I also met this problem. All required R package are installed. My java version is

openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Zulu 8.62.0.19-CA-linux64) (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Zulu 8.62.0.19-CA-linux64) (build 25.332-b09, mixed mode)