Error in checkGrep(grep(".A.txt", files)) when running maegatk

[uqjlu8]

Hi I am trying to run maegatk on my dataset. I have installed all the modules required as stated in the tutorial. java, bwa, bedtools, freebayes, R (4.1.2, with data.table, Matrix, GenomicRanges, SummarizedExperiment). I am running it on python 3.7

I have tried to run the program on both the test dataset, and my own dataset using the commands below:

maegatk bcall --input $bam -o $resul_out -c $ncores -b $barcodes -mr $minReads -z

I keep getting the same error in both instances:

Mon Mar 14 15:46:24 AEST 2022: maegatk v0.1.1 Mon Mar 14 15:46:24 AEST 2022: Found bam file: Data/test_maester.bam for genotyping. Mon Mar 14 15:46:24 AEST 2022: Will determine barcodes with at least: 100 mitochondrial reads. Mon Mar 14 15:46:24 AEST 2022: User specified mitochondrial genome matches .bam file Mon Mar 14 15:46:30 AEST 2022: Finished determining/splitting barcodes for genotyping. Mon Mar 14 15:46:31 AEST 2022: Genotyping samples with 24 threads Error in checkGrep(grep(".A.txt", files)) : Improper folder specification; file missing / extra file present. See documentation Calls: importMito -> checkGrep Execution halted

I have attached the a list of all the files generated using (ls -lRh $result_folder), scatter.log, gather.log

test_result_file_list.txt maegatk.snakemake_scatter.log.txt maegatk.snakemake_gather.log.txt

Any help would be greatly appreciated.

Thanks

[WenxuLiu]

i got the same problem.. expecting resolutions.

[andrecossa5]

Hi,

I am experiencing the same problem on test data. Thank you for handling this :)

[uqjlu8]

Also, the definition of -mr (minimum reads) was used incorrectly. The function used to call -C in freebayes is actually minimum alternative read, which is not the same thing as minimum read.

[caleblareau]

Also, the definition of -mr (minimum reads) was used incorrectly. The function used to call -C in freebayes is actually minimum alternative read, which is not the same thing as minimum read.

Here we use -mr only in the context of the fgbio consensus demultiplexing (https://github.com/caleblareau/maegatk/blob/6662c7c4188c2c254e9416450295553700473794/maegatk/bin/python/oneSample_maegatk.py#L70)

[caleblareau]

Hi @uqjlu8, your error stems from an improperly specified bam file header:

Error in the scatter log (line 83):

ERROR::READ_GROUP_NOT_FOUND:Record 1, Read name NS500239:389:H3F5JBGXC:1:12210:7650:20169_GTCACAATCGGATGGA-2+CTGTCACATT, RG ID on SAMRecord not found in header: Gen22_LG_D22:0:1:H3F5JBGXC:1

This biostars post discusses it a bit more: https://www.biostars.org/p/50338/

In short, this is an error upstream of the maegatk took and something that would require you to check with how you processed your single-cell sequencing data. May I ask what pipeline you used to go from fastq to bam files?

[uqjlu8]

Hi Caleb The original *.bam was generated using cellranger count to do the mapping of the fastqs

Best regards

Jennifer

---

From: Caleb Lareau @.> Sent: Saturday, 9 April 2022 3:59 PM To: caleblareau/maegatk @.> Cc: Miss Jennifer Lu @.>; Mention @.> Subject: Re: [caleblareau/maegatk] Error in checkGrep(grep(".A.txt", files)) when running maegatk (Issue #5)

Hi @uqjlu8https://github.com/uqjlu8, your error stems from an improperly specified bam file header:

Error in the scatter log (line 83):

ERROR::READ_GROUP_NOT_FOUND:Record 1, Read name NS500239:389:H3F5JBGXC:1:12210:7650:20169_GTCACAATCGGATGGA-2+CTGTCACATT, RG ID on SAMRecord not found in header: Gen22_LG_D22:0:1:H3F5JBGXC:1

This biostars post discusses it a bit more: https://www.biostars.org/p/50338/

In short, this is an error upstream of the maegatk took and something that would require you to check with how you processed your single-cell sequencing data. May I ask what pipeline you used to go from fastq to bam files?

— Reply to this email directly, view it on GitHubhttps://github.com/caleblareau/maegatk/issues/5#issuecomment-1093720510, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AL7N3ZNRQ3FEDSFDR2YEYU3VEEMDBANCNFSM5QUQIDVA. You are receiving this because you were mentioned.Message ID: @.***>

[caleblareau]

just to clarify-- did the test data work OK for you?

[uqjlu8]

Hi Caleb, unfortunately not. I was getting the same CheckGrep error. That's why I submitted the original ticket

Regards Jennifer

---

From: Caleb Lareau @.> Sent: Saturday, 23 April 2022 11:39 AM To: caleblareau/maegatk @.> Cc: Miss Jennifer Lu @.>; Mention @.> Subject: Re: [caleblareau/maegatk] Error in checkGrep(grep(".A.txt", files)) when running maegatk (Issue #5)

just to clarify-- did the test data work OK for you?

— Reply to this email directly, view it on GitHubhttps://github.com/caleblareau/maegatk/issues/5#issuecomment-1107162348, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AL7N3ZLA7LULPEJNS4DIPETVGNIGRANCNFSM5QUQIDVA. You are receiving this because you were mentioned.Message ID: @.***>

[ttab963]

Hello @caleblareau, I also face same problem.

Test data or my own data cannot proceed at making ~/final/SAMPLE_NAME.A.txt.gz part. Files are intact and I am quite sure this(https://www.biostars.org/p/50338/) is not the point.

Hope you provide any response. Thanks.

[caleblareau]

Hmmm okay— I don’t see what an obvious solution is here other than trying to manipulate the input bam file header… What is samtools view -H of the input bam file?

On Apr 9, 2022, at 1:45 AM, uqjlu8 @.**@.>> wrote:

Hi Caleb The original *.bam was generated using cellranger count to do the mapping of the fastqs

Best regards

Jennifer

---

From: Caleb Lareau @.> Sent: Saturday, 9 April 2022 3:59 PM To: caleblareau/maegatk @.> Cc: Miss Jennifer Lu @.>; Mention @.> Subject: Re: [caleblareau/maegatk] Error in checkGrep(grep(".A.txt", files)) when running maegatk (Issue #5)

Hi @uqjlu8https://github.com/uqjlu8, your error stems from an improperly specified bam file header:

Error in the scatter log (line 83):

ERROR::READ_GROUP_NOT_FOUND:Record 1, Read name NS500239:389:H3F5JBGXC:1:12210:7650:20169_GTCACAATCGGATGGA-2+CTGTCACATT, RG ID on SAMRecord not found in header: Gen22_LG_D22:0:1:H3F5JBGXC:1

This biostars post discusses it a bit more: https://www.biostars.org/p/50338/

In short, this is an error upstream of the maegatk took and something that would require you to check with how you processed your single-cell sequencing data. May I ask what pipeline you used to go from fastq to bam files?

— Reply to this email directly, view it on GitHubhttps://github.com/caleblareau/maegatk/issues/5#issuecomment-1093720510, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AL7N3ZNRQ3FEDSFDR2YEYU3VEEMDBANCNFSM5QUQIDVA. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHubhttps://github.com/caleblareau/maegatk/issues/5#issuecomment-1093820925, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD32FYJTMXZZ6Z6K7NS2MSLVEE7UHANCNFSM5QUQIDVA. You are receiving this because you commented.Message ID: @.***>

[showteeth]

I got the same problem with the test data.

I noticed that the version is 0.1.2 in setup.py, and the PyPi version is 0.1.1. Can the difference between the two versions solve this problem?

YB

[caleblareau]

version 0.1.2 was for indel calling, so I don't think it'll impact anything for you. You can try though-- just clone the repo then install it from the local branch:

`` python3 -m pip install -e maegatk

[si3]

Hello

I am running into a similar issue. It seems like this might be related to this: https://github.com/caleblareau/mgatk/issues/22

I ran maegatk with --snake-sdout flag and the error stems from the following:

AttributeError in line 54 of ~/venv3/lib/python3.6/site-packages/maegatk/bin/snake/Snakefile.maegatk.Gather 'InputFiles' object has no attribute 'As'

Any idea on how to work around this?

Stef

[noranekonobokkusu]

Hi @si3 the README file is now slightly expanded and explains what intermediate output files you should expect. If there is no As in Gather, then it means that line 35 here fails and the output of Scatter is incomplete, so I suggest looking into the Scatter log. Scatter should have generated a set of files in temp_bam,ready_bam,sparse_matrices folders. If you are keeping intermediate files, you can check which files are missing, it should also be reflected in Scatter's log. These are the files that should be present for each CB:

-> l temp_bam/ | head
total 215944
-rw-r--r-- 1 safina vangalenlab  1387286 Jan  6 15:57 CCCGTCGTGGTA-1.temp0.bam
-rw-r--r-- 1 safina vangalenlab  1365836 Jan  6 15:57 CCCGTCGTGGTA-1.temp1.bam
-rw-r--r-- 1 safina vangalenlab 13993678 Jan  6 15:57 CCCGTCGTGGTA-1.temp1.5.sam
-rw-r--r-- 1 safina vangalenlab  1320355 Jan  6 15:57 CCCGTCGTGGTA-1.temp1.5.bam
-rw-r--r-- 1 safina vangalenlab   489313 Jan  6 15:57 CCCGTCGTGGTA-1.temp2.bam
-rw-r--r-- 1 safina vangalenlab  1925330 Jan  6 15:57 CCCGTCGTGGTA-1.temp0.fastq
-rw-r--r-- 1 safina vangalenlab  1123027 Jan  6 15:57 GACCGTGCATTT-1.temp0.bam
-rw-r--r-- 1 safina vangalenlab  1122782 Jan  6 15:57 GACCGTGCATTT-1.temp1.bam
-rw-r--r-- 1 safina vangalenlab 11246101 Jan  6 15:57 GACCGTGCATTT-1.temp1.5.sam
(base) 

-> l ready_bam/ | head
total 6728
-rw-r--r-- 1 safina vangalenlab 216594 Jan  6 15:57 CCCGTCGTGGTA-1.qc.bam
-rw-r--r-- 1 safina vangalenlab     96 Jan  6 15:57 CCCGTCGTGGTA-1.qc.bam.bai
-rw-r--r-- 1 safina vangalenlab 219598 Jan  6 15:57 GACCGTGCATTT-1.qc.bam
-rw-r--r-- 1 safina vangalenlab     96 Jan  6 15:57 GACCGTGCATTT-1.qc.bam.bai
-rw-r--r-- 1 safina vangalenlab 189703 Jan  6 15:58 CCACAAAACATG-1.qc.bam
-rw-r--r-- 1 safina vangalenlab     96 Jan  6 15:58 CCACAAAACATG-1.qc.bam.bai
-rw-r--r-- 1 safina vangalenlab 261193 Jan  6 15:58 GGCGCTAATGAA-1.qc.bam
-rw-r--r-- 1 safina vangalenlab     96 Jan  6 15:58 GGCGCTAATGAA-1.qc.bam.bai
-rw-r--r-- 1 safina vangalenlab 219108 Jan  6 15:59 TTCCTACGCAAT-1.qc.bam
(base) 

-> l sparse_matrices/ | head
total 27904
-rw-r--r-- 1 safina vangalenlab 159767 Jan  6 15:57 CCCGTCGTGGTA-1.A.txt
-rw-r--r-- 1 safina vangalenlab 159806 Jan  6 15:57 CCCGTCGTGGTA-1.C.txt
-rw-r--r-- 1 safina vangalenlab  77507 Jan  6 15:57 CCCGTCGTGGTA-1.G.txt
-rw-r--r-- 1 safina vangalenlab 125697 Jan  6 15:57 CCCGTCGTGGTA-1.T.txt
-rw-r--r-- 1 safina vangalenlab 308142 Jan  6 15:57 CCCGTCGTGGTA-1.coverage.txt
-rw-r--r-- 1 safina vangalenlab 162181 Jan  6 15:57 GACCGTGCATTT-1.A.txt
-rw-r--r-- 1 safina vangalenlab 160649 Jan  6 15:57 GACCGTGCATTT-1.C.txt
-rw-r--r-- 1 safina vangalenlab  77111 Jan  6 15:57 GACCGTGCATTT-1.G.txt
-rw-r--r-- 1 safina vangalenlab 129431 Jan  6 15:57 GACCGTGCATTT-1.T.txt

Do you have all of these files?

[NBurnaevskiy]

Same error here. Did you have a chance to find a fix? Thank you.

[noranekonobokkusu]

Hi @NBurnaevskiy,

were you able to identify which part of the pipeline failed? Try going through the content of intermediate files as described in https://github.com/caleblareau/maegatk?tab=readme-ov-file#output-files

[NBurnaevskiy]

@noranekonobokkusu , Thank you for your reply. I looked into the temp directory. There is expected number of bam files in the 'barcoded_bams' folder. However, 'temp_bam' and 'ready_bam' are empty. 'sparse_matrices' is also empty. Would you have ideas how to proceed?

[noranekonobokkusu]

Yes, try looking into the snakemake-scatter text log file in the logs/ folder, it should have some error messages.

Also note that a recently added option --skip-barcodesplit allows you to skip the barcode splitting step if it already finished successfully. It shouldn't matter if you are running maegatk on a small test dataset.

[NBurnaevskiy]

I actually don't have that file. Here is the content of the 'logs' filterlogs rmdupslogs base.maegatk.log maegatk.parameters.txt maegatk.snakemake_gather.log

[noranekonobokkusu]

@NBurnaevskiy this is new. Are you running it on a test dataset? Are there any errors (or any messages) in the output of this run? (like sh.o or sh.e files, or just stdout/err?)

I also noticed that yesterday you opened and then closed a yaml-related issue. Might it be that you introduced a fix that causes snakemake-scatter not to be executed?

[noranekonobokkusu]

If you are running a test file (which is recommended for debugging), can you delete the entire output directory and re-run it from scratch? And then see what is in logs/base.maegatk.log and whether you have /.internal/parseltongue/snake.scatter.yaml which are instructions for scatter

[NBurnaevskiy]

Yes, we had to introduce a fix into yaml command. We followed the instruction from error message and yaml documentation. We had error message:

"AttributeError: "dump()" has been removed, use

yaml = YAML(typ='unsafe', pure=True) yaml.dump(...)

instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/cli.py", line 300

yaml.dump(dict1, yaml_file, default_flow_style=False, Dumper=yaml.RoundTripDumper)"

In the cli.py file we changed line 300 from yaml.dump(dict1, yaml_file, default_flow_style=False, Dumper=yaml.RoundTripDumper)

to yaml=YAML() yaml.default_flow_style = False yaml.dump(dict1, yaml_file)

That allow the script not to crash as that point. We will now to change to what error message recommended: yaml = YAML(typ='unsafe', pure=True) yaml.dump(dict1, yaml_file, default_flow_style=False, Dumper=yaml.RoundTripDumper)

Do you think that could cause current issue with temp files?

[noranekonobokkusu]

I suspect recent modifications @caleblareau introduced on yaml are not present in the current python package on PyPi. Can you try replacing your entire cli.py with the code from https://github.com/caleblareau/maegatk/blob/master/maegatk/cli.py, and then try suggestions from my previous comment and report results?

[NBurnaevskiy]

I just ran a script on the test file (with our current yaml). Exactly the same result.

[NBurnaevskiy]

I am waiting for our admin to change yaml file.

[noranekonobokkusu]

You can try installing maegatk locally which would allow you to modify the code yourself. Old yaml commands got outdated with the current version of yaml. I would try updating the maegatk code rather than downgrading yaml version.

[NBurnaevskiy]

Just got response from our admin. They did exactly what you suggested, replaced old yaml file with updated one.

Test run failed with identical symptoms.

Yes, we do have have file snake.scatter.yaml in ./internal/parseltongue folder

[NBurnaevskiy]

Error log from maegatk.snakemake_gather.log

Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Job stats: job count

---

all 1 make_depth_table 1 make_final_sparse_matrices 1 total 3

Select jobs to execute...

[Thu Mar 21 10:13:43 2024] rule make_final_sparse_matrices: output: /home/nburnaevskiy/maegatk_test/output/final/maegatk.A.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.C.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.G.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.T.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.coverage.txt.gz jobid: 2 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/final/maegatk.A.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.C.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.coverage.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.G.txt.gz, /home/nburnaevskiy/maegatk_test/output/final/maegatk.T.txt.gz resources: tmpdir=/tmp

[Thu Mar 21 10:13:43 2024] rule make_depth_table: output: /home/nburnaevskiy/maegatk_test/output/final/maegatk.depthTable.txt jobid: 1 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/final/maegatk.depthTable.txt resources: tmpdir=/tmp

Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... gzip: /home/nburnaevskiy/maegatk_test/output/final/maegatk.A.txt: No such file or directory gzip: /home/nburnaevskiy/maegatk_test/output/final/maegatk.C.txt: No such file or directory gzip: /home/nburnaevskiy/maegatk_test/output/final/maegatk.G.txt: No such file or directory gzip: /home/nburnaevskiy/maegatk_test/output/final/maegatk.T.txt: No such file or directory gzip: /home/nburnaevskiy/maegatk_test/output/final/maegatk.coverage.txt: No such file or directory Waiting at most 5 seconds for missing files. [Thu Mar 21 10:13:44 2024] Finished job 1. 1 of 3 steps (33%) done MissingOutputException in rule make_final_sparse_matrices in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Gather, line 33: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/final/maegatk.A.txt.gz /home/nburnaevskiy/maegatk_test/output/final/maegatk.C.txt.gz /home/nburnaevskiy/maegatk_test/output/final/maegatk.G.txt.gz /home/nburnaevskiy/maegatk_test/output/final/maegatk.T.txt.gz /home/nburnaevskiy/maegatk_test/output/final/maegatk.coverage.txt.gz Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-03-21T101342.762479.snakemake.log

[NBurnaevskiy]

Now the tool produced slightly more results but crashed anyway. We finally the file maegatk.snakemake_scatter.log. Looks like yaml gives a problem in another file.

Its content Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Job stats: job count

---

all 1 make_sample_list 1 process_one_sample 10 total 12

Select jobs to execute...

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/GGGTTCGCCTCC-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGGTTCGCCTCC-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGGTTCGCCTCC-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/GGGTTCGCCTCC-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.coverage.txt jobid: 8 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/GGGTTCGCCTCC-1.depth.txt wildcards: sample=GGGTTCGCCTCC-1 resources: tmpdir=/tmp

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/TGGTAGTGAACC-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TGGTAGTGAACC-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TGGTAGTGAACC-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/TGGTAGTGAACC-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.coverage.txt jobid: 10 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/TGGTAGTGAACC-1.depth.txt wildcards: sample=TGGTAGTGAACC-1 resources: tmpdir=/tmp

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/ACAATTAGCACT-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/ACAATTAGCACT-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/ACAATTAGCACT-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/ACAATTAGCACT-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.coverage.txt jobid: 2 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/ACAATTAGCACT-1.depth.txt wildcards: sample=ACAATTAGCACT-1 resources: tmpdir=/tmp

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/CCACAAAACATG-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCACAAAACATG-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCACAAAACATG-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/CCACAAAACATG-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.coverage.txt jobid: 3 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/CCACAAAACATG-1.depth.txt wildcards: sample=CCACAAAACATG-1 resources: tmpdir=/tmp

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/CTAACCCGGAAT-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CTAACCCGGAAT-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CTAACCCGGAAT-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/CTAACCCGGAAT-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.coverage.txt jobid: 5 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/CTAACCCGGAAT-1.depth.txt wildcards: sample=CTAACCCGGAAT-1 resources: tmpdir=/tmp

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/GGCGCTAATGAA-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGCGCTAATGAA-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGCGCTAATGAA-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/GGCGCTAATGAA-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.coverage.txt jobid: 7 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/GGCGCTAATGAA-1.depth.txt wildcards: sample=GGCGCTAATGAA-1 resources: tmpdir=/tmp

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/GTACCCACAGCC-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GTACCCACAGCC-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GTACCCACAGCC-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/GTACCCACAGCC-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.coverage.txt jobid: 9 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/GTACCCACAGCC-1.depth.txt wildcards: sample=GTACCCACAGCC-1 resources: tmpdir=/tmp

[Thu Mar 21 12:11:51 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/TTCCTACGCAAT-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TTCCTACGCAAT-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TTCCTACGCAAT-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/TTCCTACGCAAT-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.coverage.txt jobid: 11 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/TTCCTACGCAAT-1.depth.txt wildcards: sample=TTCCTACGCAAT-1 resources: tmpdir=/tmp

Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. [Thu Mar 21 12:11:52 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/CCCGTCGTGGTA-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCCGTCGTGGTA-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCCGTCGTGGTA-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/CCCGTCGTGGTA-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.coverage.txt jobid: 4 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/CCCGTCGTGGTA-1.depth.txt wildcards: sample=CCCGTCGTGGTA-1 resources: tmpdir=/tmp

Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... [Thu Mar 21 12:11:52 2024] rule process_one_sample: input: /home/nburnaevskiy/maegatk_test/output/.internal/samples/GACCGTGCATTT-1.bam.txt output: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GACCGTGCATTT-1.qc.bam, /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GACCGTGCATTT-1.qc.bam.bai, /home/nburnaevskiy/maegatk_test/output/qc/depth/GACCGTGCATTT-1.depth.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.A.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.C.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.G.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.T.txt, /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.coverage.txt jobid: 6 reason: Missing output files: /home/nburnaevskiy/maegatk_test/output/qc/depth/GACCGTGCATTT-1.depth.txt wildcards: sample=GACCGTGCATTT-1 resources: tmpdir=/tmp

Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Waiting at most 5 seconds for missing files. Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Config file /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Select jobs to execute... Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. Traceback (most recent call last): File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17, in config = yaml.load(stream, Loader=yaml.Loader) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1085, in load error_deprecation('load', 'load', arg=_error_dep_arg, comment=_error_dep_comment) File "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/ruamel/yaml/main.py", line 1039, in error_deprecation raise AttributeError(s, name=None) AttributeError: "load()" has been removed, use

yaml = YAML(typ='rt') yaml.load(...)

and register any classes that you use, or check the tag attribute on the loaded data, instead of file "/net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 17

config = yaml.load(stream, Loader=yaml.Loader)

Waiting at most 5 seconds for missing files. MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGGTTCGCCTCC-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGGTTCGCCTCC-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/GGGTTCGCCTCC-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGGTTCGCCTCC-1.coverage.txt MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TGGTAGTGAACC-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TGGTAGTGAACC-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/TGGTAGTGAACC-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TGGTAGTGAACC-1.coverage.txt Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/ACAATTAGCACT-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/ACAATTAGCACT-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/ACAATTAGCACT-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/ACAATTAGCACT-1.coverage.txt Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCACAAAACATG-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCACAAAACATG-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/CCACAAAACATG-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCACAAAACATG-1.coverage.txt Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/GGGTTCGCCTCC-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGGTTCGCCTCC-1.qc.bam GGGTTCGCCTCC-1 python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/TGGTAGTGAACC-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TGGTAGTGAACC-1.qc.bam TGGTAGTGAACC-1 Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/ACAATTAGCACT-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/ACAATTAGCACT-1.qc.bam ACAATTAGCACT-1 MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGCGCTAATGAA-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGCGCTAATGAA-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/GGCGCTAATGAA-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GGCGCTAATGAA-1.coverage.txt MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GTACCCACAGCC-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GTACCCACAGCC-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/GTACCCACAGCC-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GTACCCACAGCC-1.coverage.txt Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/CCACAAAACATG-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCACAAAACATG-1.qc.bam CCACAAAACATG-1 MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CTAACCCGGAAT-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CTAACCCGGAAT-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/CTAACCCGGAAT-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CTAACCCGGAAT-1.coverage.txt python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/GGCGCTAATGAA-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GGCGCTAATGAA-1.qc.bam GGCGCTAATGAA-1 MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TTCCTACGCAAT-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TTCCTACGCAAT-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/TTCCTACGCAAT-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/TTCCTACGCAAT-1.coverage.txt Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/GTACCCACAGCC-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GTACCCACAGCC-1.qc.bam GTACCCACAGCC-1 MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCCGTCGTGGTA-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCCGTCGTGGTA-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/CCCGTCGTGGTA-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/CCCGTCGTGGTA-1.coverage.txt Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/CTAACCCGGAAT-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CTAACCCGGAAT-1.qc.bam CTAACCCGGAAT-1 MissingOutputException in rule process_one_sample in file /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/snake/Snakefile.maegatk.Scatter, line 21: Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GACCGTGCATTT-1.qc.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GACCGTGCATTT-1.qc.bam.bai /home/nburnaevskiy/maegatk_test/output/qc/depth/GACCGTGCATTT-1.depth.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.A.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.C.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.G.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.T.txt /home/nburnaevskiy/maegatk_test/output/temp/sparse_matrices/GACCGTGCATTT-1.coverage.txt python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/TTCCTACGCAAT-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/TTCCTACGCAAT-1.qc.bam TTCCTACGCAAT-1 Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/GACCGTGCATTT-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/GACCGTGCATTT-1.qc.bam GACCGTGCATTT-1 python /net/module/sw/maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py /home/nburnaevskiy/maegatk_test/output/.internal/parseltongue/snake.scatter.yaml /home/nburnaevskiy/maegatk_test/output/temp/barcoded_bams/CCCGTCGTGGTA-1.bam /home/nburnaevskiy/maegatk_test/output/temp/ready_bam/CCCGTCGTGGTA-1.qc.bam CCCGTCGTGGTA-1 Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-03-21T121150.050858.snakemake.log

[noranekonobokkusu]

Did they replace the yaml command or the python file in the installation folder (like, in ~/.local/lib/python3.9/site-packages/maegatk/)? snakemake_gather failed because it didn't have required input files. Are you sure you are not getting any other error messages at any stage of running it, and that you replaced cli.py entirely and didn't change anything else? and that you didn't have any existing (even empty) output directories like temp_bam and ready_bam before rerunning the command? I don't see a way to reproduce the absence of snakemake_scatter output.

[noranekonobokkusu]

@NBurnaevskiy just saw your new comment, let me take a look.

[NBurnaevskiy]

Ksenia, thank you for all your responses.

[noranekonobokkusu]

I see, so yaml syntax in oneSample_maegatk.py got outdated, too. I will recreate in on my computer (I currently have an older yaml so everything seems to work) and try fixing it.

[NBurnaevskiy]

we are iteratively working through issues. yaml update creates problems in few places. first we updated file oneSample_maegatk.py, to follow new yaml syntax. we updated line 16 and 17 to make this with open(configFile, 'r') as stream: yaml = YAML(typ='rt') config = yaml.load(stream, Loader=yaml.Loader)

Then we updated header to change this from ruamel import yaml into this from ruamel.yaml import YAML

Now we got an error: File ".../maegatk/0.2.0/lib/python3.10/site-packages/maegatk/bin/python/oneSample_maegatk.py", line 18, in config = yaml.load(stream, Loader=yaml.Loader) AttributeError: 'YAML' object has no attribute 'Loader'. Did you mean: 'Reader'?

[noranekonobokkusu]

I am also trying the same - they dropped the load function https://yaml.readthedocs.io/en/latest/, but I am still figuring out what is the correct way to rewrite that

[NBurnaevskiy]

maybe it should be config = yaml.load(stream, Loader=Loader) ?

[noranekonobokkusu]

It seems dropping the argument altogether allows the pipeline to proceed and start generating files in temp_bam/: yaml = YAML(typ='rt') config = yaml.load(stream)

[noranekonobokkusu]

Ok, I can confirm that config = yaml.load(stream) allows the pipeline to produce the final .rds file. @caleblareau can you confirm these changes are enough?

from ruamel.yaml import YAML
...
yaml = YAML(typ='rt')
config = yaml.load(stream)

[NBurnaevskiy]

I can confirm that the script produced final output files. There were some complains in the log, like this samtools: /net/module/sw/maegatk/0.2.0/bin/../lib/libtinfow.so.6: no version information available (required by samtools)

but hopefully they are not critical.

I will now try real data and let you know if it completes. Thank you very much Ksenia. I hope that this tools will useful for us.