Closed boaty closed 2 years ago
Hmm... GT_Pro is designed to handle any short reads that is no longer than 500nt, so it should work with 150nt reads. It looks like you are experiencing a quite similar problem as https://github.com/zjshi/gt-pro/issues/42, which is new to us. Could you please share your input file (SRR413665_2.fastq.gz) if that is realistic. Thanks!
Thanks,
Sure. We got Metagenome fastq file. The raw fastq was processed by QC(KneadData)+host removal(bowtie2 against human genome) and merged by microbiome helper(function concat_paried_end.pl)
After those treatments, we got a filtered and joined fastq, here's some example.
And noticed that some fastq files were randomly successful while running GT-PRO, means GT-PRO worked sometimes but failed at other times. I do not know how it happened.
@A00133:159:H35K7DSXY:4:1101:5032:1000:N:0:ATTATC#0/1
ATAATGCCGTGAAGTTTACCGGGGAAGGCAGCATTGAGTTCGGGTTTGATGTGCGCGAAGACGGTTTCCTGCATTTCTATGTCACCGACACCGGTTGCGGGATACCGGAGGAACGGTTGGAAGAAATCTTCGGGAATTTTGTCAAACTCA
+
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF,F:FFFFFFFFFFFFFFFF:FFFFFFFF:FFFFFFFFFFFFFFFFFFFF
@A00133:159:H35K7DSXY:4:1101:7988:1391:N:0:ATTATC#0/1
AAAAGGATTGGGTAAGGGACTTGATTCATTAATTACAGACAAGGTATCAAAGCCGGTAAAACCAAAGAGCAATCATGCAGCGGATGCTGTAATGATAGATATTAATA
+
FFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFF,FFFFFFFFFFF,FFF
@A00456:422:HFV2TDSXY:2:1101:1208:8124:N:0:GCTTGG#0/1
ATATACTGCTTGAGCCACTAATTGTTTGTAGTCCTTCATCACTAATATCCTCCTTTAAAAATAAAAACGTCTCTTGCATGGCGTTAAATAACACTTTGCAAGAGACGAGAAATTCCCGCGGTACCACTCTATTTGATCAATATCCACTCAAGAGCAACACAGAACAAAAAGAGCGATTTCACACCCGCCGCGCCCATGCGGGCAAATGGCGGGCAGCTGAAGGAAGCGCGGGCGACCGCGGTATCCTGCGAGTGGGATCCGGCGAAATAGAAGTACAAGGAAAGGAG
+
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF::FF:FFFFF,FFFFFFF,,FFFFF:F:F:FFFFF,FFFFFFFFFFFFFFFFFFFFFFFF,F:FFF:FF:FFFFFFFFFFFFFF,F:FFFFFF,F:FF:FFFFFFFFF:FFF,FF::FFFFFFF:FFFF::F:
Thank you boaty! I tried a couple of ways but I couldn't reproduce the error you were seeing with three sequences provided. However, I found two empty lines (Line no. 5 and 10) in your FASTQ file, perhaps that is the reason preventing GT_Pro from behaving properly?
Thanks Shi,
I retried GT-pro with the example file SRR413665_2.fastq.gz from GitHub, I noticed that sometimes GT-pro gave me a error with error code 1067.
/home/Desktop/tools/gt-pro/GT_Pro genotype -d /data/GTpro_database/20190723_881species -C ./test SRR413665_2.fastq.gz -t 10 -f
gt_pro /data/GTpro_database/20190723_881species 10 force_overwrite
1644481038613: [Info] Starting to load DB: /data/GTpro_database/20190723_881species
1644481038613: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_snps.bin
1644481038868: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_kmer_index.bin
1644481039784: [Info] Using -l 32 -m 36 as optimal for system RAM
1644481039784: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_mmer_bloom_36.bin
1644481040451: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_lmer_index_32.bin
1644481042738: [Info] Done with init for optimized DB with 2856121626 kmers. That took 4 seconds.
1644481042742: [Info] Forcing recompute for input: SRR413665_2.fastq.gz
1644481042863: [Info] Waiting for all readers to quiesce
1644481045743: [Done] searching is completed for the 600000 reads input from SRR413665_2.fastq.gz
1644481045786: [Stats] 102624 snps, 600000 reads, 1.28 hits/snp, for SRR413665_2.fastq.gz
1644481045796: [ERROR] Error writing output SRR413665_2__gtpro__20190723_881species.tsv.gz cl 1067
1644481045798: 0.6 million reads were scanned after 3 seconds
*** Failed for ALL 1 input files. ***
1644481045811: Totally done: 3 seconds elapsed processing reads, after DB was loaded.
then I tried again and finally it returned with results
/home/Desktop/tools/gt-pro/GT_Pro genotype -d /data/GTpro_database/20190723_881species -C ./test SRR413665_2.fastq.gz -t 10 -f
gt_pro /data/GTpro_database/20190723_881species 10 force_overwrite
1644481164346: [Info] Starting to load DB: /data/GTpro_database/20190723_881species
1644481164346: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_snps.bin
1644481164600: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_kmer_index.bin
1644481165534: [Info] Using -l 32 -m 36 as optimal for system RAM
1644481165534: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_mmer_bloom_36.bin
1644481166240: [Info] MMAPPING /data/GTpro_database/20190723_881species_optimized_db_lmer_index_32.bin
1644481168554: [Info] Done with init for optimized DB with 2856121626 kmers. That took 4 seconds.
1644481168641: [Info] Waiting for all readers to quiesce
1644481171566: [Done] searching is completed for the 600000 reads input from SRR413665_2.fastq.gz
1644481171616: [Stats] 102624 snps, 600000 reads, 1.28 hits/snp, for SRR413665_2.fastq.gz
1644481171627: 0.6 million reads were scanned after 3 seconds
1644481171627: Successfully processed 1 input files containing 600000 reads.
1644481171642: Totally done: 3 seconds elapsed processing reads, after DB was loaded.
I have no idea why it happened, is it the last one's result usable?
Thanks
Hi boaty, just a quick question: did you happen to run this case on an HPC environment, if not, which OS and version did you use?
Hi shi, I run the program on our remote server, it's a single HPC not cluster. The server has Linux Mint 20 Ulyana installed with max 160 threads and max 1000G mem.
System: Kernel: 5.4.0-65-generic x86_64 bits: 64 compiler: gcc v: 9.3.0 Desktop: Xfce 4.14.2 tk: Gtk 3.24.13 wm: xfwm4 dm: LightDM Distro: Linux Mint 20 Ulyana base: Ubuntu 20.04 focal
Hi boaty, thanks for the information! Error cl 1067 is an uncharacterized I/O failure. This is likely an issue specific to some OS/disk settings or status. If GT-Pro does not complain, the results should clean or reliable. In your case, it is good to go. The problem is related to this issue: https://github.com/zjshi/gt-pro/issues/43. As I am still gathering information to reproduce the error on my side, would you please let me know if you changed any of the parameters when you ran it the second time?
Hi Shi,
Thanks for your response. I did not change any parameters for the second run.
Moreover, I re-run the test file, I found somehow the problem is link to "-C" attribute. if I just gave GT-pro the fastq path directly without "-C" to specify a folder, it returned no error.
Aha, thanks. You helped me a lot, boaty! I think I now understand what the issue is: GT-Pro will by default output to the same directory of input file, so it will raise the error when the input file is in a write restricted location if -C is not specified to redirect the output. So two work-arounds: 1. put input file in the directory where write is allowed and 2. using -C to assign output directory.
GT-pro version: 1.0.1 python version: 3.10.1
Hi,
While running GT-pro optimise with example fastq file SRR413665_2.fastq.gz, program returned those error messages as below.
bash /home/Desktop/tools/gt-pro/GT_Pro optimize --db /data/GTpro_database/20190723_881species --in /home/Desktop/tools/gt-pro/test/SRR413665_2.fastq.gz
It seems to be some issue about read length. My fastq files are merged reads for length of 150nt.
I also tired to run genotype with fastq file for testing, but I noticed that for same fastq file, sometimes GT-pro gave me a error file with those words inside.
sometimes it worked well with genotype results . But when it failed, it shows in stout
En plus, in step-by-step tutorial (https://github.com/zjshi/gt-pro/blob/master/ExampleTutorial.md) and GT-pro optimiser's help, attribute: --out is still included, but the GT-pro.sh does not seem to recognize it.
Thanks