Closed angwg closed 7 years ago
Dear George,
Something is still wrong in recognizing the sequence IDs. Could you please send me several lines in the file "c.harvest.scn" and several lines in "c.fa.mod.ltrTE.stg1"? For the second file, you may need to rerun LTR_retriever with the addition of the -v parameter.
Also, listing all file information will help me to find out which step is wrong. Simply paste the stdout of the following command will work:
ls -l c.fa*
Thanks, Shujun
Hi Shujun, here's the file information and truncated versions of those files. I hope this helps!
File information:
-rwxrwxrwx 1 root root 599447795 Jun 6 11:40 c.fa
-rwxrwxrwx 1 root root 863394 Jun 6 11:40 c.fa.des
-rwxrwxrwx 1 root root 148163792 Jun 6 11:40 c.fa.esq
-rwxrwxrwx 1 root root 592654030 Jun 6 11:40 c.fa.lcp
-rwxrwxrwx 1 root root 369982096 Jun 6 11:40 c.fa.llv
-rwxrwxrwx 1 root root 258291 Jun 6 11:40 c.fa.md5
-rw-rw-r-- 1 mcampbel mcampbel 598678341 Jun 6 15:45 c.fa.mod
-rw-rw-r-- 1 mcampbel mcampbel 1318462 Jun 6 16:01 c.fa.mod.defalse
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.ltrTE
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.ltrTE.clust
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.ltrTE.clust.clstr
-rw-rw-r-- 1 mcampbel mcampbel 36584656 Jun 6 15:45 c.fa.mod.ltrTE.fa
-rw-rw-r-- 1 mcampbel mcampbel 61962 Jun 6 15:49 c.fa.mod.ltrTE.fa.cleanup
-rw-rw-r-- 1 mcampbel mcampbel 4349201 Jun 6 16:02 c.fa.mod.ltrTE.mask.lib
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.ltrTE.nmtf
-rw-rw-r-- 1 mcampbel mcampbel 8513337 Jun 6 16:01 c.fa.mod.ltrTE.pass
-rw-rw-r-- 1 mcampbel mcampbel 236799 Jun 6 16:02 c.fa.mod.ltrTE.pass.clust.clstr
-rw-rw-r-- 1 mcampbel mcampbel 169060 Jun 6 16:01 c.fa.mod.ltrTE.pass.list
-rw-rw-r-- 1 mcampbel mcampbel 10545 Jun 6 16:12 c.fa.mod.ltrTE.pass.nmtf.list
-rw-rw-r-- 1 mcampbel mcampbel 31292503 Jun 6 15:49 c.fa.mod.ltrTE.stg1
-rw-rw-r-- 1 mcampbel mcampbel 3469593 Jun 6 16:02 c.fa.mod.ltrTE.stg2
-rw-rw-r-- 1 mcampbel mcampbel 3668719 Jun 6 16:06 c.fa.mod.ltrTE.stg3.cln
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.ltrTE.stg3.cln.clean
-rw-rw-r-- 1 mcampbel mcampbel 47 Jun 6 16:12 c.fa.mod.ltrTE.stg3.cln.clean.exclude.list
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.ltrTE.stg3.cln.exclude.list
-rw-rw-r-- 1 mcampbel mcampbel 382770 Jun 6 16:12 c.fa.mod.ltrTE.stg3.dna.out
-rw-rw-r-- 1 mcampbel mcampbel 431120 Jun 6 16:09 c.fa.mod.ltrTE.stg3.line.out
-rw-rw-r-- 1 mcampbel mcampbel 813890 Jun 6 16:12 c.fa.mod.ltrTE.stg3.otherTE.out
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.ltrTE.stg3.plantP.out
-rw-rw-r-- 1 mcampbel mcampbel 4091070 Jun 6 16:02 c.fa.mod.ltrTE.trunc
-rw-rw-r-- 1 mcampbel mcampbel 21259403 Jun 6 16:06 c.fa.mod.ltrTE.trunc.cat
-rw-rw-r-- 1 mcampbel mcampbel 199126 Jun 6 16:06 c.fa.mod.ltrTE.trunc.cln
-rw-rw-r-- 1 mcampbel mcampbel 36070 Jun 6 16:02 c.fa.mod.ltrTE.trunc.list
-rw-rw-r-- 1 mcampbel mcampbel 4171210 Jun 6 16:06 c.fa.mod.ltrTE.trunc.masked
-rw-rw-r-- 1 mcampbel mcampbel 68116 Jun 6 16:06 c.fa.mod.ltrTE.trunc.masked.cleanup
-rw-rw-r-- 1 mcampbel mcampbel 443306 Jun 6 16:06 c.fa.mod.ltrTE.trunc.ori.out
-rw-rw-r-- 1 mcampbel mcampbel 487052 Jun 6 16:06 c.fa.mod.ltrTE.trunc.out
-rw-rw-r-- 1 mcampbel mcampbel 1990 Jun 6 16:06 c.fa.mod.ltrTE.trunc.tbl
-rw-rw-r-- 1 mcampbel mcampbel 4290 Jun 6 16:02 c.fa.mod.ltrTE.veryfalse
-rw-rw-r-- 1 mcampbel mcampbel 879608 Jun 6 16:02 c.fa.mod.ltrTE.veryfalse.fa
-rw-rw-r-- 1 mcampbel mcampbel 10570 Jun 6 16:02 c.fa.mod.ltrTE.veryfalse.list
-rw-rw-r-- 1 mcampbel mcampbel 10545 Jun 6 16:12 c.fa.mod.nmtf.pass.list
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.nmtf.prelib
-rw-rw-r-- 1 mcampbel mcampbel 169060 Jun 6 16:12 c.fa.mod.pass.list
-rw-rw-r-- 1 mcampbel mcampbel 0 Jun 6 16:12 c.fa.mod.prelib
-rw-rw-r-- 1 mcampbel mcampbel 773693 Jun 6 16:12 c.fa.mod.retriever.all.scn.adj
-rw-rw-r-- 1 mcampbel mcampbel 14056 Jun 6 16:12 c.fa.mod.retriever.all.scn.adj.list
-rw-rw-r-- 1 mcampbel mcampbel 398533 Jun 6 15:45 c.fa.mod.retriever.scn
-rw-rw-r-- 1 mcampbel mcampbel 773693 Jun 6 16:01 c.fa.mod.retriever.scn.adj
-rw-rw-r-- 1 mcampbel mcampbel 77124 Jun 6 16:02 c.fa.mod.retriever.scn.adj.list
-rw-rw-r-- 1 mcampbel mcampbel 235826 Jun 6 15:49 c.fa.mod.retriever.scn.extend
-rw-rw-r-- 1 mcampbel mcampbel 31753991 Jun 6 15:49 c.fa.mod.retriever.scn.extend.fa
-rw-rw-r-- 1 mcampbel mcampbel 64637030 Jun 6 15:50 c.fa.mod.retriever.scn.extend.fa.aa
-rw-rw-r-- 1 mcampbel mcampbel 370434 Jun 6 15:51 c.fa.mod.retriever.scn.extend.fa.aa.anno
-rw-rw-r-- 1 mcampbel mcampbel 6767136 Jun 6 15:51 c.fa.mod.retriever.scn.extend.fa.aa.scn
-rw-rw-r-- 1 mcampbel mcampbel 2534574 Jun 6 15:51 c.fa.mod.retriever.scn.extend.fa.aa.tbl
-rw-rw-r-- 1 mcampbel mcampbel 282640 Jun 6 15:45 c.fa.mod.retriever.scn.full
-rw-rw-r-- 1 mcampbel mcampbel 577980 Jun 6 15:45 c.fa.mod.retriever.scn.list
-rwxrwxrwx 1 root root 482 Jun 6 11:40 c.fa.prj
-rwxrwxrwx 1 root root 62608 Jun 6 11:40 c.fa.sds
-rwxrwxrwx 1 root root 31312 Jun 6 11:40 c.fa.ssp
-rwxrwxrwx 1 root root 4741232240 Jun 6 11:40 c.fa.suf
c.harvest.scn and c.fa.mod.ltrTE.stg1
Dear George,
Thank you for attaching the files along with the file information. From my end, the file format looks OK. The core program identified 1400 intact LTRs but also complaining missing some candidate sequences. There are two possible reasons for this error:
From 1400 intact LTRs to no library produced, this is the other bug when no LINE and DNA TE coding sequence was detected. I have fixed this one and pushed it to GitHub. Please try the new code.
Best, Shujun
Hi Shujun, I downloaded the latest release and I'm getting a similar error. In a separate run, I also ran a script to trim the sequence ids below 15 characters in the genome c.fa before running LTR_harvest on it, but that did not seem to change anything. I also checked the cpu usage on my machine and there are enough cpus available to run on the default settings (4 threads).
Here's the shell output.
##########################
### LTR_retriever v1.2 ###
##########################
Contributors: Shujun Ou, Ning Jiang
Parameters: -genome c.fa -inharvest c.harvest.scn
Wed Jun 7 14:38:45 EDT 2017 The longest sequence ID in the genome contains 117 characters, which is longer than the limit (15)
Trying to reformat seq IDs...
Attempt 1...
Wed Jun 7 14:38:49 EDT 2017 Seq ID conversion successful!
Wed Jun 7 14:38:49 EDT 2017 Start to convert inputs...
Total candidates: 5597
Total uniq candidates: 5597
Wed Jun 7 14:38:53 EDT 2017 Start to clean up candidates...
Sequences with 10 missing bp or 0.8 missing data rate will be discarded.
Sequences containing tandem repeats will be discarded.
Wed Jun 7 14:43:03 EDT 2017 4654 clean candidates remained
Wed Jun 7 14:43:03 EDT 2017 Start to analyze the structure of candidates...
The terminal motif, TSD, boundary, orientation, age, and family will be identified in this step.
BLAST engine error: NCBI C++ Exception:
"/home/coremake/release_build/build/PrepareRelease_Linux64-Centos_JSID_01_491_130.14.22.10_9052_1363121432/c++/src/algo/blast/api/blast_setup.hpp", line 189: Error: Sequence contains no data
BLAST engine error: Warning: Sequence contains no data
Wed Jun 7 14:54:00 EDT 2017 Intact LTR found: 1399
Wed Jun 7 14:54:46 EDT 2017 Start to analyze truncated LTRs...
Truncated LTRs without the intact version will be retained in the LTR library.
Use -notrunc if you don't want to keep them.
Wed Jun 7 14:54:46 EDT 2017 711 truncated LTRs found
Wed Jun 7 14:58:39 EDT 2017 126 truncated LTR sequences have added to the library
Wed Jun 7 14:58:39 EDT 2017 Start to remove DNA TE and LINE transposases, and remove plant protein sequences...
Total library sequences: 1093
sort: multi-character tab ‘$\t’
ERROR: LOC list is empty.
Wed Jun 7 15:04:31 EDT 2017 Retained clean sequence: 0
ERROR: 1399 intact LTRs have found, but the pre-library file c.fa.mod.ltrTE is empty.
Something is wrong at this point. Please report the bug to https://github.com/oushujun/LTR_retriever/issues
Program halt!
shell output for pretrimmed genome
##########################
### LTR_retriever v1.2 ###
##########################
Contributors: Shujun Ou, Ning Jiang
Parameters: -genome ctrim.fa -inharvest c.harvest.scn
Wed Jun 7 15:12:22 EDT 2017 Start to convert inputs...
Total candidates: 5597
Total uniq candidates: 5597
Wed Jun 7 15:12:27 EDT 2017 Start to clean up candidates...
Sequences with 10 missing bp or 0.8 missing data rate will be discarded.
Sequences containing tandem repeats will be discarded.
Wed Jun 7 15:16:38 EDT 2017 4654 clean candidates remained
Wed Jun 7 15:16:38 EDT 2017 Start to analyze the structure of candidates...
The terminal motif, TSD, boundary, orientation, age, and family will be identified in this step.
BLAST engine error: NCBI C++ Exception:
"/home/coremake/release_build/build/PrepareRelease_Linux64-Centos_JSID_01_491_130.14.22.10_9052_1363121432/c++/src/algo/blast/api/blast_setup.hpp", line 189: Error: Sequence contains no data
BLAST engine error: NCBI C++ Exception:
"/home/coremake/release_build/build/PrepareRelease_Linux64-Centos_JSID_01_491_130.14.22.10_9052_1363121432/c++/src/algo/blast/api/blast_setup.hpp", line 189: Error: Sequence contains no data
BLAST engine error: Warning: Sequence contains no data
Wed Jun 7 15:28:24 EDT 2017 Intact LTR found: 1401
Wed Jun 7 15:29:09 EDT 2017 Start to analyze truncated LTRs...
Truncated LTRs without the intact version will be retained in the LTR library.
Use -notrunc if you don't want to keep them.
Wed Jun 7 15:29:10 EDT 2017 710 truncated LTRs found
Wed Jun 7 15:33:00 EDT 2017 125 truncated LTR sequences have added to the library
Wed Jun 7 15:33:00 EDT 2017 Start to remove DNA TE and LINE transposases, and remove plant protein sequences...
Total library sequences: 1094
sort: multi-character tab ‘$\t’
ERROR: LOC list is empty.
Wed Jun 7 15:38:56 EDT 2017 Retained clean sequence: 0
ERROR: 1401 intact LTRs have found, but the pre-library file ctrim.fa.ltrTE is empty.
Something is wrong at this point. Please report the bug to https://github.com/oushujun/LTR_retriever/issues
Program halt!
Dear George,
I am sorry for the troubles you are experiencing. Could you please send me the file "c.fa.mod.retriever.scn.adj" and "c.fa.mod.defalse" so that I can further determine what was wrong? You can either attach them in this thread or send to my email. Thank you!
Best, Shujun
I've been trying to run LTR_retriever on my data with the following command:
LTR_retriever -genome c.fa -inharvest c.harvest.scn
I'm using a freshly cloned version of LTR_retriever (as of 6/4/2017), and ensured that all files and folders have read/write permissions.While the program runs to completion, several errors show up and no LTR's are found.
The following errors occur multiple times, in an inconsistent manner across separate runs. Running the program multiple times results in these errors popping up a different number of times in each run.
As for the program not finding any LTRs in the dataset, that's strange since the data is a genome wherein a significant portion is composed of transposons. I'm not sure if this is because something went wrong or if it's because there really wasn't anything there. I've also run LTR_retriever with additional inputs but the result is the same. Are there any sample data sets to confirm if the program is working properly?
Here's the entire output that the program gives: