ShirelyI commented 1 year ago

Describe the issue Running Ltr_retriever...Died at /software/RepeatModeler-2.0.4/LTRPipeline line 693.

A concise description of the bug, including any error messages.

Reproduction steps

Steps to reproduce the behavior, including the command lines given to the program

BuildDatabase -name G G.fasta RepeatModeler -database G -threads 16 -LTRStruct >repeatmodeler.log 2>&1

and links to publicly available genome assemblies and other data files (if available).

Log output

LTR Structural Analysis

Running LtrHarvest... : 00:05:55 (hh:mm:ss) Elapsed Time Running Ltr_retriever...Died at /data/data1/zx/software/RepeatModeler-2.0.4/LTRPipeline line 693. LTRPipeline Time: 00:06:04 (hh:mm:ss) Elapsed Time

Please paste or attach any and all log output, which includes useful information including data file statistics and version numbers. An easy way to capture this is to redirect the log output to a file e.g RepeatModeler -database mydb >& output.log. The log output should include the "random seed" value at the start of the run. This number will be necessary in order to reproduce the run exactly.

Environment (please include as much of the following information as you can find out):

How did you install RepeatModeler? e.g. manual installation from repeatmasker.org, bioconda, the Dfam TE Tools container, or as part of another bioinformatics tool? manual installation from repeatmasker.org
Which version of RepeatModeler do you have? The output of RepeatModeler without any options will be a help page with the version of the program displayed at the top. RepeatModeler 2.0.4
Which version of RepeatMasker is this RepeatModeler installation using? Have you installed RepBase RepeatMasker Edition for RepeatMasker, or the full Dfam database? RepeatMasker 4.1.4, RepBase RepeatMasker Edition for RepeatMasker installed
Operating system and version. The output of uname -a and lsb_release -a can be used to find this.

Additional context

Add any other context you have about the problem here. Some possible examples:
- If an older version of RepeatModeler worked before
- If the problem only happens with specific data files

ehs220 commented 1 year ago

Hi.

Ran into the same issue. Any resolution yet?

I checked really quick and the soft link directing to the tmp directory is broken. The reason is that such file is not available in the path specified.

upasana3 commented 1 year ago

I was trying to run repeatmodeler with four fungal genomes. I got the same error for all four genomes. Has anyone figured out the solution to this issue yet?

upasana3 commented 1 year ago

Hi.

Ran into the same issue. Any resolution yet?

I checked really quick and the soft link directing to the tmp directory is broken. The reason is that such file is not available in the path specified.

Were you able to get it running without the error? If yes, can you please share how you fixed it?

benyoung93 commented 1 year ago

Throwing my hat into the ring also with this error. All other steps completed successfully using the conda install of repeatmodeler. I paste below the programs and versions in my conda environment. Happy to provide any more info for this (log files, etc etc) if that is useful.

# packages in environment at /nethome/bdy8/mambaforge/envs/repeatmodeler_env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
biopython                 1.81            py310h1fa729e_0    conda-forge
blast                     2.14.0               h7d5a4b4_1    bioconda
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cairo                     1.16.0            hbbf8b49_1016    conda-forge
cd-hit                    4.8.1                h43eeafb_9    bioconda
cffi                      1.15.1          py310h255011f_3    conda-forge
curl                      7.87.0               h6312ad2_0    conda-forge
drmaa                     0.7.9                   py_1000    conda-forge
entrez-direct             16.2                 he881be0_1    bioconda
expat                     2.5.0                hcb278e6_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
genometools-genometools   1.6.2           py310h3db02ab_6    bioconda
gettext                   0.21.1               h27087fc_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
h5py                      3.8.0           nompi_py310ha66b2ad_101    conda-forge
harfbuzz                  7.3.0                hdb3a94d_0    conda-forge
hdf5                      1.14.0          nompi_h5231ba7_103    conda-forge
hmmer                     3.3.2                hdbdd923_4    bioconda
icu                       72.1                 hcb278e6_0    conda-forge
isa-l                     2.30.0               ha770c72_4    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               hf9c8cef_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libcurl                   7.87.0               h6312ad2_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libglib                   2.76.3               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libidn2                   2.3.4                h166bdaf_0    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ltr_retriever             2.9.4                hdfd78af_0    bioconda
mafft                     7.520                h031d066_2    bioconda
mysql-connector-c         6.1.11            h6eb9d5d_1007    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nettle                    3.8.1                hc379101_1    conda-forge
ninja-nj                  0.98                 h4ac6f70_2    bioconda
nseg                      1.0.1                h031d066_4    bioconda
numpy                     1.24.3          py310ha4c1d20_0    conda-forge
openssl                   1.1.1u               hd590300_0    conda-forge
pango                     1.50.14              heaa33ce_1    conda-forge
pbzip2                    1.1.13                        0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
perl                      5.32.1          2_h7f98852_perl5    conda-forge
perl-archive-tar          2.40            pl5321hdfd78af_0    bioconda
perl-base                 2.23            pl5321hd8ed1ab_0    conda-forge
perl-business-isbn        3.007           pl5321hd8ed1ab_0    conda-forge
perl-business-isbn-data   20210112.006    pl5321hd8ed1ab_0    conda-forge
perl-carp                 1.50            pl5321hd8ed1ab_0    conda-forge
perl-common-sense         3.75            pl5321hd8ed1ab_0    conda-forge
perl-compress-raw-bzip2   2.201           pl5321h166bdaf_0    conda-forge
perl-compress-raw-zlib    2.202           pl5321h166bdaf_0    conda-forge
perl-constant             1.33            pl5321hd8ed1ab_0    conda-forge
perl-devel-size           0.83            pl5321h031d066_4    bioconda
perl-digest-hmac          1.04            pl5321hdfd78af_0    bioconda
perl-digest-md5           2.58            pl5321h166bdaf_0    conda-forge
perl-encode               3.19            pl5321h166bdaf_0    conda-forge
perl-encode-locale        1.05            pl5321hdfd78af_7    bioconda
perl-exporter             5.74            pl5321hd8ed1ab_0    conda-forge
perl-exporter-tiny        1.002002        pl5321hd8ed1ab_0    conda-forge
perl-extutils-makemaker   7.70            pl5321hd8ed1ab_0    conda-forge
perl-file-listing         6.15            pl5321hdfd78af_0    bioconda
perl-file-path            2.18            pl5321hd8ed1ab_0    conda-forge
perl-file-spec            3.48_01         pl5321hdfd78af_2    bioconda
perl-file-temp            0.2304          pl5321hd8ed1ab_0    conda-forge
perl-file-which           1.24            pl5321hd8ed1ab_0    conda-forge
perl-html-parser          3.81            pl5321h4ac6f70_1    bioconda
perl-html-tagset          3.20            pl5321hdfd78af_4    bioconda
perl-http-cookies         6.10            pl5321hdfd78af_0    bioconda
perl-http-daemon          6.16            pl5321hdfd78af_0    bioconda
perl-http-date            6.05            pl5321hdfd78af_0    bioconda
perl-http-message         6.36            pl5321hdfd78af_0    bioconda
perl-http-negotiate       6.01            pl5321hdfd78af_4    bioconda
perl-io-compress          2.201           pl5321hdbdd923_2    bioconda
perl-io-html              1.004           pl5321hdfd78af_0    bioconda
perl-io-socket-ssl        2.075           pl5321hd8ed1ab_0    conda-forge
perl-io-zlib              1.14            pl5321hdfd78af_0    bioconda
perl-json                 4.10            pl5321hdfd78af_0    bioconda
perl-json-xs              2.34            pl5321h4ac6f70_6    bioconda
perl-libwww-perl          6.67            pl5321hdfd78af_0    bioconda
perl-list-moreutils       0.430           pl5321hdfd78af_0    bioconda
perl-list-moreutils-xs    0.430           pl5321h031d066_2    bioconda
perl-lwp-mediatypes       6.04            pl5321hdfd78af_1    bioconda
perl-mime-base64          3.16            pl5321h166bdaf_0    conda-forge
perl-net-http             6.22            pl5321hdfd78af_0    bioconda
perl-net-ssleay           1.92            pl5321haa6b8db_1    conda-forge
perl-ntlm                 1.09            pl5321hdfd78af_5    bioconda
perl-parent               0.241           pl5321hd8ed1ab_0    conda-forge
perl-pathtools            3.75            pl5321h166bdaf_0    conda-forge
perl-scalar-list-utils    1.63            pl5321h166bdaf_0    conda-forge
perl-socket               2.027           pl5321h031d066_4    bioconda
perl-storable             3.15            pl5321h166bdaf_0    conda-forge
perl-test-fatal           0.016           pl5321ha770c72_0    conda-forge
perl-test-simple          1.302191        pl5321hd8ed1ab_0    conda-forge
perl-test-warnings        0.031           pl5321ha770c72_0    conda-forge
perl-text-soundex         3.05            pl5321hd8ed1ab_1001    conda-forge
perl-time-local           1.35            pl5321hdfd78af_0    bioconda
perl-timedate             2.33            pl5321hdfd78af_2    bioconda
perl-try-tiny             0.31            pl5321ha770c72_0    conda-forge
perl-types-serialiser     1.01            pl5321hdfd78af_0    bioconda
perl-uri                  5.17            pl5321ha770c72_0    conda-forge
perl-url-encode           0.03            pl5321h9ee0642_0    bioconda
perl-www-robotrules       6.02            pl5321hdfd78af_4    bioconda
pigz                      2.6                  h27826a3_0    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pp                        1.6.4.4                    py_0    bioconda
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
python                    3.10.8          h257c98d_0_cpython    conda-forge
python-isal               1.1.0           py310h2372a71_2    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.2                  h8228510_1    conda-forge
recon                     1.08                 h031d066_6    bioconda
repeatmasker              4.1.5           pl5321hdfd78af_0    bioconda
repeatmodeler             2.0.4           pl5321hdfd78af_0    bioconda
repeatscout               1.0.6                hec16e2b_3    bioconda
rmblast                   2.14.0               h21a3994_1    bioconda
setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
tesorter                  1.4.6              pyhdfd78af_0    bioconda
tk                        8.6.12               h27826a3_0    conda-forge
trf                       4.09.1               h031d066_4    bioconda
tzdata                    2023c                h71feb2d_0    conda-forge
ucsc-fatotwobit           447                  h954228d_0    bioconda
ucsc-twobitinfo           447                  h954228d_0    bioconda
ucsc-twobittofa           447                  h954228d_0    bioconda
wget                      1.20.3               ha56f1ee_1    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xopen                     1.7.0           py310hff52083_2    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.5                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstandard                 0.19.0          py310hdeb6495_1    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge

benyoung93 commented 1 year ago

@ehs220 @ShirelyI been working on identyfying the problem and I think I have it.

It seems that if you go to the directory for the LTR with the RM_xxxx run file the symlink for the seq.fa is broken.

My path with the broken seq.fa /scratch/projects/omics/ofav_genome/repeatmodeler/RM_317.WedJun140937302023/LTR_31461.ThuJun152153012023/LRET_31461.ThuJun152305422023

What I have done so far is rm -r seq.fa form this directory, and recreated the sym link ln -s /scratch/projects/omics/ofav_genome/ntlink_5r_s55/ofav_s55_prm.fa.k32.w100.z1000.ntLink.5rounds.fa seq.fa. This has made it a nice blue instead of the angry red it was (indicating broken).

What I am now running is in the same directory the command within the LTR_retriever.log, this is as follows

repeatmodeler -repeatmasker /nethome/bdy8/mambaforge/envs/repeatmodeler_env/share/RepeatMasker -blastplus /nethome/bdy8/mambaforge/envs/repeatmodeler_env/bin -cdhit_path /nethome/bdy8/mambaforge/envs/repeatmodeler_env/bin -trf_path /nethome/bdy8/mambaforge/envs/repeatmodeler_env/bin/trf -genome seq.fa -inharvest /scratch/projects/omics/ofav_genome/repeatmodeler/RM_317.WedJun140937302023/LTR_31461.ThuJun152153012023/raw-struct-results.txt -noanno -threads 20

This is creating files succesfully this time, so it looks like it is working.

@rmhubley I think the problem is within the LTRPipeline is the following lines (line 602 and 603). The symlink is failing. My perl is beyond atrocious so I have 0 idea how to fix this.

  # Create a symlink inside the temp directory called "seq.fa" pointing
  # to the sequence file ( using full path ).  LTR_retriever creates
  # a prodigous amount of temporary files and output files in the same
  # directory as the sequence file.  This ensures they all get created
  # inside the temporary directory.
  my $full_seqFile = File::Spec->rel2abs( $seqFile );
  symlink( $full_seqFile, "$tmpDir/seq.fa" );

Happy to provide any additional information if needed, and I can update if fixing the symlink actually works.

Ben

rmhubley commented 1 year ago

@benyoung93 - thanks for the suggestion. I am not able to reproduce this problem, so it's hard to know if this is the problem or why it would be so. If someone can reproduce this problem (especially with a small input file -- say a 10MB portion of an assembly), then we could try to identify the issue. Another option is to try changing that line in the code from:

symlink( $full_seqFile, "$tmpDir/seq.fa");

to:

system("cp $full_seqFile $tmpDir/seq.fa");

and see if copying rather than linking fixes it for them.

minhasbushra commented 1 year ago

Hi, I just came across the same issue. I had never had this problem before, even with the latest version. I had to reinstall a new copy of the repeat modeler, and that's when the ltr pipeline failed. I tried it with a small sample of my genome and had the same error. Here is the genome sample if that would help.

sample.fa.zip

minhasbushra commented 1 year ago

@benyoung93 - thanks for the suggestion. I am not able to reproduce this problem, so it's hard to know if this is the problem or why it would be so. If someone can reproduce this problem (especially with a small input file -- say a 10MB portion of an assembly), then we could try to identify the issue. Another option is to try changing that line in the code from:
symlink( $full_seqFile, "$tmpDir/seq.fa");
to:
system("cp $full_seqFile $tmpDir/seq.fa");
and see if copying rather than linking fixes it for them.

Tried this, did not worked. If that helps, the pipeline was working fine and suddenly stopped working

benyoung93 commented 1 year ago

HI all

Sorry, I have been in the lab doing lab work.

I can try and reproduce this problem in the next few weeks if you are still interested in finding a bug fix.

Ben

poldohko commented 1 year ago

Hello everyone I share the output of the analysis done with the fasta provided by @minhasbushra, previously I made the change suggested by @benyoung93 (system instead of symlink) and added a "print" after the "die" where the error occurs, I hope it helps. I honestly don't know if this error causes the final results to be different. sample.txt

chhhhai commented 8 months ago

@benyoung93 - thanks for the suggestion. I am not able to reproduce this problem, so it's hard to know if this is the problem or why it would be so. If someone can reproduce this problem (especially with a small input file -- say a 10MB portion of an assembly), then we could try to identify the issue. Another option is to try changing that line in the code from:
symlink( $full_seqFile, "$tmpDir/seq.fa");
to:
system("cp $full_seqFile $tmpDir/seq.fa");
and see if copying rather than linking fixes it for them.
Tried this, did not worked. If that helps, the pipeline was working fine and suddenly stopped working

@minhasbushra Finally, did the error has been solved? if solved, could give me some advice about it? thanks

Dfam-consortium / RepeatModeler

Running Ltr_retriever...Died at /software/RepeatModeler-2.0.4/LTRPipeline line 693. #201

LTR Structural Analysis