Open AnnaSOFI opened 6 years ago
Would it be possible to share the FASTA?
Hi,
The FASTA file is too big to post here, but it has been downloaded from here:
Edit. I could also send the file if this is more convenient?
Anna
Hi, I'm having a very similar problem trying to run bowtie2-build on two different FASTAs (one from metaphlan2's marker database and another that was built from chocophlan). Was there ever a solution found to this? I'm not actually getting an error message, but I think its stopping before it actually finishing the index (see below for the output). This is on version 2.3.4.1, I used bioconda to install.
INFO: Command: bowtie2-build-s --wrapper basic-0 -f Humann2Test/A1HF_humann2_temp/A1HF_custom_chocophlan_database.ffn Humann2Test/A1HF_humann2_temp/A1HF_bowtie2_index
Settings:
Output files: "Humann2Test/A1HF_humann2_temp/A1HF_bowtie2_index.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
Humann2Test/A1HF_humann2_temp/A1HF_custom_chocophlan_database.ffn
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:00
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:00
bmax according to bmaxDivN setting: 6990730
Using parameters --bmax 5243048 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 5243048 --dcv 1024
Constructing suffix-array element generator
If a solution or workaround was found, I'd be very interested to know. Thanks for your help, Chris
Hi,
I did not get official advice on this one, but now it is working. I split my enormous FASTA file into two (8.4GB --> 4.2GB) just yesterday and the indexing seemed now to be working for the first part at least (I did not yet run the alignment but at least the .rev files have appeared after indexing ins complete)! Could you try this approach also? ☺ I think I’m going to align my data against both of the FASTA files and then combine the alignment sam files.
BRs,
Anna Sorjamaa
Anna Sorjamaa, PhD student Research Group of Docent Justus Reunanen Biocenter Oulu / Cancer Research and Translational Medicine Research Unit University of Oulu Aapistie 5, P.O. box 5281 90014 University of Oulu, Finland Tel: +35845 879 0354 Email: anna.sorjamaa@oulu.fimailto:anna.sorjamaa@oulu.fi
From: whidbeyc notifications@github.com Sent: Tuesday, July 3, 2018 7:49 AM To: BenLangmead/bowtie2 bowtie2@noreply.github.com Cc: Anna Sorjamaa anna.sorjamaa@oulu.fi; Author author@noreply.github.com Subject: Re: [BenLangmead/bowtie2] bowtie2-build runs without error but without constructing .rev files (#194)
Hi, I'm having a very similar problem trying to run bowtie2-build on two different FASTAs (one from metaphlan2's marker database and another that was built from chocophlan). Was there ever a solution found to this? I'm not actually getting an error message, but I think its stopping before it actually finishing the index (see below for the output). This is on version 2.3.4.1, I used bioconda to install.
INFO: Command: bowtie2-build-s --wrapper basic-0 -f Humann2Test/A1HF_humann2_temp/A1HF_custom_chocophlan_database.ffn Humann2Test/A1HF_humann2_temp/A1HF_bowtie2_index
Settings:
Output files: "Humann2Test/A1HF_humann2_temp/A1HF_bowtie2_index.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
Humann2Test/A1HF_humann2_temp/A1HF_custom_chocophlan_database.ffn
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:00
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:00
bmax according to bmaxDivN setting: 6990730
Using parameters --bmax 5243048 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 5243048 --dcv 1024
Constructing suffix-array element generator
If a solution or workaround was found, I'd be very interested to know. Thanks for your help, Chris
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/BenLangmead/bowtie2/issues/194#issuecomment-402012159, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AmuBYw-mctGb1StfhKlzjICmUM4hFcOqks5uCvetgaJpZM4U3y1H.
Hi, I'm having exactly this problem as well - I installed bowtie2 automatically as part of humann2 (version 2.3.4.1), and bowtie2-build will not build .rev.bt2 files from any fasta file I give it, including as part of the MetaPhlAn pipeline indexing the database, but doesn't throw any errors either. It also produces a .2.bt2 file of 0kb, and (for the failed MetaPhlAn database index, where I have a comparison) the other files are smaller than the correctly-indexed database files provided by the makers of that software.
Really keen to figure out if this is a bowtie problem, a server problem, a data problem, or something else!
This is the output I get when I try:
Settings: Output files: "04_MAPPING/contigs.*.bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: 03_CONTIGS/contigs.fa Building a SMALL index Reading reference sizes Time reading reference sizes: 00:00:01 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:01 bmax according to bmaxDivN setting: 17568529 Using parameters --bmax 13176397 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 13176397 --dcv 1024 Constructing suffix-array element generator
I have already begun looking into this. I will update this thread as soon as I have more information.
Quick update: I installed versions 2.2.4 and 2.3.0 in conda environments and tried indexing the same file. Version 2.2.4 created all 6 output files without a problem; version 2.3.0 stopped at the same point as 2.3.4.1 and created the same 4 too-small files, but gave an error Segmentation fault (core dumped)
when it did.
@whidbeyc -- can you share your command line? How big is your index? @AnnaSOFI -- I am not able to access the files on the website you provided. Would it be possible to share the link some other way? Maybe a dropbox link?
Same problem here.
I install bowtie2 2.3.4.1 with conda.
Build the index of a fungi genome with no errors.
The fungi genome is ~10^7 bp.
The command I used:
bowtie2-build -f fungi.fa fungi_bt2
bowtie2 --local -x fungi_bt2 -f -U contigs.fasta -S bowtie.contigs.fungi.sam -p 1
The error I got:
Could not open index file fungi_bt2.rev.1.bt2
Could not open index file fungi_bt2.rev.2.bt2
Segmentation fault (core dumped)
(ERR): bowtie2-align exited with value 139
Same here!
I'm trying to build an index from a fasta file that contains ~360,000 contigs (the file size is 194Mb). I'm using bowtie2 version 2.3.4.1 installed from conda (conda install --yes -c bioconda bowtie2=2.3.4.1
) with the following command: bowtie2-build final.contigs.fa contig_index
.
The program runs with no errors and finishes quickly but generates only four files (contig_index.1.bt2 [5.5Mb], contig_index.2.bt2 [0Mb], contig_index.3.bt2 [3.1Mb], and contig_index.4.bt2 [45Mb]), but no .rev1.bt2 or .rev2.bt2 files.
Here is the output I get when I try to build the indices:
Settings:
Output files: "contig_index.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
/home/ubuntu/output_169_subset/final.contigs.fa
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:02
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:01
bmax according to bmaxDivN setting: 46852187
Using parameters --bmax 35139141 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 35139141 --dcv 1024
Constructing suffix-array element generator
Any advice? Thanks, Simon
Quick update for people who need a quick fix, installing bowtie2 from the other conda link works (at least the bowtie2-build
), although it's version 2.2.6.
Here is the link: conda install -c bioconda/label/broken bowtie2
I probably should have done this before jumping into debugging code, but I just tried building an index with a conda
--installed bowtie2
build and got the same output. This issue seems to be isolated to the conda
build of bowtie2
as evidenced below:
Conda:
root@a50a2aa0a1ff:/bowtie2# bowtie2-build example/reference/lambda_virus.fa out
Settings:
Output files: "out.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
example/reference/lambda_virus.fa
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:00
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:00
bmax according to bmaxDivN setting: 12125
Using parameters --bmax 9094 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 9094 --dcv 1024
Constructing suffix-array element generator
root@a50a2aa0a1ff:/bowtie2# ls out*
out.1.bt2 out.2.bt2 out.3.bt2 out.4.bt2
Local build:
root@a50a2aa0a1ff:/bowtie2# ./bowtie2-build example/reference/lambda_virus.fa out
Settings:
Output files: "out.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
example/reference/lambda_virus.fa
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:00
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:00
bmax according to bmaxDivN setting: 12125
Using parameters --bmax 9094 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 9094 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
Building sPrime
Building sPrimeOrder
V-Sorting samples
V-Sorting samples time: 00:00:00
Allocating rank array
Ranking v-sort output
Ranking v-sort output time: 00:00:00
Invoking Larsson-Sadakane on ranks
Invoking Larsson-Sadakane on ranks time: 00:00:00
Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
(Using difference cover)
Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
Splitting and merging time: 00:00:00
Avg bucket size: 48502 (target: 9093)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
No samples; assembling all-inclusive block
Sorting block of length 48502 for bucket 1
(Using difference cover)
Sorting block time: 00:00:00
Returning block of 48503 for bucket 1
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 12334
fchr[G]: 23696
fchr[T]: 36516
fchr[$]: 48502
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 4210730 bytes to primary EBWT file: out.1.bt2
Wrote 12132 bytes to secondary EBWT file: out.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
len: 48502
bwtLen: 48503
sz: 12126
bwtSz: 12126
lineRate: 6
offRate: 4
offMask: 0xfffffff0
ftabChars: 10
eftabLen: 20
eftabSz: 80
ftabLen: 1048577
ftabSz: 4194308
offsLen: 3032
offsSz: 12128
lineSz: 64
sideSz: 64
sideBwtSz: 48
sideBwtLen: 192
numSides: 253
numLines: 253
ebwtTotLen: 16192
ebwtTotSz: 16192
color: 0
reverse: 0
Total time for call to driver() for forward index: 00:00:00
Reading reference sizes
Time reading reference sizes: 00:00:00
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:00
Time to reverse reference sequence: 00:00:00
bmax according to bmaxDivN setting: 12125
Using parameters --bmax 9094 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 9094 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
Building sPrime
Building sPrimeOrder
V-Sorting samples
V-Sorting samples time: 00:00:00
Allocating rank array
Ranking v-sort output
Ranking v-sort output time: 00:00:00
Invoking Larsson-Sadakane on ranks
Invoking Larsson-Sadakane on ranks time: 00:00:00
Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
(Using difference cover)
Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
Splitting and merging time: 00:00:00
Avg bucket size: 48502 (target: 9093)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
No samples; assembling all-inclusive block
Sorting block of length 48502 for bucket 1
(Using difference cover)
Sorting block time: 00:00:00
Returning block of 48503 for bucket 1
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 12334
fchr[G]: 23696
fchr[T]: 36516
fchr[$]: 48502
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 4210730 bytes to primary EBWT file: out.rev.1.bt2
Wrote 12132 bytes to secondary EBWT file: out.rev.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
len: 48502
bwtLen: 48503
sz: 12126
bwtSz: 12126
lineRate: 6
offRate: 4
offMask: 0xfffffff0
ftabChars: 10
eftabLen: 20
eftabSz: 80
ftabLen: 1048577
ftabSz: 4194308
offsLen: 3032
offsSz: 12128
lineSz: 64
sideSz: 64
sideBwtSz: 48
sideBwtLen: 192
numSides: 253
numLines: 253
ebwtTotLen: 16192
ebwtTotSz: 16192
color: 0
reverse: 1
Total time for backward call to driver() for mirror index: 00:00:00
root@a50a2aa0a1ff:/bowtie2# ls out*
out.1.bt2 out.2.bt2 out.3.bt2 out.4.bt2 out.rev.1.bt2 out.rev.2.bt2 outq.cpp outq.h
We do provide pre-built bowtie2
packages. Any objections to trying those?
I’m using bowtie on an instance on AWS and I’ve had problems with pre-built software before. Any chance of fixing the conda built or should I get the source code and try it to build it myself?
Thanks for looking into this!
— <Sent from a mobile device, please excuse any typo>
On Jul 19, 2018, at 18:48, ch4rr0 notifications@github.com wrote:
We do provide pre-built bowtie2 packages. Any objections to trying those?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
I'll see what I can do with regards to the conda build tomorrow. I will keep you posted.
Yeah, my IT department won't grant permissions for installing software myself on our bioinformatics server except through conda or pip, so an updated/working conda install would be very handy.
I put together a script that works within the conda framework to replace the problematic bowtie2
binaries with ones that work. Here's what it does:
conda-build
to build bowtie2
locally (with tbb
and libz
statically linked) then creates a local conda
packagebowtie2
runtime and uses the conda install
command to replace it with the local buildYou can find out more about this process here.
Once the package has been build, conda install
will output the following
The following NEW packages will be INSTALLED:
bowtie2: 2.3.4.1-py27h6bb024c_1 local
Proceed ([y]/n)? y
Make sure that the package is to be installed is marked local
and not bioconda
as seen above
Copy the below script verbatim and save to file e.g. bt2_conda.sh
. From the command line execute the script by issuing the command: bash bt2_conda.sh
. Let me know if you encounter any issues.
#!/bin/bash
if [[ ! -e `which conda-build` ]]; then
conda install conda-build
fi
if [[ ! -e `which wget` && ! -e `which curl` ]]; then
echo "Please make sure that either curl or wget is installed"
exit 1
fi
function cleanup {
files_to_remove=`ls /tmp/bowtie2`
for f in $files_to_remove; do
echo "Deleting $f"
rm -f $f
done
rm -r /tmp/bowtie2
}
trap cleanup EXIT
mkdir /tmp/bowtie2 && cd /tmp/bowtie2
cat <<EOF > build.sh
#!/bin/bash
LDFLAGS=""
make static-libs && make RELEASE_BUILD=1
binaries="\
bowtie2 \
bowtie2-align-l \
bowtie2-align-s \
bowtie2-build \
bowtie2-build-l \
bowtie2-build-s \
bowtie2-inspect \
bowtie2-inspect-l \
bowtie2-inspect-s \
"
directories="scripts"
pythonfiles="bowtie2-build bowtie2-inspect"
PY3_BUILD="\${PY_VER%.*}"
if [ \$PY3_BUILD -eq 3 ]; then
for i in \$pythonfiles; do
2to3 --write \$i
done
fi
for i in \$binaries; do
cp \$i \$PREFIX/bin && chmod +x \$PREFIX/bin/\$i
done
for d in \$directories; do
cp -r \$d \$PREFIX/bin
done
EOF
cat <<EOF > meta.yaml
{% set version = "2.3.4.1" %}
package:
name: bowtie2
version: {{ version }}
source:
url: http://downloads.sourceforge.net/project/bowtie-bio/bowtie2/{{ version }}/bowtie2-{{ version }}-source.zip
sha256: a1efef603b91ecc11cfdb822087ae00ecf2dd922e03c85eea1ed7f8230c119dc
patches:
- bowtie2.patch
build:
number: 1
requirements:
build:
- {{ compiler('cxx') }}
host:
- python
run:
- python
- perl
test:
commands:
- bowtie2 --help
- bowtie2-align-l --help
- bowtie2-align-s --help
- bowtie2-build --help
- bowtie2-build-l --help
- bowtie2-build-s --help
- bowtie2-inspect --help
- bowtie2-inspect-l --help
- bowtie2-inspect-s --help
about:
home: 'http://bowtie-bio.sourceforge.net/bowtie2/index.shtml'
license: GPLv3
summary: Fast and sensitive read alignment
extra:
identifiers:
- biotools:bowtie2
- doi:10.1038/nmeth.1923
EOF
cat <<EOF > bowtie2.patch
--- bowtie2.orig 2017-01-20 19:41:25.706765000 -0500
+++ bowtie2 2017-01-20 16:23:38.574188000 -0500
@@ -38,10 +38,10 @@
my (\$vol,\$script_path,\$prog);
\$prog = File::Spec->rel2abs( __FILE__ );
-while (-f \$prog && -l \$prog){
- my (undef, \$dir, undef) = File::Spec->splitpath(\$prog);
- \$prog = File::Spec->rel2abs(readlink(\$prog), \$dir);
-}
+#while (-f \$prog && -l \$prog){
+# my (undef, \$dir, undef) = File::Spec->splitpath(\$prog);
+# \$prog = File::Spec->rel2abs(readlink(\$prog), \$dir);
+#}
(\$vol,\$script_path,\$prog)
= File::Spec->splitpath(\$prog);
EOF
cd -
conda-build /tmp/bowtie2
if [[ $? -ne 0 ]]; then
echo "Build failed... exiting"
exit 1
fi
echo "Build complete... Uninstalling your current bowtie2 runtime."
echo "Would you like to continue? yes/no?"
read ans
case "$ans" in
yes)
conda uninstall bowtie2
;;
*)
echo Exiting...
exit 1
;;
esac
conda install --use-local bowtie2
Thank you so much! I’ll try it tomorrow morning and report back if I have any problems! Simon
I encountered the same problem. The bowtie2-build is 2.3.4.1 from conda. I actually used a very small fasta file
a1 CATGTCCAGCTTCTCTTCAGTACCGCTCACCAGCCTAGGTGGGACCACTGACTGTGAGTC TGCAGTGGCCACCGCCCAGTCTCTGTGTCTCAAGCTCCAAGAGACGGTCACACACTAACC TGCAAGCCAAGGCTGGTGACTTTGACCATCCCTAACGCATGAGTTTTCCATGGAAACCTG GTCGGTGAACCTGACACGAAATTCCCAATTCCCCTTTACTCTGTACTGTGTGGCTGGTGC TCTTGTTTTCGTTCTCTCTCTCTCTCTCTCTCTCTCTCAAGTTGATTCCTCCATGTTGCT TTACAGAGACCTGCCAACTACCCAGGAATGTAAAAGCATTCATAGTATTTGTCTAGTAGA
No the two .rev.*.bt2 files Can anyone help with this issue?
I ended up being able to use the pre-built bowtie2
binary and works well. Get it here: https://sourceforge.net/projects/bowtie-bio/files/bowtie2/2.3.4.1
or from the command line
wget https://sourceforge.net/projects/bowtie-bio/files/bowtie2/2.3.4.1/bowtie2-2.3.4.1-linux-x86_64.zip/download -O bowtie2-2.3.4.1-linux-x86_64.zip
My research group is having the same (or similar) problem when running bioconda::bowtie2=2.3.4.1
as part of bioconda::humann2=0.11.1
. The error always something like:
Building a SMALL index
Index is corrupt: File size for /tmp/global/LLMGP_82144987094/MI-208-H/MI-208-H_humann2_temp/MI-208-H_bowtie2_index.1.bt2 should have been 132894396 but is actually 0.
Index is corrupt: File size for /tmp/global/LLMGP_82144987094/MI-208-H/MI-208-H_humann2_temp/MI-208-H_bowtie2_index.2.bt2 should have been 63736900 but is actually 0.
Please check if there is a problem with the disk or if disk is full.
Error: Encountered internal Bowtie 2 exception (#1)
...and sometimes in the "rev" index files are are zero-sized.
This error seems to happen stochastically for us, and usually only when the I/O load is really heavy. If I rerun the "problematic" samples in isolation (instead of 20-30 parallel humann2 jobs), then they usually work. Maybe it's some sort of file latency issue?
I recently had a pull request merged that changes the way conda builds bowtie 2. This new build process will be in effect for our most recent release of bowtie 2, v2.3.4.2. The updated build process is identical to the way we build our bowtie 2 binaries for distribution. The resulting binaries will have all dependencies statically linked which should solve what I think has been the root of this issue, the dynamically linked TBB library. Please give this new version a try and let me know if this problem still persists.
I will be closing this thread. Feel free to reopen if you believe that v2.3.4.2 has not addressed this issue.
Hello,
I am re-opening this thread since I'm not able to get proper index files with either 2.3.3.1 or 2.3.4.1, but I get them with 2.2.9. I am working on a cluster (not through conda). If it matters, the input files contains nearly 1M sequences. The stdout is
[domeni@r102 new_contig_mappings]$ bowtie2-build thawponds_assembly.fa thawponds_assembly
Settings:
Output files: "thawponds_assembly.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
thawponds_assembly.fa
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:18
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:10
bmax according to bmaxDivN setting: 569603352
Using parameters --bmax 427202514 --dcv 1024
Doing ahead-of-time memory usage test
Passed! Constructing with these parameters: --bmax 427202514 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
Building sPrime
Building sPrimeOrder
V-Sorting samples
V-Sorting samples time: 00:00:48
Allocating rank array
Ranking v-sort output
Ranking v-sort output time: 00:00:15
Invoking Larsson-Sadakane on ranks
Invoking Larsson-Sadakane on ranks time: 00:00:28
Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
(Using difference cover)
Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
Splitting and merging
Splitting and merging time: 00:00:00
Avg bucket size: 2.27841e+09 (target: 427202513)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 1
No samples; assembling all-inclusive block
and I get all files but the rev ones. I've read the previous messages in this thread, it's not clear to me if I should ask the admins to re-install the previous 2.3.x versions or install 2.3.4.2.
Many thanks,
Domenico
Hello,
How big is your input file? Are you using one of our pre-built binary packages?
Hi,
I still have the problem, that bowtie2-build does not create rev files and/or produces segmentation faults. Here are the versions I tested:
Hello @MarieLataretu ,
Can you please share the FASTA and the command line for the index that you are trying to build? This issue seems to happen sporadically and hence has been difficult to debug.
Hi,
this is the input fasta (unzipped):
ftp://ftp.ensembl.org/pub/release-92/fasta/mus_musculus/dna/Mus_musculus.GRCm38.dna.primary_assembly.fa.gz
here we add 'chr' to chromosomes as prefix:
sed -r '/^>/ s/>([1-9MXY])/>chr\1/' Mus_musculus.GRCm38.dna.primary_assembly.fa > Mus_musculus.GRCm38.dna.primary_assembly.chr.fa
and this is the command:
mkdir bowtie2-index
nice [/path/to/]bowtie2-build -t 20 /path/to/genome/Mus_musculus.GRCm38.dna.primary_assembly.chr.fa bowtie2-index/Mus_musculus.GRCm38.dna.primary_assembly.chr &> bowtie2-index/bowtie2-build.log
I have the same problem without the prefix and when the fasta file is in the same directory as the output.
If you have the bowtie2-build-s-debug
binary available can you try using it when building the index? I'd also appreciate it if you can save and share the debug output with me.
Sure!
the command is now:
nice /path/to/bowtie2-build-s-debug -t 20 /path/to/Mus_musculus.GRCm38.dna.primary_assembly.chr.fa bowtie2-index/Mus_musculus.GRCm38.dna.primary_assembly.chr.debug &> bowtie2-index/debug.log
and the resulting log file:
Settings:
Output files: "bowtie2-index/Mus_musculus.GRCm38.dna.primary_assembly.debug.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 20
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: enabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
/data/fass1/genomes/Eukaryots/mus_musculus_done/03052018/Mus_musculus.GRCm38.dna.primary_assembly.fa
Building a SMALL index
Reading reference sizes
Time reading reference sizes: 00:00:47
assert_leq: expected (20) <= (16)
bt2_idx.h:208
bowtie2-build-s-debug: bt2_idx.h:208: bool EbwtParams::repOk() const: Assertion `0' failed.
Only 3 and 4 index files were created (before only the rev files were missing); the running time dropped from > 10 minutes to ~ 1 minute.
Thanks for looking into this!
I switched to a dedicated Linux box and now am finally able to recreate the issue consistently. Will update this thread if/when I am able to find the cause of this issue. Thank you for your patience.
Did you pass in -t 20
with the intention of increasing the number of threads? When run in debug mode Bowtie2 will fail on the assertion that ftabChars should be less than or equal to 16, but you passed in the value 20. This also seems to affect the release build as well preventing Bowtie 2 from creating the reverse indexes. Try removing -t 20
, as I did, and see if the reverse files get created.
Apparently I had the same error, (Bowtie 2 version 2.3.5) "-rev" files were not generated. Is it possible that the new version has been corrected? I had problems and in the end it seems that I already produced the 6 files.
conda install -c bioconda bowtie2 bowtie2-build --threads 24 $genome $out > out.msg 2> out.err
At the moment it is running well and in parallel.
Aha,
-t 20 --thread 20
does not work, whereas --thread 20
works, as well as no -t
and --threads
parameters set!
The version is bowtie2-2.3.5.1.
I'm having a similar issue. After using conda I've used pre-built bowtie2 (23.5.1) downloaded from: https://netix.dl.sourceforge.net/project/bowtie-bio/bowtie2/2.3.5.1/bowtie2-2.3.5.1-linux-x86_64.zip
unzipped and executed the build command:
bowtie2-build ref_db.fa ref_db_packed --threads 2 --large-index --packed
The process ends suddenly , without any out of memory or or core-dump message.
Settings: Output files: "ref_db_packed.*.bt2l" Line rate: 7 (line is 128 bytes) Lines per side: 1 (side is 128 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: packed Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 8 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void*:8, int:4, long:8, size_t:8 Input files DNA, FASTA: ref_db.fa Building a LARGE index Reading reference sizes Time reading reference sizes: 00:16:12 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:11:35 bmax according to bmaxDivN setting: 17192161509 Using parameters --bmax 12894121132 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 12894121132 --dcv 1024 --packed Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 01:59:07 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:58:21 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 01:26:11 Sanity-checking and returning Building samples Reserving space for 22 sample suffixes Generating random suffixes QSorting 22 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 22 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Split 4, merged 12; iterating... Splitting and merging Splitting and merging time: 00:00:00 Split 3, merged 1; iterating... Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 9.82409e+09 (target: 12894121131) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 14 Getting block 2 of 14 Reserving size (12894121132) for bucket 1 Reserving size (12894121132) for bucket 2 Calculating Z arrays for bucket 1 Calculating Z arrays for bucket 2 Entering block accumulator loop for bucket 1: Entering block accumulator loop for bucket 2: bucket 1: 10% bucket 2: 10% bucket 1: 20% bucket 2: 20% bucket 1: 30% bucket 2: 30% bucket 1: 40% bucket 2: 40% bucket 1: 50% bucket 2: 50% bucket 1: 60% bucket 2: 60% bucket 1: 70% bucket 2: 70% bucket 1: 80% bucket 1: 90% bucket 2: 80% bucket 1: 100% Sorting block of length 12843066619 for bucket 1 (Using difference cover) bucket 2: 90% bucket 2: 100% Sorting block of length 7007973381 for bucket 2 (Using difference cover) Sorting block time: 06:08:46 Returning block of 7007973382 for bucket 2 Getting block 3 of 14 Reserving size (12894121132) for bucket 3 Calculating Z arrays for bucket 3 Entering block accumulator loop for bucket 3: bucket 3: 10% bucket 3: 20% bucket 3: 30% bucket 3: 40% bucket 3: 50% bucket 3: 60% bucket 3: 70%
The genome is pretty big (~100GB) so it might be the issue, but I'm not sure since while tracking the usage (both CPU and RAM) it seems to be OK.
Any idea what happening here?
(I think it is not a version issue because I tried both 2.4.3 via conda and pre-built 2.3.5.1).
Thanks!
I’m going to publish a beta build today that I think solves this issue. If you can help test the build when it is out that will be appreciated.
Having some trouble creating the build. Will try again tomorrow.
Thanks. I'll try when it will be ready. Just want to update that it do seems to be (also?) a memory issue. I've logged the memory while running and it seemed to use 100% of memory when it stopped. Is there anything to do except of using --packed and increasing --offset to reduce RAM usage while building the index? Thanks again!
I don't think there's any other way to decrease the size of index files other than using --packed
. You can try using more threads to speed up the process. Your issue seems slightly different from what's being reported here. It seems like bowtie2-build
is stopping prematurely as opposed to running to completion and not generating the reverse index. In any event here is a link to the build I created earlier. It's a one-off so I didn't bother uploading to GitHub.
Here's the SHA256 sum incase you want to verify the file's integrity (ignore the version number as I didn't bother updating the VERSION
file in the repo).
03d99c3c8b12219422575772b08076cd27c81eff0b4e3f0522fc998456898412 bowtie2-2.3.5.1-linux-x86_64.zip
Thanks! I'll try it once the current index build will finish. It somehow takes two and a half days... It enters the Ebwt loop again and again already 12 times (here is one of the twelve):
Entering Ebwt loop Getting block 1 of 1 No samples; assembling all-inclusive block Ran out of memory; automatically trying more memory-economical parameters. Using parameters --bmax 1089169922 --dcv 4096 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 1089169922 --dcv 4096 --packed Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 02:28:08 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:35:30 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:35:52 Sanity-checking and returning Building samples Reserving space for 254 sample suffixes Generating random suffixes QSorting 254 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 254 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 1.37537e+11 (target: 1089169921) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 1 No samples; assembling all-inclusive block Ran out of memory; automatically trying more memory-economical parameters. Using parameters --bmax 816877442 --dcv 4096 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 816877442 --dcv 4096 --packed Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 02:33:43 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:38:27 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:35:40 Sanity-checking and returning Building samples Reserving space for 338 sample suffixes Generating random suffixes QSorting 338 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 338 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 1.37537e+11 (target: 816877441) Converting suffix-array elements to index image Allocating ftab, absorbFtab
Does it seems to be stucked in infinite loop or it seems normal to you? My best!
Hi ch4rr0,
-threads 20
works fine -t 20
ends with the following message:-t/--ftabChars arg must not exceed 16
Error: Encountered internal Bowtie 2 exception (#1)
-t 15 --threads 20
is aborted: up_t_and_threads.logCommand:
nice /path/to/bowtie2-2.4.0-beta-linux-x86_64/bowtie2-build-s-debug [-t 20|--threads 20|-t 15 --threads 20] /path/to/Mus_musculus.GRCm38.dna.primary_assembly.chr.fa bowtie2-index/Mus_musculus.GRCm38.dna.primary_assembly.chr.debug
Thanks for the detailed log. I will look into the issue and get back to you once it's been resolved.
I believe I'm having the same problem.
I'm using MacOS Catalina and zsh. I used bowtie2 pre-compiled binaries.
After trying to use bowtie2-build for the first time it generated 4 files, but no .1.bt2 or .rev.1.bt2.
After trying to run the same command in a different folder, I got this:
ltelles@ltelles-mac lambda_virus % ~/Desktop/documents_desktop/bioinformatics/bowtie2-2.3.5.1-macos-x86_64/bowtie2-build ~/Desktop/documents_desktop/bioinformatics/bowtie2-2.3.5.1-macos-x86_6/example/reference/lambda_virus.fa lambda_virus
Settings:
Output files: "lambda_virus.*.bt2"
Line rate: 6 (line is 64 bytes)
Lines per side: 1 (side is 64 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Max bucket size: default
Max bucket size, sqrt multiplier: default
Max bucket size, len divisor: 4
Difference-cover sample period: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
/Users/ltelles/Desktop/documents_desktop/bioinformatics/bowtie2-2.3.5.1-macos-x86_6/example/reference/lambda_virus.fa
Error: could not open /Users/ltelles/Desktop/documents_desktop/bioinformatics/bowtie2-2.3.5.1-macos-x86_6/example/reference/lambda_virus.fa
Total time for call to driver() for forward index: 00:00:00
Error: Encountered internal Bowtie 2 exception (#1)
Command: /Users/ltelles/Desktop/documents_desktop/bioinformatics/bowtie2-2.3.5.1-macos-x86_64/bowtie2-build-s --wrapper basic-0 /Users/ltelles/Desktop/documents_desktop/bioinformatics/bowtie2-2.3.5.1-macos-x86_6/example/reference/lambda_virus.fa lambda_virus
I am sure I used the correct paths for bowtie2-build
and for the lambda_virus.fa
files.
@MarieLataretu -- I fixed the issue that was causing this in a recent commit, and confirmed it worked by building the GRCm38 without failure. I have not created a build with that includes this commit, but the issue should only affect bowtie2-build when running in debug mode.
@pmdbio -- You appear to have a typo in your path: missing the 4
after in x86_64
.
~/Desktop/documents_desktop/bioinformatics/bowtie2-2.3.5.1-macos-x86_6/example/reference/lambda_virus.fa
@YiJessePi -- For such a large reference, 100GB, it's not unusual for bowtie2-build to take a long time. The output may seem repetitive, but unless bowtie2-build has crashed it's almost guaranteed that it is working on building the index.
@ch4rr0 Thanks for spotting that. Indeed it was a typo on the path.
you just need to set it to produce a large index using the option and not small. it automatically detects and produces a small index sometimes which don't have the rev files but some tools are build to expect a large index or might just be coincidence that worked for me one time
@ch4rr0 Hi! I hope to help me pls. I´m using the subsystem of Linux in windows10 pro, and I run "Bowtie" by UBUNTU, i downloaded it from Microsoft Store. My problem is this: I stay in Ubuntu:
joeltorres@DESKTOP-EBIAPSJ: ~$ bowtie
No index, query, or output file specified!
Usage:
bowtie [options]* } [
@ch4rr0 Hi! I hope to help me pls. I´m using the subsystem of Linux on windows10 pro, and I dowloaded and run "Bowtie" by UBUNTU, i downloaded UBUNTU from Microsoft Store. My problem is this: I stay in Ubuntu:
joeltorres@DESKTOP-EBIAPSJ: ~$ bowtie No index, query, or output file specified! Usage: bowtie [options]* {-1 -2 | --12 | --interleaved etc...
I dont understand, how can i resolve this? I need your help. I'm new in bioinformatics
Hi there, I also had this problem and I saw this comment on Biostars... I try to use another reference file without Ns nucleotides (i.e., only those coding for transcripts) and the ".rev" files were (in appearance) correctly generated. I hope that helps!
@joeltorres96 — what command are you trying to run? You provided an example of you running bowtie without any parameters which will result in the usage message getting printed.
Hi,
I have just started learning bowtie2. I encountered a problem with database indexing with bowtie2-build command: indexing runs through but the output doesn't have .rev files, thus preventing me from doing alignments to the particular database. The db file to be indexed is a FASTA file, that is a collection of bacterial genomes. Previously I have successfully indexed and aligned to a db of the same kind (merged fasta files), so now I have no clue what is wrong with the setting or what is going on. I get the 4 other files in reasonable time, although multiple memory-usage test were performed by the software before passing.
I could not find any help online regarding this question. I wonder what I'm doing wrong?
BRs,
Anna